Plan of Record — Dancing Stick Figures
Figure 1 - Chibi Designs (Sprite DX, q-version, 3D wireframe, game, ragnarok online, prototype rendering)
After Dancing Stick Figures, we are experimenting with the idea of building a flat shaded chibi figure with articulated face.
The basic idea is that:
Create a very basic flat shaded chibi figure rig.
Live animate the body using Nvidia ARDY or Nvidia MotionBricks.
Use Nvidia Audio2Face to animate the facial movements.
Use Qwen3-TTS(sohee) for character voice acting.
Create a demo website which shows a prototype of this.
Make it into a dataset, and train a streaming model out of it.
The basic idea
procedurally
generated character
+
real-time generated
voice, expressions, movements
↓
DATASET
&
Pipeline
↓
Trained Streamer model
(long term vision)
An alternative
Alternative to this idea is to use SOTA video generation models to generate bunch of live action videos of characters, then use those to learn streamer model directly bypassing all procedural generation efforts.
Constraints
128x128 Output Space
Start with limited motions.
What are we going to do
We are actually performing both experiments in parallel.
Track A (Cladia): Procedural Char + ARDY + Audio2Face
Track B (Pixel): Train a model that replicates SpriteDX character animations
Track A is a good middle ground because there exists lots of prior art in VRoid VRChat community. The facial action also directly ties with the ARKit approach in Audio2Face.
Track B is more of a test of what kinda of model would be trainable within the kind of compute and dataset budget we have. Also this is to uncover any issues in training such model in back burner.
-- Sprited Dev
