What's after Dancing Stick Figures

We submitted Dancing Stick Figures paper to arXiv. Now what?

Here is my mental map:

          [Digital Being Track]

      Dancing Stick Figures Dataset        <-- done
                   ↓
   Real-Time Auto Regressive Stick Figure  <-- starting
                   ↓
                Web Demo

So, my next stop would be to build "Real-Time Auto Regressive Stick Figure."

For this, we did M6, v9 experiments which is Auto Regressive Latent Diffusion that turns prompts into animated 64x64 sprite stream near real time.

V9 - Latent Auto Regressive Diffusion (64x64)

We used Dancing Stick Figures dataset and produced M6 and later v9 model which works just like Nvidia ARDY but outputs 64x64 RGBA video stream.

We do get real time performance but the inference stream is not great.

https://youtu.be/IFxf8b_seXM

This model doesn't really perform well and I think there is some drift which make the model perform worse once it's been running for a while.

Using RTX Pro 6000, the model is able to produce realtime 20fps actions with decent amount of room. It is about 2x slower than what ARDY natively produces.

Model setup	VRAM	Speed	Fresh-prompt latency
ARDY, encoder on GPU	16.05 GB	~9.8×	0.11 s
ARDY, encoder on CPU	1.19 GB GPU	~7× after encoding	3.38 s
V9 on GPU	1.37 GB	~4–4.8×	0.44 s

Here we need to decide whether we continue on with latent pixel diffusion, or just use ARDY and render procedurally.

My passion is at pixel streaming model (what I call expressive canvas) but rig-motion models like ARDY probably is more research friendly.

V9 actually tries to solve this problem by inferencing both ARDY output as well as diffused canvas. However, unless I am able to train it for long time with ARDY, I will have to spend lots of time replicating the ARDY's success.

---

Since our source motion library is basically what ARDY already knows. I probably want to study it further in its native form first.

Let's keep it native for now but project it to 2D canvas.

---

Next Steps

1. Do an cold-outreach on the project for help.

2. Study ARDY paper.

3. Experiment further with ARDY's weaknesses.

4. Try to improve ARDY v1 and contribute back.

There are three things I can focus on:

1. Expand the ARDY's ability (more motion, make it better)

2. Make our own ARDY and dataset.

3. Make motion prompt library of ARDY and release.

4. Iterate on Dancing Chibi Figures.

5. Write technical report about DSF v9 (pixel + rig inference).

There are quite a few areas I can contribute. But I need to choose one that can produce an artifact (report, dataset or product) within 2 weeks.

---

At this time, if ultimate goal is to create a streamer model for sprites, I need to first prove that believable streaming model can be made for stick figures.

Let's work on streaming model for stick figures.

M6/V9 are good start but they don't work well. For next technical report, we will try to build an academic streaming model using stick figures.

-- Sprited Dev