SpriteDX - How to allow users to upload their own character

Motivation

Instead of generating characters from a prompt, people may already have characters they’ve worked on. In those cases, I still want to allow people to use the tool to generate animation frames.

---

UX Design

Before:

Template

- Template Selector
Prompt
- Character Name
- Character Description
- Seed

After:

Character

- ( • ) Generate
- Template
- Prompt
- ( ) Upload [Select File]

And:

• We will fold Character Name (default to unisex name) and Seed into Advanced mode.

• Upload your character will basically disable “Describe Your Character“ and “Select a Template.“

Iteration #2

Something that is possible to be generalized better in pipeline schema.

(●) Generate Character
     Template
     Describe your character…

(○) Upload Character
     [ Select image file ]
     PNG / WEBP · Idle pose · Front-facing recommended

Iteration #3

May be it is getting too complex. Perhaps, we just do what we can.

v Template
  [Template Selector                   ]
  [Template Preview                    ]
v Prompt
  [                                    ]
  Seed             [           12314122]
  Or Upload        [Select your file...]

Then as a bonus feature allow the inputs to be disabled when certain conditions are met. For, now we won’t do anything like that.

---

How to Implement

1. Once we have the image uploading feature, we will need to store it in a OPFS or equivalent.

2. Once it is stored, we will reference it using some url scheme.

3. Then, send the image in data-uri to the comfyUI server.

4. ComfyUI workflow will notice that there is already an image file and bypass the generation flow. This part is going to be tricky but I think there has been previous arts.

---

Alternative to Control Flows in ComfyUI

Control flows in ComfyUI may be difficult and flaky to implement. An alternative is to have frontend decide whether to run the workflow or not. This could possibly save time and resource.

We can have something like:

if certain condition is met then:
   run child stage A
otherwise:
   run child stage B

Basically, the stage is a “controller” stage which determines which one of the stages to run based on input.

This goes well with the idea that SpriteDX is a Orchestration layer on top of ComfyUI.

---

Schema Design

Iteration #1

    {
      "id": "generate",
      "type": "if",
      "inputs": { … },
      …,
      "stages": [{
        "condition": {              // if
          "sandbox": "quickjs",
          "with": {
            "upload": "..upload"
          },
          "script": "!upload"
        },
        …
      }, {
        "condition": { … }           // else if
      }, {
        …                            // else
      }]
    },

Or something simpler with limited functionality would be:

Iteration #2

    {
      "id": "generate",
      "when": {             
        "sandbox": "quickjs",
        "with": {
          "upload": "..upload"
        },
        "script": "!upload"
      }, … 
    },

Which would only run “when” is satisfied.

Then there would need to be a separate stage that handles else case explicitly.

---

Discussion

The logic gets complicated to transcribe into JSON format.

It almost seems like it would be easier if we just let people to specify all this using code instead of having it specified declaratively using limited syntax in JSON.

Since we already have ability to run JS and Python in sandbox mode, how about we allow pipeline designers to simply provide the full pipeline in a code.

Something like:

export params = [...]

export async function main(context) {
  // Do all that in the main. Like running and emitting statuses.
  // context object provides ability to emit status,
  // and submit comfyUI request and whatever is needed to run things.
  // we could even make it typescript compatible so that it is 
  // easy to make it.
  // Json is getting really complex and perhaps this is the way forward.
}

---

Let’s stop here and continue tomorrow

— Sprited Dev 🌱