# SpriteDX - Pipelining Character Generation

The character generation pipeline is built around 4 core stages: character generation, animation generation, editing and post-processing.

Yesterday, we defined the [pipeline JSON schema](https://blog.sprited.app/spritedx-pipelines). Today, our focus is on converting the hardcoded character generation pipeline into this JSON format.

---

## Before we get started…

Let’s review the schema at high level.

```json
{
  "id": "pipeline id", "name": "pipeline name", 
  "controls": [ …ui controls that appears on parameter board… ],
  "inputs": { …inputs at pipeline level… },
  "outputs": { …outputs at pipeline level… },
  "stages": [{
    "id": "stage id", "name": "stage name", 
    "inputs": { …inputs at stage level… },
    "outputs": { …outputs at stage level… },
    "run": { …information on how to run this stage… }
  }, …]
}
```

This is the schema at high level. If we were to make it really bare bone, we could simplify this further.

```diff
 {
   "id": "pipeline id", "name": "pipeline name", 
-  "controls": [ …ui controls that appears on parameter board… ],
-  "inputs": { …inputs at pipeline level… },
-  "outputs": { …outputs at pipeline level… },
   "stages": [{
     "id": "stage id", "name": "stage name", 
     "inputs": { …inputs at stage level… },
     "outputs": { …outputs at stage level… },
     "run": { …information on how to run this stage… }
   }, …]
 }
```

In this version, we are viewing the pipeline to be simply a bag of stages.

For the `controls`, we can generate it by hoisting the parameters from the stages. For example, given following pipeline,

```json
{
  "id": "character-pipeline-v1", "name": "Generate Character",
  "stages": [{ 
    "id": "generate", "name": "Character Generation",
    "inputs": { "prompt": { … } },
    "outputs": { "referenceImage": { "type": "image" } },
    "run": { … }
  }]
}
```

The pipeline’s `inputs` will auto generated to be `{ "generate.inputs.prompt": … }`, and `outputs` will be `{ "generate.outputs.referenceImage" }`.

For the `controls` we can generate a section for each stage and use the name of the stage as section header then put default input boxes for the input parameters.

In my mind this keeps the JSON structure simpler. Also it becomes conceptually easier and more facile to have one to one mapping to the pipeline stages and the sections in the parameter board.

Now, then let’s imagine we defined our character pipeline into this, we will be able to get rid of the top section.

```javascript
{
  "id": "character-pipeline-v1", "name": "Generate Character"
  // "controls": [
  //   { "type": "section", "label": "Template", "items": ["this.templateImage"] },
  //   { "type": "section", "label": "Prompt", "items": ["generate.prompt", "this.seed"] },
  //   { "type": "section", "label": "Animation", "items": ["animate.sceneDescription", "animate.shots"] }
  // ],
  // "inputs": {
  //   "templateImage": { "type": "image", "defaultValue": "template1.png", "editor": {
  //     "type": "select", "options": [{ "label": "Template 1", "value": "template1.png" }, { "label": "Template 2", "value": "template2.png" }]
  //   } },
  //   "seed": { "type": "number", "label": "Seed", "defaultValue": 42 }
  // },
  // "outputs": { "animatedWEBP": { "type": "image" } }, "expectedDuration": 60,
  "stages": [{ 
    "id": "generate", "name": "Character",
    "statusMessage": "Generating Reference…",
    "inputs": {
      "templateImage": { "type": "image", "defaultValue": null, "editor": { … } },
      "prompt": { "type": "string", "defaultValue": "…", "editor": { "type": "textarea" } },
      "seed": { "type": "number", "defaultValue": 42 }
    },
    "outputs": { "referenceImage": { "type": "image" } },
    "run": { "type": "comfyui-workflow", "workflowRef": "path/to/workflow.json",
      "inputs": {
        "51.inputs.image": { "source": "this.inputs.templateImage" },
        "69.inputs.prompt": { "source": "this.inputs.prompt" }
      },
      "outputs": { "referenceImage": { "source": "51" } }
    }
  }, { 
    "id": "animate", "name": "Animation",
    "statusMessage": "Animating Character…",
    "inputs": {
      "referenceImage": { "type": "image", "source": "..generate.referenceImage" },
      "sceneDescription": { "type": "string", "label": "Scene Description", "defaultValue": "…", "control": { "type": "textarea" } },
      "shots": { "type": "Array<{ id: string, prompt: string, loop: boolean }>",
        "defaultValue": [
          { "id": "greet", "prompt": "...", "loop": false },
          { "id": "idle",  "prompt": "...", "loop": true  },
          { "id": "run",   "prompt": "...", "loop": true  }
        ],
        "editor": { "type": "table", "columns": [{ "key": "id", "label": "ID" }, { "key": "prompt", "label": "Prompt" }, { "key": "loop", "label": "Loop" }] }
      },
      "fullPrompt": { "type": "string", "computed": [
        { "type": "map", "inputs": ["shots"], "as": "shot", "return": "`[SHOT ${idx}] ${shot.prompt} \n\n`" },
        { "type": "reduce", "inputs": ["computed"], "as": ["acc", "curr"], "initialValue": "`${sceneDescription}\n\n`", "return": "`${acc}${curr}`" }
      ] }
    },
    "outputs": { "animatedWEBP": { "type": "image", "source": "run.outputs.animatedWEBP" } },
    "run": { "type": "comfyui-workflow", "workflowRef": "path/to/workflow.json",
      "inputs": {
        "51.inputs.image": { "source": "this.inputs.referenceImage" },
        "69.inputs.prompt": { "source": "this.inputs.fullPrompt" }
      },
      "outputs": { "animatedWEBP": { "source": "51" } }
    }
  }]
}
```

and the UI will look like

```plaintext
v Character
  Template Picker
  Prompt
  Seed
v Animation
  Scene Description
  Shots
…
```

So, the ability to easily create a separate “Template“ section is now compromised. However, this creates nice one-to-one matching and makes it much easier to debug an troubleshoot.

Then, now lets look at a stage in more detail.

```json
{ 
  "id": "generate", "statusMessage": "Generating Reference…",
  "inputs": {
    "templateImage": { "type": "image", "defaultValue": null, "source": "..templateImage" },
    "prompt": { "type": "string", "defaultValue": "…", "editor": { "type": "textarea" } },
    "seed": { "type": "number", "defaultValue": 42 }
  },
  "outputs": { "referenceImage": { "type": "image" } },
  "run": { "type": "comfyui-workflow", "workflowRef": "path/to/workflow.json",
    "inputs": {
      "51.inputs.image": { "source": "this.inputs.templateImage" },
      "69.inputs.prompt": { "source": "this.inputs.prompt" }
    },
    "outputs": { "referenceImage": { "source": "51" } }
  }
}
```

Its seems rather complicated with the two type of inputs/outputs pair appearing at stage level and again inside `run`. We could simplify this further.

```json
{ 
  "id": "generate", "statusMessage": "Generating Reference…",
  "runner": "comfyui",
  "inputs": {
    "templateImage": { "type": "image", "defaultValue": null, "comfyui": "59.image" },
    "prompt": { "type": "string", "defaultValue": "…", "comfyui": "59.prompt" "editor": { "type": "textarea" } },
    "seed": { "type": "number", "defaultValue": 42, "comfyui": "59.seed" }
  },
  "outputs": { "referenceImage": { "type": "image", "from": "workflow.59" } },
  "comfyuiWorkflowRef": "path/to/workflow.json"
}
```

In this version, we have removed “run“ payload and merged it into the stage payload.

This seems much simpler. Let’s use try out this format. Here is first 2 stage pipeline.

```json
{
  "id": "character-pipeline-v1",
  "name": "Generate Character",
  "description": "This is SpriteDXʼs default pipeline for 2D character sprite sheet generation.",
  "stages": [
    {
      "id": "generate",
      "name": "Character",
      "statusMessage": "Generating Reference…",
      "runner": "comfyui",
      "workflowRef": "path/to/workflow.json",
      "inputs": {
        "templateImage": {
          "type": "image",
          "options": [{ "label": "Template 1", "value": "template1.png" }],
          "mapTo": "17"
        },
        "prompt": {
          "type": "string",
          "defaultValue": "…",
          "multiline": true,
          "mapTo": "69.prompt"
        },
        "seed": { "type": "number", "defaultValue": 42, "mapTo": "69.prompt" }
      },
      "outputs": { "referenceImage": { "type": "image", "mapTo": "69" } }
    },
    {
      "id": "animate",
      "name": "Animation",
      "runner": "ComfyUI",
      "workflowRef": "path/to/workflow.json",
      "statusMessage": "Animating Character…",
      "inputs": {
        "referenceImage": { "type": "image", "mapTo": "1" },
        "sceneDescription": {
          "type": "string",
          "label": "Scene Description",
          "defaultValue": "…",
          "multiline": true
        },
        "shots": {
          "type": "Array<{ id: string, prompt: string, loop: boolean }>",
          "defaultValue": [
            { "id": "greet", "prompt": "...", "loop": false },
            { "id": "idle", "prompt": "...", "loop": true },
            { "id": "run", "prompt": "...", "loop": true }
          ],
          "editor": {
            "type": "table",
            "columns": [
              { "key": "id", "label": "ID" },
              { "key": "prompt", "label": "Prompt" },
              { "key": "loop", "label": "Loop" }
            ]
          }
        },
        "fullPrompt": {
          "type": "string",
          "computed": [
            {
              "type": "map",
              "args": ["shots"],
              "as": "shot",
              "return": "`[SHOT ${idx}] ${shot.prompt} \n\n`"
            },
            {
              "type": "reduce",
              "args": ["computed"],
              "as": ["acc", "curr"],
              "initialValue": "`${sceneDescription}\n\n`",
              "return": "`${acc}${curr}`"
            }
          ],
          "mapTo": "51.prompt"
        }
      },
      "outputs": { "animatedWEBP": { "type": "image", "mapTo": "21" } }
    }
  ]
}
```

It looks rather long at a glance, but at least there is no duplication or extra hashes. Let’s proceed with this.

For schema construction, we are using [Typebox](https://github.com/sinclairzx81/typebox). We are starting pretty basic here.

```typescript
import { Type as T, type Static } from '@sinclair/typebox'

const Id = T.String({ pattern: String.raw`^[A-Za-z0-9_-]+$` });

const InputSchema = T.Object({
  type: T.Union([
    T.Literal('image'),
    T.Literal('number'),
    T.Literal('string')
  ]),
  label: T.Optional(T.String()),
  defaultValue: T.Optional(T.Unknown()),
  mapTo: T.Optional(T.String())
})

const BaseStageSchema = T.Object({
  id: Id,
  name: T.String(),
  description: T.Optional(T.String()),
  statusMessage: T.Optional(T.String()),
  inputs: T.Record(Id, InputSchema),
  outputs: T.Record(Id, InputSchema),
  expectedDuration: T.Optional(T.Number()),
})

const RecursiveStageSchema = (StageSchema: any) => T.Intersect([
  BaseStageSchema,
  T.Object({
    stages: T.Array(StageSchema)
  })
])

const ComfyStageSchema = T.Intersect([
  BaseStageSchema,
  T.Object({
    runner: T.Literal('ComfyUI'),
    workflowRef: T.String(),
    stages: T.Optional(T.Undefined())
  })
])

export const StageSchema = T.Recursive((StageSchema) =>
  T.Union([
    RecursiveStageSchema(StageSchema),
    ComfyStageSchema
  ])
);

export const PipelineSchema = StageSchema;

export type Stage = Static<typeof StageSchema>
export type Pipeline = Static<typeof PipelineSchema>
```

From the typing standpoint, `Stage` and `Pipeline` are currently set to be equal.

So, in theory, you could technically have a single stage as a pipeline, and you can have stages that have nested stages. That’s all future-proofing anyways. Most of the pipelines we create will have a simple structure of pipelines owning single level of stages.

---

I integrated this schema and working on creating UI controls based on the `pipeline.json`. Still work in progress but have to call it a day!

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1757134243398/d4f987b6-09f8-4a05-a654-56ce4d772b43.png align="center")

— Sprited Dev 🌱