Post Not Found

Let’s continue our work on Pipeline Editor today. The pipeline editor is the “Pro” feature which allows people to create and customize their own pipelines.

Where we stand now? We have one pipeline — character generation pipeline — and it is hard-coded.

What do we do today? We are going to convert it into a JSON.

What is Pipeline?

First of all, let’s define what “pipeline” is for SpriteDX. A Pipeline is set of steps that we must take to generate characters and other assets. We call those steps “stages.” As discussed in previous posts, the default character generation pipeline contains 4 stages—character generation, animation generation, editing, post-processing.

Pipeline also defines set of “inputs“ needed from users for the stages.

In this screenshot, let’s name the side panel, “Parameter Board”, our Pipeline JSON should be able to describe the structure of this parameter board and all of the stages.

Pipelines and Stages

In this section, we define the relationship between pipelines and stages.

At high level, each pipeline has stages. Then the question arises whether a stage can have its on set of stages?

While in vast majority of cases, we will have stages be the leaf nodes. However, for schema design, it is best to keep things flexible so that in the future we can have nested stages.

A Pipeline is simply the root stage.
A Stage is a unit of work that may contain steps (atomic operations) and/or sub-stages (nested phases).
This recursive definition means that stages can be both leaves (doing actual work) or containers (grouping other stages).

Why allow nested stages?

Flexibility → future-proofs the schema. Even if today’s use cases don’t need nesting, tomorrow’s more complex workflows might.
Composability → a stage can be plucked out and reused as a standalone pipeline, or embedded into a bigger one.
Clarity → keeps the model uniform: everything is a stage, whether it’s the whole pipeline or a small sub-piece.

Example Model

Let’s start with an empty pipeline.

{
  "name": "Generate Character",   // Public Title
  "type": "pipeline"              // Type of the object, enforces name, controls.
  "description": "This is…",      // Public Description
  "inputs": {},                   // Inputs
  "outputs": {},                  // Outputs
  "stages": []                    // Array of subtasks         
  "controls": [],                 // Controls in Parameter Panel              
  "mapping": {}                   // Mapping from inputs to stages
}

Let’s define first 2 stages.

{
  "id": "character-pipeline-v1",
  "name": "Single Stage Character Generation",
  "description": "This is...",
  "controls": [
    { "type": "section", "label": "Template", "items": ["this.templateImage"] },
    { "type": "section", "label": "Prompt", "items": ["generate.prompt", "this.seed"] },
    { "type": "section", "label": "Animation", "items": ["animate.sceneDescription", "animate.shots"] }
  ],
  "inputs": {
    "templateImage": {
      "type": "image",
      "defaultValue": "template1.png",
      "editor": {
        "type": "select",
        "options": [
          { "label": "Template 1", "value": "template1.png" },
          { "label": "Template 2", "value": "template2.png" }
        ]
      }
    },
    "seed": {
      "type": "number",
      "defaultValue": 42
    }
  },
  "outputs": {
    "animatedWEBP": { "type": "image" }
  },
  "stages": [{
    "id": "generate",
    "statusMessage": "Generating Reference…",
    "inputs": {
      "templateImage": { "type": "image", "defaultValue": null },
      "prompt": { "type": "string", "defaultValue": "…", "editor": { "type": "textarea" } },
      "seed": { "type": "number", "defaultValue": 42 }
    },
    "outputs": {
      "referenceImage": { "type": "image" }
    },
    "expectedDuration": 60,
    "run": {
      "type": "comfyui-workflow",
      "workflowRef": "path/to/workflow.json",
      "inputMappings": {
        "referenceImage": ["51.inputs.image"],
        "fullPrompt": ["69.inputs.prompt"]
      },
      "outputMappings": {
        "51": ["animatedWEBP"]
      }
    }
  }, {
    "id": "animate",
    "statusMessage": "Animating Character…",
    "inputs": {
      "referenceImage": { "type": "image" },
      "sceneDescription": { "type": "string", "defaultValue": "…", "control": { "type": "textarea" } },
      "shots": {
        "type": "Array<{ id: string, prompt: string, loop: boolean }>",
        "defaultValue": [
          { "id": "greet", "prompt": "...", "loop": false },
          { "id": "idle",  "prompt": "...", "loop": true  },
          { "id": "run",   "prompt": "...", "loop": true  }
        ],
        "editor": {
          "type": "table",
          "columns": [
            { "key": "id", "label": "ID" },
            { "key": "prompt", "label": "Prompt" },
            { "key": "loop", "label": "Loop" }
          ]
        }
      },
      "fullPrompt": {
        "type": "string",
        "computed": [{
          "type": "map",
          "inputs": ["shots"],
          "as": "shot",
          "return": "`[SHOT ${idx}] ${shot.prompt} \n\n`"
        }, {
          "type": "reduce",
          "inputs": ["computed"],
          "as": ["acc", "curr"],
          "initialValue": "`${sceneDescription}\n\n`",
          "return": "`${acc}${curr}`"
        }]
      }
    },
    "outputs": {
      "animatedWEBP": { "type": "image" }
    },
    "run": {
      "type": "comfyui-workflow",
      "workflowRef": "path/to/workflow.json",
      "inputMappings": {
        "referenceImage": ["51.inputs.image"],
        "fullPrompt": ["69.inputs.prompt"]
      },
      "outputMappings": {
        "51": ["animatedWEBP"]
      }
    }
  }],
  "mapping": {
    "this.inputs.templateImage": ["generate.inputs.templateImage"],
    "generate.outputs.referenceImage": ["animate.inputs.referenceImage"],
    "animation.outputs.animatedWEBP": ["this.outputs.animatedWEBP"]
  }
}

At high level, we have stages which describes unit of works. Stages have run field which defines a runner and currently supported runner type is comfyuiworkflow.

Each stage defines inputs and outputs, and mapping at pipeline level, maps pipeline inputs and outputs to stage inputs and outputs.

This is little more verbose that how I would like. The “mapping“ section can simply be folded into the inputs. Let me try that.

{
  "id": "character-pipeline-v1", "name": "Single Stage Character Generation", "description": "This is...",
  "controls": [
    { "type": "section", "label": "Template", "items": ["this.templateImage"] },
    { "type": "section", "label": "Prompt", "items": ["generate.prompt", "this.seed"] },
    { "type": "section", "label": "Animation", "items": ["animate.sceneDescription", "animate.shots"] }
  ],
  "inputs": {
    "templateImage": { "type": "image", "defaultValue": "template1.png", "editor": {
      "type": "select", "options": [{ "label": "Template 1", "value": "template1.png" }, { "label": "Template 2", "value": "template2.png" }]
    } },
    "seed": { "type": "number", "defaultValue": 42 }
  },
  "outputs": { "animatedWEBP": { "type": "image" } }, "expectedDuration": 60,
  "stages": [{ "id": "generate", "statusMessage": "Generating Reference…",
    "inputs": {
      "templateImage": { "type": "image", "defaultValue": null, "source": "..templateImage" },
      "prompt": { "type": "string", "defaultValue": "…", "editor": { "type": "textarea" } },
      "seed": { "type": "number", "defaultValue": 42 }
    },
    "outputs": { "referenceImage": { "type": "image" } },
    "run": { "type": "comfyui-workflow", "workflowRef": "path/to/workflow.json",
      "inputs": {
        "51.inputs.image": { "source": "this.inputs.templateImage" },
        "69.inputs.prompt": { "source": "this.inputs.prompt" }
      },
      "outputs": { "referenceImage": { "source": "51" } }
    }
  }, { "id": "animate", "statusMessage": "Animating Character…",
    "inputs": {
      "referenceImage": { "type": "image", "source": "..generate.referenceImage" },
      "sceneDescription": { "type": "string", "defaultValue": "…", "control": { "type": "textarea" } },
      "shots": { "type": "Array<{ id: string, prompt: string, loop: boolean }>",
        "defaultValue": [
          { "id": "greet", "prompt": "...", "loop": false },
          { "id": "idle",  "prompt": "...", "loop": true  },
          { "id": "run",   "prompt": "...", "loop": true  }
        ],
        "editor": { "type": "table", "columns": [{ "key": "id", "label": "ID" }, { "key": "prompt", "label": "Prompt" }, { "key": "loop", "label": "Loop" }] }
      },
      "fullPrompt": { "type": "string", "computed": [
        { "type": "map", "inputs": ["shots"], "as": "shot", "return": "`[SHOT ${idx}] ${shot.prompt} \n\n`" },
        { "type": "reduce", "inputs": ["computed"], "as": ["acc", "curr"], "initialValue": "`${sceneDescription}\n\n`", "return": "`${acc}${curr}`" }
      ] }
    },
    "outputs": { "animatedWEBP": { "type": "image", "source": "run.outputs.animatedWEBP" } },
    "run": { "type": "comfyui-workflow", "workflowRef": "path/to/workflow.json",
      "inputs": {
        "51.inputs.image": { "source": "this.inputs.referenceImage" },
        "69.inputs.prompt": { "source": "this.inputs.fullPrompt" }
      },
      "outputs": { "animatedWEBP": { "source": "51" } }
    }
  }]
}