SpriteDX - Template Picker Integration
99 days til Alpha. We added computed properties and before we move into Stage 3, I have some missing wiring in the Template Picker.
The Problem
Currently, the template picker passes the correct image to the workflow based on the user selection of the template. But the template dimensions and layout is all hard coded.
We need to wire up the row: 2, col: 4, and indices: 1,3,5,7 information and map it to a workflow that can handle more generalized form of the template (i.e. x by y template layout with indices for extraction).
Alternatives
Is this template format the right format to use? With tile image with alpha mask and some descriptors like rows and columns? Well, one thing for sure is that this is not the only form. There are some alternatives.
Option 1: Single Sheet Fill In The Blank
Basically the format above where we supply a n-by-m template image with “holes”, then specify the grid and index positions to extract the character from. To me, this is the most self explanatory format. That is, when user is using SpriteDX, they can intuitively can tell that there are holes and we are trying to fill a hole.
So, if user wants to create their own template image, they can just look at this format and replace the characters in there to create their own template.
On the other hand, those holes may throw off some people who are expecting more of “magical experience” and don’t really want to understand what is going on.
Option 2: Set of Individual Characters
Alternatively, we could have the template be a set of images rather than a single image with n-by-m characters. In this mode, users could customize their template by uploading set of individual character images. Here is a very crude mockup:
In this mode, we simply present the individual characters in a grid. It’s like saying “provide me N character references“ and I will draw you the N+1-th one.
To be frank, after mocking it up here, it doesn’t look that bad. There is no notion of “filling in the blank,“ but that can be more of implementation detail.
One other side benefit is that we do not need to specify the grid dimensions or the indices. Instead, given set of images we get to structure our own grid. So, if we are provided 4 references, we can create the 2 row by 3 column and fill in 4 of the tiles and generate the remaining 2.
Intuition
Okay, now let’s talk about why we haven’t gone with Option 2 from the get-go. The main problem is that multiple images is more conceptually taxing than providing single image. It is like providing single sheet of paper as opposed to providing a multiple sheets of papers.
Rather than doing the multi sheet approach, I plan to stick with the single sheet approach.
ComfyUI Workflow Setup
So, current ComfyUI looks like this:
We call FLUX .10 [fill] model with empty regions and we extract regions out of it.
It currently does not use information like row, col and indices. Let’s try to make use of it.
Here is a version that takes in the row, col and indices and produces list of generated images.
Integration
We are using some custom nodes here.
Let’s add them to our docker file.
RUN comfy-node-install comfyui-impact-pack comfyui-custom-scripts comfyui-list-utils
Then pushed to https://hub.docker.com/r/sprited/worker-comfyui.
Then exported the workflow and replaced the previous workflow file.
Great. Now, the row, col and indices fields are getting mapped correctly to ComfyUI workflows.
Next Steps
So, what’s next?
Stage 3: Next, we will move on to re-enabling the stage 3 using the pipeline.json. In stage 3, we split the shots and extract loops. Hopefully it is going to be straightforward.
Specs: Otherwise, the
specssyntax is little fragile because right now, we assume that specs is encoded inside the image url. And we don’t have handling for default values and we don’t really have any way to enter in the specs in the UI.Image Uploader: On a related note, we do not have any way to upload images into image field. We should implement that and test out uploading templates.
Local ComfyUI instance: So far we’ve been running ComfyUI inside RunPod Serverless. In order to be truly capable local tool, we should add ability to run/wire-up local instance of comfy.
Custom Pipeline User Flow: Now that we have pipeline.json, we can now allow users to upload there own pipeline.json and start running it.
We will focus on Stage 3 integration first to close the loop. Until we have a fully functional pipeline.json, I will not move forward to next tasks.
— Sprited Dev 🌱
