SpriteDX - Stage 4 Pipeline JSON Integration (Part 1)
We just finished Stage 3 integration. Let’s move on to Stage 4.
Stage 4’s name is “post processing.“ In this stage, we remove backgrounds from the animated clips idle and run. Here is how the Comfy Workflow look like right now:
Converting Animated WebP into sprite sheet
First step is to convert animated webp file into a spritesheet. This spritesheet will then be fed into Flux1 Kontext. We do not know how many frames we are going to get as input, so we have to build the system to be robust against the number of frames.
For the spritesheet, we are thinking of using 4×4 grid of frames. If number of frames go over 16, we will have to create a second sheet or truncate some frames. Let’s list of some options.
Option 1 — Truncate Frames
We can truncate any frames that go over 16 frame limit.Option 2 — Skip Every Other Frame
We can detect there are more than 16 frames and recursively half the frame rate until ≦ 16 frame is reached.Option 3 — Create Subsequent Sheet
We can create more sheets as necessary removing any number-of-frame cap.Option 4: Use Adaptive Grid Size
We can use 5×5 or 6×6 grids which will accommodate more number for frames. However, there is limit to the input image size in Flux Kontext Pro. So we will end up with scaled images.
Option 3 seems like the obvious choice since it has least impact on quality and removes the frame cap. However, we have to be really careful because the background removal process may introduce noises. Different run may exhibit subtilely different behavior. At the very least, we would probably want to fix the seed number for the runs so that the behavior is somewhat similar.
I think we should try out the Option 3 first then pivot if things don’t work out. Let’s work on that.
After few hours of hacking later…
Ehhh, got it working but now I ended up with this mess.
It all works as expected but to be honest I’m not sure if it is worth all the trouble here. Since Stage 3 has ability to limit the source duration so we could just limit the number of frames going into this flow…
Let’s clean this up a bit.
Okay, got it to working state. It is still quite complex and wish it it was much simpler to do this, but that’s what I have.
Here is a result for long webp file with backgrounds removed (i.e. transparent).
So far the artifact I was worried about doesn’t seem to appear noticeably.
Integrating into the Pipeline
Now, let’s see what it will take to integrate this into SpriteDX. At a glance, there isn’t any new custom nodes introduced here. So no new custom nodes to install.
However, now we will have to work with set of inputs rather than a single resource.
We again have some choices… My brain is little fried so I may be little bit rough here, but largely we have two options.
Repeat separate pipeline per input.
Pass multiple images into the workflow and have it handle multiple files on it own.
Doing 2 would make it nice and clean on the pipeline side but we will end up with lots of complexity on ComfyUI workflow side.
SpriteDX should play an orchestrator role here. We need a semantic that will help with repeating “workflow” per input.
I’m running out of time, and we shouldn’t rush here. So let’s continue in Part 2.
— Sprited Dev 🌱
