SpriteDX - Error Recovery

I’m working on error detection and error recovery today.

ComfyUI Custom Node

First, in the following workflow, I created “Pixel Stats“ custom node which will first get a masked region and measure mean and standard deviation of pixel values in that region.

The mask we are creating is only going to select 1% inner border around the image and collect pixel values for computing mean and standard deviations.

We get single mean and standard deviation per frame then get maximum mean and standard deviation across the frames independently.

On the bottom right of the workflow, we have max_std of 0.249 which reflects that there was rather high pixel value differences indicating that around the image border, there were some pixel value discrepancy, and min_mean of 0.88 which is small enough to be able to tell that some of the areas in the image were not a white background.

Criteria

Given these two numbers, we will use the following rule:

• max_std < 0.1 (which enforces uniform pixel values around the border)

• and min_mean > 0.9 (which enforces white as the background color)

Then, we updated worker-comfyui to return emitted values back.

Confidence Value

In practice, I want a single confidence score, so let’s compute that number by:

w (1.0 - max_std) + (1-w) (min_mean)
where w is configurable (default 0.7)

We are giving std more weight because generated background may not be completely white.

For this problematic generation (i.e. the character gets cut off), the confidence value comes out to be 0.79.

And for successful generation, we have confidence value of 0.99.

So, yes, so far we have a detector that is working as expected.

Mapping outputs

Now, we need to map outputs to output nodes in SpriteDX system.

Rerunning Stage

Next, we need a way to rerun the stage when confidence value does not meet the criteria. To do this, I’m adding following to pipeline.json stage definition.

"retryPolicy": {
  "maxAttempts": 3,
  "criteria": { "confidence": { ">=": 0.9 } },
  "onFailure": "proceed"
}
"outputs": {
  … 
  "confidence": { "type": "number", "mapTo": "57.value"}
}

It basically defines a gate confidence >= 0.9 and maximum attempts.

---

So far, things are working as expected but there are little more errors to handle like:

• Animation getting split into less than or more than 3 shots: current video shot splitter node does not have any pre-knowledge of k=3, and it just splits them into multiple shots and we have no guarantee on how many shots. Some solutions were proposed, but we didn’t get too far. I will look more into it.

Let me stop here today.

— Sprited Dev 🌱