# Sprite Animation Generation - Background Removal

To summarize this week’s progress: we’re knee-deep in building **SpriteDX**, a Unity editor plugin that generates fully animated characters from simple text prompts. Next up: crafting demo sprites that really show off what this baby can do.

**Previous Posts**

* [Generating Demo Sprites for SpriteDX](https://blog.sprited.app/dev-log-generating-demo-sprites-for-spritedx)
    
* [Sprite Animation Generation: Pixel Grid Misalignment Problem](https://blog.sprited.app/sprite-animation-generation-pixel-alignment-problem)
    

We faced two problems—**Pixel Grid Misalignment** and **Background Removal Imperfection**.

The **Pixel Grid Misalignment** issue is a bit of a subjective snag. Our “off-grid“ pixels still look like pixel art—but they don’t snap to the grid perfectly, which can trip up any manual touch-ups.

On the flip side, those tiny sub-pixel offsets let us draw eye glints and fine details that feel alive. If we force everything onto the native grid, you either get glaringly bright pixels or lose those sparks altogether.

Now, pixel art purist would be angered to see a sprite generation tool that doesn’t generate “true“ pixel arts, so yeah, we need to think through this further and consider training a model that will digitize this into native pixel resolutions as discussed [here](https://blog.sprited.app/sprite-animation-generation-pixel-alignment-problem#heading-possible-solution).

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1753297759404/b59ab400-013c-45e6-b4a6-d68679cdd76b.png align="center")

IMO, **Background Removal Imperfection** is a more immediate issue. State of the art video generation models like Seedance 1 Pro does not support transparency and we need an automated way to make the background transparent.

---

## Inspyrenet

We tried `Inspyrenet` custom node in Comfy, but they were still not able to capture enclosed areas properly.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1753297448521/ec4fcb75-58dc-47b6-afaa-e8a5b67fb440.png align="center")

---

## Segment Anything

Segment Anything was also not able to capture this triangle region.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1753298648865/2f838b5e-20df-4116-8ced-5aa7a1a81f0e.png align="center")

---

## Segment Anything 2 Video

Version 2 takes in the whole video and is quite reliable but still fails on the triangle region below the chin.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1753305348956/34d0a992-fd86-473b-a67c-4559e06e2208.png align="center")

---

## Robust Video Matting

RVM looked promising for webcam dataset but did not perform very well on the sprite dataset we had.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1753299600015/60b85146-eaaf-4ae4-8cf5-10ff8a35be23.png align="center")

---

## BRIAAI Matting

BRIAAI Matting was much better on most of the frames, but didn’t work so well in some others.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1753300244674/09300be6-8904-4b7c-85af-021b814150f7.png align="center")

---

## Color Keying + SAM2

One of the potential issue is that we are using white background on a character wearing white costume. I’m trying a color key approach.

So, I regenerated the video after coloring the background purple `#FF00FF`.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1753304022837/4fb9e5db-56f7-49f5-9427-209315821538.png align="center")

BRIAAI node still failed right below the arm pit.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1753305825780/3a2bbb24-c6d3-4edb-a864-87f153076cb7.png align="center")

But, SAM2 was able to produce correct segmentation even in enclosed zones.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1753305739474/09f15b87-32a0-4fe8-ba9e-6968359cb96a.png align="center")

Licensing wise, SAM2 is Apache-2.0 and is available for commercial use.

However, one downside is that it seems to create segmentations with rounded corners which are detrimental to pixel arts.

Either we will want to post process this mask or find alternative models.

---

## Conclusion

I couldn’t find one perfect model for this. Color-key + SAM2 video was the best option so far but it is not perfect. For the time being, we will choose best model from here and use it for the SpriteDX prototype.

—Sprited Dev 🌱