# Digital Being — Type 2 — State Expansion Today's focus will be to expand SpriteDX's capabilities to generate more diverse actions and states. Here is what we have now: * Reference Photo * Idle State * Greet Action * Run Cycle ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/4fd516f7-72a2-4309-bc88-d9949665d95e.png align="center") We'd like to add more states. Let's experiment with some states that can be created from idle state. First, we will take out the first frame of idle state. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/5858a17f-b5e2-4aa8-9c36-8524aed125dd.png align="center") ## Sitting Down State Let's pose the character. We are using Nano Banana Pro. > **Iteration 1:** 2d side scroller game character sprite is sitting down on imaginary flat grass tile (invisible) on pure white BG without shadows > > **Iteration 2:** 2d side scroller game character sprite is sitting down on imaginary flat grass tile (invisible) legs crossed and leaning back a little on pure white BG without shadows > > **Iteration 3**: 2d side scroller game character sprite is sitting down on imaginary flat grass tile (invisible) legs crossed and leaning back a little on pure white BG without shadows > > This is 2d side scroller platformer character. character's pelvis and the foot should be at same level. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/3ca905ad-6980-49be-ab57-a1a8bb30081b.png align="center") **Problem**: It is difficult to control the pose. **Analysis**: It would be nice if we can provide non-textual pose descriptions. Or environmental constraints like ground. Tried some to control the pose by giving it some visual cues. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/16aaab17-cee3-4efa-b19d-d3fde6837124.png align="center") ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/7d7da488-c3ac-4213-85e0-5b3ca9ddafa6.png align="center") Most of the time the character is deformed because the face position is locked in place. As an alternative, let's just use Seedance 1 Pro to let it simulate sitting down. > character sits down ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/6ddc8457-3ace-484a-8b07-86d80a2a7771.webp align="center") That's better. There is some zooming effect. Let's see if we can address it. > Character fully sits down on the ground and rests for 2 seconds then stands up again fast. > > #pure-white-background > > #game-sprite-animation ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/e9f90937-4243-4189-b4aa-85cdb8b38b9f.png align="center") ![](https://comfy.sprited.ai/api/view?filename=anim_00002_.webp&subfolder=&type=output&rand=0.5578107240934514 align="center") Our editing pipeline extracts the sit down animation. Blinking too fast but yeah. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/4e1c4c90-6b83-4a98-a625-a2b5575dd229.webp align="center") For sitting down animation, we probably want to hold the first or last frame. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/c18a4c4c-2ae6-42ba-8fbc-1bbccd9e8f10.webp align="center") Increased last frame's duration. **Analysis**: We lost the sitting down animation, but we achieved our goal of getting a Sitting down state. Let's not worry about the transition states. Let's assume in-betweening is a solved problem. * * * ## Walk Cycle Let's start with Seedance 1 Pro. > Character slowly strolls towards right (in-place) > > #white-bg #character-in-place #zoomless #no-camera-movement This produces character literally walking in place without moving much. Let's modified version of our current run cycle prompt. ```xml ``` Hmm, somehow it starts dancing then tries skipping LOL. Tried a few times similar results. Let's be direct and just try something more intuitive: > walk\_cycle.gif The character does not show walk cycle but just walks around. > walk\_cycle.gif > > character at the center of camera. > > shows walk cycle of game sprite. character is walkign towards right for 2s then stops. This worked some what. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/ab907d99-4143-46dc-b73f-3d2cbabb493a.webp align="center") Another sample after loop-detection. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/e6e114d3-8efd-40e1-af86-98a5e3a88bcf.webp align="center") Yield is not super great but at least we have something working! Now, let's move on to some other states. * * * ## Look At This one tests whether we can control the character to look up. > ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/678ec19f-70e3-4789-9497-3d7ebe7d6425.png align="center") > > character looks at red dot for a second, then looks at blue dot for a second then green then orange then goes back to looking straight > > #gaze #head-movement #zoomless #static-camera #full-body #white-bg Tried it with different durations. It works surprisingly well but the number of movements were limited to 3. That is, the character will look at red, then blue then green but skip orange one. When I increase the duration to 8, it does look at 4 different directions. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/8f32901c-652e-4f58-80cc-394638de8c88.webp align="center") But, it still doesn't seem ideal. It is difficult to generate exactly right sequence. Trying out a different prompt patterns. > shot1: Character is looking straight right shot2: Character is looking towards red dot shot3: Character is looking towards blue dot shot4: Character is looking towards green dot shot5: Character is looking towards orange dot shot6: Character is looking straight right again > > #gaze #head-movement #feet-fixed #zoomless #static-camera #full-body #white-bg The results are very random. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/7e9e7ce0-8d2f-41c2-8440-3f13b0504acd.webp align="center") Let's simplify this a bit. > character stays still while loooking upwards > > #static-camera #white-bg #game-sprite #gif ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/8c30fa84-0a5f-4e20-9d84-d06f4a50e93f.webp align="center") **Analysis**:Just asking the character to look at certain direction was not so successful. The pointers. Colored squares were useful but also led to lots of superfluous generations. **Proposal**: I think for `LookAt` state needs to be handled separately. Perhaps we can use different models. Let's try using NanoBananaPro. > Character is lookin up towards the red dot ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/4767e91c-9585-4580-98ba-614a1ad4afaa.png align="center") NBP did it in one shot. There isn't much of transition but at least we got a solution. Then we can cleanup those pointer dots and be done. Let's assume that for these states we will use NBP. * * * ## Emotion - Sadness Let's use Seedance 1 Pro. > Character is depressed. It is welling up. > > #static-camera #zoomless #full-body #gesture #game-asset #sprites ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/504ce0ad-8126-410e-84d3-5f4e2692f895.webp align="center") Not sure if we can use that as a state. Tried other prompts > Character is showing emotion "sadness" ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/21b8c1f4-e622-49b4-af14-2922de2e0ed3.webp align="center") Its dramatic but not sure if it really works for my case. We also need to blur out the canvas boundaries for something like this. > Character is sad and she squats in-place and crys her heart out. > > #cartoony #static-camera #zoomless #full-body #gesture #game-asset #sprites #white-bg ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/33ebd0e8-102d-45ad-8d4a-4dc1535014bf.webp align="center") Processed loop-cut version: ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/a6c6ac99-7682-45a5-a039-434ecd9db544.webp align="center") This works, but really like that warming animation. Perhaps, we need to detect the loop but also add in the initial transition. ```plaintext Transition In ------> Loop ------------> Transition Out ^ | ------- ``` Let's call this done even though we haven't really worked on reliability of generation. This imperfection is not necessarily a bad thing. * * * ## Emotion - Overjoy Same here, we are using Seedance 1 Pro. > Character is overjoyed she is jumping up and down super excited!!! > > #cartoony #static-camera #zoomless #full-body #gesture #game-asset #sprites #white-bg Using the default reference image gets me: ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/a3d0c7ba-3b55-46bb-a0f1-26e259ee1492.webp align="center") Works for me! We are doing another experiment by expanding the canvas to allow for more movement space. > Character is overjoyed she is jumping up and down super excited!!! > > #cartoony #static-camera #zoomless #full-body #gesture #game-asset #sprites #white-bg > > ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/042f07a7-4d8e-41cd-b669-797d6011d1a8.png align="center") In this mode, we are using 640x640 canvas. We get more movements. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/6757c5b9-9f1a-4f94-8538-7ef442cc118e.webp align="center") **Analysis**: We will have to somehow figure out how to situate the character in the right spot. Or, we can simply have all our generations be done in that extended canvas so that we don't have to worry about situating the characters. Let's call this state done. * * * ## State - Hunger > Character is super hungry. Character is squatting down, grabbing its stomach and begging for food. > > #cartoony #static-camera #zoomless #full-body #gesture #game-asset #sprites #white-bg Some trials ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/f8e6bf4c-a320-4e41-baf6-acc97c497b67.webp align="center") ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/1cc001d7-3f6c-461c-99bb-0d37966f280a.webp align="center") **Analysis**: Even though the action shows the hunger, it doesn't seem enough. We can augment these animations with a message bubble that shows meat or sandwiches... For now, let's leave it be. > Character is experssing hunger. She is pissed that you are not feeding her. She is sitting down and grabbing stomach. > > #cartoony #exaggerated #static-camera #zoomless #full-body #gesture #game-asset #sprites #white-bg ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/59a1d184-2d6f-4699-aea6-83a8ea24d3f8.webp align="center") Extracted Loop ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/ee126b37-6bc0-44c1-bbf8-e135cf567e2b.webp align="center") This seems a bit better. Let's call it at that. * * * ## Sleeping Character could be sleeping. > Character is super sleepy. She lays on her front and takes a nap at least for 4s. > > #cartoony #exaggerated #static-camera #zoomless #full-body #gesture #game-asset #sprites #white-bg In this particular result, we see style drift. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/ff473514-fef6-4084-a47c-710af786ba4e.webp align="center") Let's try out few more samples. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/21b0a2dd-efbe-4ca7-b217-4951957bfbf7.webp align="center") This one was fine. Extracted loop: ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/51d78089-e74b-4f75-84a1-85d75d29f718.webp align="center") One more sample: Overly dramatic sample. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/fbc80c4c-8554-4842-bdcb-c713647e2713.webp align="center") Let's call this state done. * * * ## Squashed This state happens when character is squashed by the blocks. This one is probably hard. But let's try this. We start with Nano Banana Pro. > Character is squashed by two imaginary large square blocks left and right. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/c3608ac9-dec9-4b66-b114-95d379d4a78e.png align="center") Results weren't good. I think we need some way to show the model what these blocks are. > Character is compressed left and right. We want dramatic picture of character getting squashed in-place without any props. > > Keep the proportions the same and design and style the same > > #white-bg ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/cfe49188-f7d1-4b4d-9068-9bf49ba8c026.png align="center") I think we give up on the squashed state, and instead work on "pain" state. * * * ## State: Getting Hit Let's try Nano Banana Pro. > Game Sprite State: Getting Hit > > #white-bg #no-special-effect #just-character #sprite-asset ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/a1128c55-8853-429f-a720-75c02f330a57.png align="center") ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/fffdc19e-fe8a-44d3-a9d0-784c37db4a12.png align="center") ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/355bdc44-2265-4d96-9b01-e13bd3a9e396.png align="center") Relatively, reliably, we are able to generate the getting hit state. * * * ## Ready-To-Fight > Character enters fighting stance (ready-to-fight) stance bare handed for 3s. > > #cartoony #exaggerated #static-camera #zoomless #full-body #gesture #game-asset #sprites #white-bg ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/3f66ce18-f9cf-4618-b770-b1bd21bbe6b7.webp align="center") ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/4c44cebf-e538-4809-a579-12b03a48f646.webp align="center") Second one seems promising. This state is good because we can not start from ready to fight frame. Let's also try NBP. > Game Sprite State: REady-to-Fight pose. > > Keep proportions and style the same #white-bg #no-special-effect #just-character #sprite-asset NBP's weakpoint is that when we generate something similar to the original image, it is often not able to translate parts of the body. For example when trying to get a character sitting down animation, it just elongates torso and have the legs sit down. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/fbd4c653-03e1-4c57-9f96-8e3ed7b8887a.png align="center") If you look at this example, the character head becomes fixed in space. So the body becomes somehow bigger to acommodate. I think we need to use Seedance 1 Pro instead and pick out the right frame. * * * ## Punching > SHOT 1: Character enters fighting stance (ready-to-fight) stance > > SHOT 2: Character throws punches > > SHOT 3: Character goes back to original state > > #cartoony #exaggerated #static-camera #zoomless #full-body #gesture #game-asset #sprites #white-bg ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/136dffc0-9ca7-4730-bead-b2df82ccb482.webp align="center") Well not working all the time of course. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/89fd3fc8-88ba-46d7-8be3-801bd617aa2b.webp align="center") **Issue**: We would want to figure out a way to detect this incorrect generations and throw them out. I think we can do video to latent mapping then we can try to match up with the reference latents to compare the similarity and throw out distant ones. * * * ## Kicking > SHOT 1: Character enters fighting stance (ready-to-fight) stance > > SHOT 2: Character does high kick 2 times (towards right) > > SHOT 3: Character goes back to original state > > #cartoony #exaggerated #static-camera #zoomless #full-body #gesture #game-asset #sprites #white-bg #effectless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/e8b3ef0b-7914-4d88-9a95-fde4b4ef8d7d.webp align="center") It's not easy to control but we can get some clips like these ones. * * * ## Pushing For characters to impact the world, we need to support things like Pushing and Pulling. Let's see what we can do here. This is probably one of the harder ones to achieve. > Character pushes large block towards the right. > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg > > ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/1813d356-64e8-46ae-9796-f9ca9cfc773c.png align="center") Trials ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/2f5d59d5-a74c-4965-b7f6-a1787201cd23.webp align="center") So, somewhat works with lots of errors of course. Also we need to learn how to remove the green box. Let's try multi-shot prompts. > SHOT1: Charater approaches the large green block > > SHOT2: Character pushes the block 2 steps to the right > > SHOT3: Character pull the block back to where it was. > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/0026e5d1-835d-4e23-a0c8-d29b10ec8324.webp align="center") Anaysis: I think for these actions, we would benefit from providing motion keys. We could try Seedance 2 Pro. An alternative is to use kinda "magic" spell. > Character stands and uses magic to push the block away from her. > > Then she does another magic to pull the block back to original position > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/9eb6e61e-b767-4bb1-a63c-a9edba968256.webp align="center") Not what I was looking for. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/b8e1b493-d44f-4c1f-8b81-926513e2d45e.webp align="center") Still no. I think we need to find out some other models that can take in skelal image and translate that to animation. Let's give up on this for now. * * * ## Laughing Out Loud > Character laughs out loud. > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/f4fd6dd2-118c-4a11-89c8-ceaf691dae79.webp align="center") Laughing out loud seems to be one of the easier ones to handle. Let's call it done. * * * ## Crafting In the world of Machi, characters need to be able to create statues and stuff. So, let's add animation state for "crafting." > Character is crafting something > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/296ff8aa-6071-45eb-8297-cb6c899c4261.webp align="center") > Character is crafting something out of thin air > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/ff845eea-3536-4a2f-8d89-1e72fa6bb3ca.webp align="center") > Character is crafting something. Weaving it then throwing it to manifest. > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/3da355c0-3727-4f22-ba5e-1e4776cb4be4.webp align="center") > Character is crafting something. Creating a mold then molding it into something then throwing it to manifest. > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/6306d90a-3bc1-44a2-bd36-ac5a2049e62e.webp align="center") > Character is acting as if it is doing magic > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/c2d1a424-4ee7-491f-93bd-fc11d19dd6f6.webp align="center") Okay, may be this will do. Not sure how to control this kinda thing but let's worry about it later. * * * ## Death Not sure if we will have death in Machi but let's model it regardless. > Character death sequence > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/671a88ab-fa3b-48fb-a242-5a1baeb66190.webp align="center") > Character death sequence and disappears > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/e8d8256f-9113-4a70-9c94-899c9cd86072.webp align="center") Yeah, not sure. Little hard. > Character death sprite-animation sequence > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless I think death can be more of special effect on the freeze frame than animation. Or we can try to freeze the character so that it becomes immobile until external help arrives. > Character freezes into a solid crystal block. > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/d2fa90b5-b6fb-4d5a-9244-71ea978bc952.webp align="center") Not what I had in mind. ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/b7d92e1d-92f1-464f-bce8-f5f44e260f0f.webp align="center") > Character INSTANTLY freezes into a solid crystal block. Then shatters. > > #no-zoom #camara-fixed #sprite-asset #game-asset #white-bg #shadowless ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/5f1c5b43-a632-4001-9d05-19579c839b39.webp align="center") Also trying NBP: > Character frozen in crystal state. It is when character dies it gets frozen into a block. > > Keep proportions and style the same #white-bg #no-special-effect #just-character #sprite-asset ![](https://cdn.hashnode.com/uploads/covers/682665f051e3d254b7cd5062/21680c5f-d538-47a4-a6fa-8b3eded7b083.png align="center") We could use one of these states as immobilized state which could work. I think I like the NBP solution better here. Or again, just add some special effect or sprite overlay to show that it is immobilized is enough. * * * ## Summary We experimented with expanding the state. Yield is not great but still about 50-70% of the cases we were able to find a good solution. I think we should automate and solidify some of these generations and put them into SpriteDX. **Next Steps**: * I want to experiment with Seedance 2 Pro. First get access to it and try to provide 3D capsule man video then ask the model to produce animation. * Alternatively, I can start compiling all these animations into states. It will be done manually first, then we will figure out a way to automatically compile them in SpriteDX. * We also kinda need a way to extract a rig out of the videos so that we can position the characters. It would help us build a correct hitbox as well. * The best thing though would be to have a real-time model that can skin the stick figure that I provide. That would be the ideal situation. 🐛 Sprite Dev