SpriteDX - Cache
So, we implemented SpriteDX Pipeline for character generation in JSON schema format. However, we are noticing significant time is used to test these flows because every time we click on “Generate,” it will try to generate freshly even though the parameters hasn’t changed.
To not run same inferencing twice on same input, the proposal is to implement a persistent cache.
Approaches
1. Post Request Caching
Here is what I’m thinking. We will build a thin layer on top of OPFS that does POST request caching. This will work similarly to GET request caching in browsers and will honor the HTTP Cache headers. Outdated cache will then be automatically garbage collected in the background when the app runs next time.
The idea is that we will build a library that will work for any type of requests. Then create a function cachedFetch() which will first look for the cached response in OPFS then short-circuit it.
There is bunch of limitation to this design but if we can implement this, we can use it for many different types of requests without worrying about cache architecture design. Everything should just work.
2. Pipeline Cache
Alternatively, we can build a custom caching solution for pipeline stage runs. In this case we will be building something that is specific to SpriteDX.
3. Continue from generation/run_xxxx
SpriteDX stages generate output files and put them into generation/run_xxxx folder. We can check if the output is already generated and skip the stage if that file is found.
The complexity here is that now we have to save the generation parameters in the run_xxxx folder and make sure no change has been made to those parameters.
Also, the cache will only be able to look up the information from latest generation rather than from library of all other generations using a cache key.
For the reproducibility aspect of the product, having all generation parameters and outputs is eventually going to be requirement, so storing parameters is going to actually be a good future proofing.
What I implemented
Ended up going with cachedFetch approach with a good amount of vive coding. It was easy to describe the problem to agent and be unit tested so went with it.
It’s working pretty well, but the RunPod Serverless is still flakey. It still hangs time to time.
So, next step would be to replicate the RunPod setup locally so that it can run ComfyUI locally instead of running it on the RunPod instance.
— Sprited Dev
