Post Not Found

Yesterday, we resolved shot boundary detection and today I want to look into Loop Segment Detection.

Here is our current approach:

Given:
1. Frames: f_1 … f_N
2. min_len: minimum number of frames.
3. max_len: maximum number of frames.
Grayscale each of them and down scale to 128×128.
Create composite frame composite_i such that:
R: Previous frame’s intensity^.
G: Current frame’s intensity.
B: Next frame’s intensity.
For each (i, j) pair of frames (within min_len, max_len):
1. Get mean squared error between the pixel values.
Pick the pair that has the lowest error, and return the segment [i, j).

This worked reasonably well, but we faced some issues:

Temporal Cheats: When min_len is small, frames close to each other often has the least error just because they are close to each other temporally.
Metric Mismatch: When picking pair A vs B, if A has less motion, there is higher likelihood that it would have lower MSE even though visual inspection reveals that higher MSE with more motion often looks smoother.
Repetitions: When there is repetitions within a loop, there is no mechanism to find the smallest loop without repetition.

We tried addressing these by hand:

To fight Temporal Cheats and Repetitions, we added length penalty.
To fight Metric Mismatch: we adjusted error term using motion metric.

However, a lot of this was tuned by hand looking at the data and really can’t tell if they are working or not.

Dataset

We are going to use the dataset we collected before. This contains character animations broken down into individual shots.

Dataset Size: 744
Each samples are labeled using the baseline approach discussed above.

Next, we need UI to review the labels and adjust them.

Did some manual review of the loops. And from the looks of it. The loops generated seems rather high quality. The failures were mostly focused around bad input data.

Input data does not have loop
Longer loop sequence is discouraged (i.e. anything over 16 frames gets a penalty)
Sometimes non-interesting loop segment is detected.

I think I will settle with current baseline for now.