How LAPSE works
Read the full report (PDF, 46 pages)
Abstract
LAPSE moves a subtitle file onto the speech in a video. It corrects a constant offset, the linear drift that a framerate conversion leaves behind, and the piecewise offsets of recuts, ad breaks and files that hold two parts, and it does this without reading the language of either side.
Speech is detected with Silero VAD, or libfvad when the neural model is not available, and turned into weighted spans. Offsets are found with a two channel FFT cross correlation on speech blocks and speech onsets, and both channels have a running mean removed before they are correlated. The report shows that the step is equivalent to high pass filtering the score curve, which takes out the broad hill that dense stretches of dialogue raise in a plain overlap score. Candidates are refined on an exact overlap score and ranked by a lock statistic that compares each offset with the same file at random offsets, over the whole file and in eight slices. For files that need splitting, a penalised dynamic programme runs on a coarse grid, with a penalty that grows with the logarithm of the number of cues. A decision procedure picks shift, drift, recut, joined parts or restart for each file, and a verdict decides whether the original subtitle is overwritten.
On 39 feature films chosen for being hard to synchronize, LAPSE 2.0.0 passed 36 in the shift and drift test, against 31 for alass 2.0.0 and 33 for ffsubsync 0.5.1. With splitting forced on it passed 32 with 3 partial. Paired exact tests put these differences in perspective: only the gap to ffsubsync's split mode (p = 0.0005) is clear on this sample. On the 27 films where verdicts were recorded, all 22 answers the engine marked solid in the shift and drift test were right, and both of its wrong answers were held back. Automatic split detection is the weak part: it passed 19 films against 32 when splitting was forced, and its verdict does not look at the split.
Contents
- Introduction
- Background and related work
- Formal model
- Architecture and libraries
- Speech detection
- Searching for one offset
- Refinement and the lock statistic
- Drift
- Split alignment
- Choosing the correction and judging it
- A worked example
- Evaluation
- Development history
- Threats to validity and limitations
- Conclusion and future work
Appendices: constants, per film results, and running the engine.
Read the full report (PDF, 46 pages)