A brand new paper out this week at Arxiv addresses a difficulty which anybody who has adopted the Hunyuan Video or Wan 2.1 AI video turbines can have come throughout by now: temporal aberrations, the place the generative course of tends to abruptly velocity up, conflate, omit, or in any other case mess up essential moments in a generated video:
Click to play. Some of the temporal glitches which might be changing into acquainted to customers of the brand new wave of generative video techniques, highlighted within the new paper. To the appropriate, the ameliorating impact of the brand new FluxFlow method. Source: https://haroldchen19.github.io/FluxFlow/
The video above options excerpts from instance check movies on the (be warned: moderately chaotic) venture web site for the paper. We can see a number of more and more acquainted points being remediated by the authors’ methodology (pictured on the appropriate within the video), which is successfully a dataset preprocessing approach relevant to any generative video structure.
In the primary instance, that includes ‘two youngsters enjoying with a ball’, generated by CogVideoX, we see (on the left within the compilation video above and within the particular instance under) that the native technology quickly jumps by means of a number of important micro-movements, dashing the kids’s exercise as much as a ‘cartoon’ pitch. By distinction, the identical dataset and methodology yield higher outcomes with the brand new preprocessing approach, dubbed FluxFlow (to the appropriate of the picture in video under):
Click to play.
In the second instance (utilizing NOVA-0.6B) we see {that a} central movement involving a cat has not directly been corrupted or considerably under-sampled on the coaching stage, to the purpose that the generative system turns into ‘paralyzed’ and is unable to make the topic transfer:
Click to play.
This syndrome, the place the movement or topic will get ‘caught’, is without doubt one of the most frequently-reported bugbears of HV and Wan, within the varied picture and video synthesis teams.
Some of those issues are associated to video captioning points within the supply dataset, which we took a have a look at this week; however the authors of the brand new work focus their efforts on the temporal qualities of the coaching knowledge as an alternative, and make a convincing argument that addressing the challenges from that perspective can yield helpful outcomes.
As talked about within the earlier article about video captioning, sure sports activities are notably tough to distil into key moments, which means that vital occasions (reminiscent of a slam-dunk) don’t get the eye they want at coaching time:
Click to play.
In the above instance, the generative system doesn’t know learn how to get to the subsequent stage of motion, and transits illogically from one pose to the subsequent, altering the angle and geometry of the participant within the course of.
These are giant actions that bought misplaced in coaching – however equally susceptible are far smaller however pivotal actions, such because the flapping of a butterfly’s wings:
Click to play.
Unlike the slam-dunk, the flapping of the wings will not be a ‘uncommon’ however moderately a persistent and monotonous occasion. However, its consistency is misplaced within the sampling course of, for the reason that motion is so fast that it is rather tough to ascertain temporally.
These are usually not notably new points, however they’re receiving better consideration now that highly effective generative video fashions can be found to fans for native set up and free technology.
The communities at Reddit and Discord have initially handled these points as ‘user-related’. This is an comprehensible presumption, for the reason that techniques in query are very new and minimally documented. Therefore varied pundits have prompt various (and never all the time efficient) treatments for a few of the glitches documented right here, reminiscent of altering the settings in varied elements of various kinds of ComfyUI workflows for Hunyuan Video (HV) and Wan 2.1.
In some circumstances, moderately than producing fast movement, each HV and Wan will produce gradual movement. Suggestions from Reddit and ChatGPT (which principally leverages Reddit) embody altering the variety of frames within the requested technology, or radically reducing the body charge*.
This is all determined stuff; the rising fact is that we do not but know the precise trigger or the precise treatment for these points; clearly, tormenting the technology settings to work round them (notably when this degrades output high quality, for example with a too-low fps charge) is simply a short-stop, and it is good to see that the analysis scene is addressing rising points this rapidly.
So, in addition to this week’s have a look at how captioning impacts coaching, let’s check out the brand new paper about temporal regularization, and what enhancements it’d supply the present generative video scene.
The central thought is moderately easy and slight, and none the more serious for that; nonetheless the paper is considerably padded with a view to attain the prescribed eight pages, and we are going to skip over this padding as vital.
The fish within the native technology of the VideoCrafter framework is static, whereas the FluxFlow-altered model captures the requisite modifications. Source: https://arxiv.org/pdf/2503.15417
The new work is titled Temporal Regularization Makes Your Video Generator Stronger, and comes from eight researchers throughout Everlyn AI, Hong Kong University of Science and Technology (HKUST), the University of Central Florida (UCF), and The University of Hong Kong (HKU).
(on the time of writing, there are some points with the paper’s accompanying venture web site)
FluxFlow
The central thought behind FluxFlow, the authors’ new pre-training schema, is to beat the widespread issues flickering and temporal inconsistency by shuffling blocks and teams of blocks within the temporal body orders because the supply knowledge is uncovered to the coaching course of:
The central thought behind FluxFlow is to maneuver blocks and teams of blocks into surprising and non-temporal positions, as a type of knowledge augmentation.
The paper explains:
‘[Artifacts] stem from a basic limitation: regardless of leveraging large-scale datasets, present fashions usually depend on simplified temporal patterns within the coaching knowledge (e.g., fastened strolling instructions or repetitive body transitions) moderately than studying various and believable temporal dynamics.
‘This challenge is additional exacerbated by the shortage of specific temporal augmentation throughout coaching, leaving fashions vulnerable to overfitting to spurious temporal correlations (e.g., “frame #5 must follow #4”) moderately than generalizing throughout various movement situations.’
Most video technology fashions, the authors clarify, nonetheless borrow too closely from picture synthesis, specializing in spatial constancy whereas largely ignoring the temporal axis. Though methods reminiscent of cropping, flipping, and coloration jittering have helped enhance static picture high quality, they don’t seem to be satisfactory options when utilized to movies, the place the phantasm of movement relies on constant transitions throughout frames.
The ensuing issues embody flickering textures, jarring cuts between frames, and repetitive or overly simplistic movement patterns.
Click to play.
The paper argues that although some fashions – together with Stable Video Diffusion and LlamaGen – compensate with more and more advanced architectures or engineered constraints, these come at a value by way of compute and adaptability.
Since temporal knowledge augmentation has already confirmed helpful in video understanding duties (in frameworks reminiscent of FineCliper, SeFAR and SVFormer) it’s shocking, the authors assert, that this tactic is never utilized in a generative context.
Disruptive Behavior
The researchers contend that straightforward, structured disruptions in temporal order throughout coaching assist fashions generalize higher to sensible, various movement:
‘By coaching on disordered sequences, the generator learns to get well believable trajectories, successfully regularizing temporal entropy. FLUXFLOW bridges the hole between discriminative and generative temporal augmentation, providing a plug-and-play enhancement answer for temporally believable video technology whereas bettering general [quality].
‘Unlike present strategies that introduce architectural modifications or depend on post-processing, FLUXFLOW operates straight on the knowledge degree, introducing managed temporal perturbations throughout coaching.’
Click to play.
Frame-level perturbations, the authors state, introduce fine-grained disruptions inside a sequence. This sort of disruption will not be dissimilar to masking augmentation, the place sections of information are randomly blocked out, to forestall the system overfitting on knowledge factors, and inspiring higher generalization.
Tests
Though the central thought right here does not run to a full-length paper, as a consequence of its simplicity, nonetheless there’s a check part that we are able to check out.
The authors examined for 4 queries regarding improved temporal high quality whereas sustaining spatial constancy; potential to study movement/optical circulate dynamics; sustaining temporal high quality in extraterm technology; and sensitivity to key hyperparameters.
The researchers utilized FluxFlow to 3 generative architectures: U-Net-based, within the type of VideoCrafter2; DiT-based, within the type of CogVideoX-2B; and AR-based, within the type of NOVA-0.6B.
For honest comparability, they fine-tuned the architectures’ base fashions with FluxFlow as a further coaching part, for one epoch, on the OpenVidHD-0.4M dataset.
The fashions had been evaluated towards two common benchmarks: UCF-101; and VBench.
For UCF, the Fréchet Video Distance (FVD) and Inception Score (IS) metrics had been used. For VBench, the researchers focused on temporal high quality, frame-wise high quality, and general high quality.
Quantitative preliminary Evaluation of FluxFlow-Frame. “+ Original” signifies coaching with out FLUXFLOW, whereas “+ Num × 1” exhibits totally different FluxFlow-Frame configurations. Best outcomes are shaded; second-best are underlined for every mannequin.
Commenting on these outcomes, the authors state:
‘Both FLUXFLOW-FRAME and FLUXFLOW-BLOCK considerably enhance temporal high quality, as evidenced by the metrics in Tabs. 1, 2 (i.e., FVD, Subject, Flicker, Motion, and Dynamic) and qualitative ends in [image below].
‘For occasion, the movement of the drifting automotive in VC2, the cat chasing its tail in NOVA, and the surfer driving a wave in CVX turn out to be noticeably extra fluid with FLUXFLOW. Importantly, these temporal enhancements are achieved with out sacrificing spatial constancy, as evidenced by the sharp particulars of water splashes, smoke trails, and wave textures, together with spatial and general constancy metrics.’
Below we see choices from the qualitative outcomes the authors discuss with (please see the unique paper for full outcomes and higher decision):
Selections from the qualitative outcomes.
The paper means that whereas each frame-level and block-level perturbations improve temporal high quality, frame-level strategies are inclined to carry out higher. This is attributed to their finer granularity, which permits extra exact temporal changes. Block-level perturbations, against this, could introduce noise as a consequence of tightly coupled spatial and temporal patterns inside blocks, lowering their effectiveness.
Conclusion
This paper, together with the Bytedance-Tsinghua captioning collaboration launched this week, has made it clear to me that the obvious shortcomings within the new technology of generative video fashions could not consequence from person error, institutional missteps, or funding limitations, however moderately from a analysis focus that has understandably prioritized extra pressing challenges, reminiscent of temporal coherence and consistency, over these lesser considerations.
Until not too long ago, the outcomes from freely-available and downloadable generative video techniques had been so compromised that no nice locus of effort emerged from the fanatic group to redress the problems (not least as a result of the problems had been basic and never trivially solvable).
Now that we’re a lot nearer to the long-predicted age of purely AI-generated photorealistic video output, it is clear that each the analysis and informal communities are taking a deeper and extra productive curiosity in resolving remaining points; with a bit of luck, these are usually not intractable obstacles.
* Wan’s native body charge is a paltry 16fps, and in response to my very own points, I word that boards have prompt reducing the body charge as little as 12fps, after which utilizing FlowFrames or different AI-based re-flowing techniques to interpolate the gaps between such a sparse variety of frames.
First revealed Friday, March 21, 2025