Runway AI's Gen-4: How Can AI Montage Go Beyond Absurdity

The film industry, alongside all creative sectors, from digital marketing to social media, stands at a technological crossroad. As artificial intelligence begins to reshape every aspect of visual storytelling and change the landscape of entertainment and digital commerce, we must assess its potentials and pitfalls.

The AI Gold Rush in Filmmaking

Major production companies are rapidly adopting AI video tools. Fabula, the acclaimed studio behind Oscar-winning A Fantastic Woman and biopic Spencer, just announced a partnership with Runway AI to integrate AI across its production pipeline. Lionsgate signed a deal with Runway last fall to explore AI-powered filmmaking. Experimental directors like Harmony Korine have already debuted AI-assisted film at Venice last year.

The broad applications of AI videos are already impressive, from pre-visualizing scenes for Amazon’s House of David to creating advertisement for Puma. Yet beneath these flashy demonstrations lies a more fundamental question: can AI-generated content evolve beyond technical spectacle to deliver truly meaningful stories?

The Spectacle vs. Substance Dilemma

Runway’s Gen-4 represents significant progress in several areas: character consistency, scene coherence, and visual fidelity. An example Runway AI releases show two main characters stay consistent across different shots ranging from walking, running, petting a cow, lighting up a match, and maintain fidelity of the look of a steppe in gloomy weather.

Yet these technical improvements don’t address the core challenge: AI excels at generating individual moments but struggles with coherent and sustained storytelling. While it can create a stunning shot of giraffes and lions roam in the New York City, can it make audiences care about a city turned into a zoo?

AI videos risk repeating the early mistakes of Computer Generated Imagery (CGI), prioritizing visual gimmicks over in-depth messages. As barriers to creative production and film making disappear, we may face a flood of visually polished but emotionally hollow contents, derivative works optimized for algorithmic efficiency, or compelling synthetic media that lacks human touch. While AI videos can wow first-time viewers, can they make audiences want to watch them more? Can AI films ever produce classic pieces that draw generations of movie-goers?

The Fragmented Reality of AI Videos

Current multi modal AI technologies center on innovations in film, media, and video games. A recent project spearheaded by researchers from Nvidia, Stanford and UCSD uses Test-Time-Training layers in machine learning models to generate 60-seconds animations of Tom and Jerry. To achieve this, the team trained the model on 81 cartoon footages between 1940 and 1948, which add up to about 7 hours. The model generates and connects multiple 3-second segments, each guided by storyboard annotating plots, settings, and camera movements. The technique highlights significant potential to scale video productions and animation series creations.

But the technology also reveal critical flaws that persist among AI video generators such as Sora, Kling, Runway, Pika, etc. One limitation is continuity error. For example, rooms, landscapes, and lighting shift unnaturally between 3-second segments. Physics defiance is another problem. For instance, in one of the earlier mentioned Tom and Jerry AI videos, Tom devours a pie that changes its color, shape, and texture at segment boundaries. Another issue is narrative disjointedness. As the segmentation of content is necessary for algorithms to effectively learn the contents, understand the prompt, and accurately generate videos, AI models struggle to show logical scene progression.

These traces of what I call AI montage also appear in Runway AI’s videos, the elephants walking across the Time Square is abruptly followed by a scene of a cheetah running across a bridge. One is set in cloudy weather while the next in a sunny day. The changes do not push the storyline forward nor do they convey any logic. The absurd, the fragmented, and the incongruous, are what AI video generators currently good at producing. For now, AI struggles to replicate the coherence of even a 5-minute cartoon, let alone a feature film.

AI Video’s Capacity for Reflection and Critique

AI-generated videos show strength as a medium for critiquing both itself and the societies that produce it. Director Jia Zhangke’s recent AI film made using Kling AI imagines a future run by robotic caretakers. The film provokes audiences to reflect on the crisis of aging populations, societal neglect, and the erosion of empathy in an era of breakneck competition, capitalism, and exploding automated technologies.

Jia’s film show robot companions taking the elderly for a walk or helping them harvest crops, in lieu of real sons and daughters. Such a theme is grounded in societal challenges today. The film critiques the substitution of human connection with automated machines and transactional relationships, and raises the concern over relentless stress and long hours in workplaces.

Just as Charlie Chaplin used industrialization-era tools to critique industrialization in Modern Times, today’s filmmakers can use AI to critique the conditions of its own existence. Consider how synthetic news anchors might expose media manipulation, or endlessly combinable streaming content could comment on algorithmic culture.

The Path Forward: Imbuing AI with Humane Stories

Just like science fictions that critique environmental disasters, human greed, and inequality, the most compelling AI films will likely be those that embrace their own artificiality to engage with real social problems.

Rather than fearing obsolescence, filmmakers might focus more intensely on what machines cannot replicate: the nuance of human emotions, complexities of human nature, the weight of lived experience, and the cultural resonance of authentic storytelling.

History suggests that film and media have always adapted to technological upheaval, from silent to sound, black-and-white to color, celluloid to digital, each time emerging with new creative possibilities. The question is no longer whether AI will change filmmaking, but how filmmakers will harness it to tell stories that matter.

The above is the detailed content of Runway AI's Gen-4: How Can AI Montage Go Beyond Absurdity. For more information, please follow other related articles on the PHP Chinese website!