As summer 2024 approaches, the AI gold rush has hit fever pitch. As I sit in a small coffee shop in Midtown Manhattan, typing away on my iPad, I overhear two colleagues discussing AI’s impact on their finance jobs, and the possibilities it will have on their future.

The flood of AI products and AI-driven projects entering the market in early 2024 has caused a lot of curiosity, confusion, and misconceptions.

One particular area that seems to be on the minds of creatives from New York to Hollywood is AI-generated movies. In fact, it caused enough confusion and anxiety to bring the entertainment industry to a halt during last year’s SAG-AFTRA and WGA negotiations.

Despite what seems to be a daily release of new products promising to take storytellers to the next level, most of what’s been delivered doesn’t seem to resonate with customers. This begs the question: why is AI failing our sky-high expectations? Did we buy into the hype, or was the future overpromised?

The Iteration Problem

Until now, we’ve mostly seen the ability to capture a text prompt and generate short clips or sequences of short stills or clips. While this has yielded some incredible results — things that were unimaginable as recent as 6 months ago — making tweaks to execute the user’s vision could require heavy manipulation of the output.

This requires storytellers to spend more time identifying and correcting the AI’s misses — converting skilled writers into story editors and making them, arguably, less efficient.

“Making a movie is all about iterating. It’s iteration. And if you can’t iterate on one of these, I don’t know how you would possibly use it in production,” says Craig Good, a former Pixar animator who has credits on Toy Story and Finding Nemo, in a recent interview with Quartz.

I interpret Craig’s quote as an issue with AI accurately translating his vision into a usable output. This could mean that he’s either had issues prompting the tool he’s using or, more likely, the technology hasn’t been able to recreate his intended vision.

With Good’s professional accreditations, I have to believe his writing ability puts him in a very small circle of world-class creatives. Making this assumption, I’m led to believe the AI doesn’t understand the nuance of human creativity.

But in reality, AI can often read and comprehend better than humans. So how does this combination lead to such an efficiency loss?

**The Storyteller’s Dilemma **

Many of the products rushing to market today are outputting high-res, short-form content based on the capabilities of AI.

The result has been a saturation of products that largely accomplish the same thing and fail to meet the needs of users like Craig.

But the goal was never to iterate every single frame. That would edge on the impossible for a full-feature film.

The goal for Craig and creatives alike is to translate their vision without the cost of lost efficiency.

This outcome is achievable, even for projects on the scale of Toy Story and Finding Nemo, but requires tools that focus on their roles and their goals — and not convert professionals to prompt editors.

As the AI summer continues to churn out new products and new avenues for creatives, we’ll begin to see an evolution in the way AI Video lifts up creatives.

“Every shot and frame counts, and these ideas throw that out the window,” a long-time animator-turned-engineer messaged me this morning, but how soon can we return efficient creative control back to the creatives?

note: this is an update on the progress of project in the article: screenplay to VSE

our first tiny win since the screenplay to scenes add-on, i managed to setup a the basis for a clip inspector that surfaces the clip we want to edit with the Khaos Machine.

Apple proves it has Vision

Setting the stage for Apple’s future is its latest groundbreaking innovation: the Apple Vision Pro. Introduced as Apple’s first spatial computer, the Vision Pro epitomizes decades of Apple’s experience in designing high-performance, wearable devices. It’s an ambitious product that integrates advanced technology into a compact, elegant form, offering a mixed-reality experience unlike any other. With ultra-high-resolution displays and an advanced Spatial Audio system, the Vision Pro can transform any space into a cinematic experience or a 3D interactive environment. This device opens up new horizons for content creation, allowing creators to immerse themselves and their audiences in experiences that were previously unimaginable. The introduction of the Apple Vision Pro not only underscores Apple’s continued commitment to innovation but also sets a new benchmark for what’s possible in the realm of digital creation and interaction.

Unification of Tools

Expanding on the vision of transforming Apple into a $4 trillion creative powerhouse, a comprehensive strategy must encompass the full suite of Apple’s creative and development tools. This includes not only flagship applications like Final Cut Pro, Logic Pro, and Xcode but also Motion, Compressor, ARKit and the plethora of other frameworks Apple has meticulously built over the years. By unifying these tools into a cohesive creative platform, Apple could offer an unmatched ecosystem that caters to every aspect of digital creation and interaction.

Imagine a platform where video editors, graphic designers, music producers, app developers, and XR creators can seamlessly collaborate and innovate. With Motion and Compressor, video professionals could create stunning visuals and effects, then efficiently encode and distribute their work. ARKit, coupled with Apple’s powerful frameworks, would enable developers to craft immersive augmented reality experiences that could transform education, entertainment, and retail.

Integrating these tools into a singular, streamlined platform would not only simplify the creative process but also foster a community of creators who could push the boundaries of what’s digitally possible. Such a platform would serve as a foundation for next-generation content creation, enabling seamless integration across devices and unlocking new creative potentials.

This holistic approach to unifying Apple’s creative and development tools underlines the company’s commitment to innovation, privacy, and a creator-owned future. It positions Apple not just as a provider of premium hardware and software but as a leader in shaping the future of how we create, share, and experience digital content. By leveraging its entire ecosystem, Apple can empower creators in unprecedented ways, catalyzing a new era of digital artistry and technological advancement.

Platform Benefits

The strategic decision to open up Apple’s professional-grade tools as part of its ecosystem carries the potential to redefine the creative landscape. By making tools such as Final Cut Pro, Logic Pro, Motion, Compressor free like Xcode, ARKit GameKit and more accessible to a broader audience, Apple could significantly enhance the diversity and quality of content and experiences within its marketplaces. This inclusivity would not only democratize high-quality content creation but also attract a wider array of creators to the Apple platform.

The benefits of this approach are manifold. For starters, it would foster a vibrant community of content creators, ranging from filmmakers and musicians to app developers and XR designers. This diversity would, in turn, enrich the Apple ecosystem with a wider variety of apps, games, music, movies, and educational content, making Apple devices even more indispensable to consumers.

Additionally, by opening up its professional tools, Apple could establish itself as the go-to platform for creative professionals and hobbyists alike. This would likely lead to an increase in the adoption of Apple products, as creators invest in the hardware that offers them the best tools for their craft. Additionally, a more robust ecosystem of tools, content and apps would enhance user engagement, keeping customers within the Apple ecosystem and driving further revenue through both hardware sales and content purchases.

In general, the platform benefits of making Apple’s professional-grade tools free are clear: it would not only solidify Apple’s position as a leader in technology and creativity but also stimulate innovation and content creation across the board. This strategy aligns perfectly with Apple’s commitment to privacy, user empowerment, and creating a rich ecosystem where creators of all levels can thrive. As a result, Apple could unlock new growth opportunities, paving the way to achieving a $4 trillion valuation by redefining the future of digital creativity and interaction.

Shift in Monetization Strategy

In envisioning Apple’s journey towards a $4 trillion valuation, a pivotal shift in its monetization strategy emerges as a critical component. Traditionally, Apple has levied significant fees on sales within its ecosystem, from app purchases to subscriptions. However, by recalibrating this approach—reducing the financial burden on creators and instead enriching the ecosystem’s value—Apple stands to ignite a renaissance in content creation and innovation.

This strategic shift would involve lowering the barriers for creators to distribute and monetize their work on Apple platforms. By reducing the commission rates on the App Store and offering more favorable terms for content creators, Apple could significantly enhance the attractiveness of its ecosystem to a broader range of innovators. This change would not only incentivize more developers, artists, and educators to choose Apple as their primary platform but also encourage existing creators to invest more resources into enhancing their offerings.

The benefits of this approach extend beyond mere increases in the quantity of content. A more supportive financial model would enable creators to focus on quality, innovation, and experimentation. Freed from the constraints of high fees, developers could afford to take risks on novel ideas, artists could produce more engaging and diverse works, and educators could develop more comprehensive and interactive learning materials.

This shift would foster a more vibrant and diverse creator community. By supporting creators from various backgrounds and disciplines, Apple could facilitate a richer, more inclusive ecosystem. This diversity would not only reflect Apple’s commitment to empowerment and creativity but also attract a wider audience to its platform, enhancing user engagement and satisfaction.

In essence, by moving away from heavily taxing network contributors and instead offering more value through its ecosystem, Apple could cultivate a thriving community of creators. This community would be the lifeblood of Apple’s ecosystem, continuously enriching it with innovative apps, captivating content, and transformative experiences. Such a vibrant ecosystem would not only solidify Apple’s position as a leader in technology and creativity but also drive sustainable growth, paving the way for Apple to achieve and possibly exceed a $4 trillion market valuation.

Commitment to Privacy and User-Owned Data

In the digital age, where data is often likened to the new oil, Apple’s steadfast commitment to privacy and user-owned data stands as a beacon of ethical leadership. This principled stance is not just a policy but a core component of Apple’s identity, distinguishing it sharply from competitors. By doubling down on its privacy-first approach, Apple not only meets the growing consumer demand for data protection but also elevates its platform as the more ethical alternative in the tech landscape.

Apple’s dedication to privacy manifests through its robust data protection measures, transparent policies, and the empowering of users with control over their personal information. Features such as App Tracking Transparency and end-to-end encryption on iMessage and FaceTime are prime examples of Apple’s commitment to safeguarding user data. Such measures reassure users that their creative content, personal information, and digital interactions are secure and respected, fostering a trust-based relationship between Apple and its global user base.

In the context of creating a unified creative platform, this commitment to privacy and data ownership becomes even more crucial. As creators produce and share more content across Apple’s ecosystem, the assurance that their intellectual property and personal data are protected is paramount. This environment not only nurtures creativity but also encourages a wider range of creators to join the Apple ecosystem, knowing their work and privacy are in safe hands.

Moreover, by positioning itself as the ethical choice, Apple can attract consumers and creators who are increasingly wary of platforms that commodify user data for advertising or other purposes. This alignment with consumer values on privacy not only strengthens Apple’s market position but also serves as a competitive advantage in attracting quality talent and content to its platform.

In sum, Apple’s unwavering commitment to privacy and user-owned data is not merely a feature of its ecosystem; it’s a foundational pillar that could significantly contribute to its journey toward a $4 trillion valuation. By upholding these values, Apple not only aligns with the growing demand for data protection but also sets a new standard for what it means to be an ethical leader in the technology industry. This approach fosters a safer, more creative, and more vibrant community of users and creators, propelling Apple forward in its mission to empower individuals through technology.

Contrast with Competitors

In the rapidly evolving tech landscape, Apple’s strategic focus on empowering creators offers a stark contrast to the ambitions of competitors like Facebook, which has pivoted towards owning digital realms such as the metaverse. This distinction is critical, as it underscores a fundamental difference in philosophy between the two tech giants. While Facebook (now Meta) envisions a future where it controls vast, immersive digital spaces, Apple is positioning itself as an enabler of individual creativity and innovation.

Apple’s emphasis on providing a unified creative platform, equipped with professional-grade tools and a commitment to privacy, positions it uniquely in the tech ecosystem. Rather than seeking to dominate a digital frontier, Apple aims to empower its users with the tools and freedom to create their own content, experiences, and applications. This approach not only fosters a more inclusive and diverse creative community but also aligns with Apple’s long-standing ethos of putting the user at the center of everything it does.

In contrast, Facebook’s pursuit of the metaverse represents a centralized vision of digital interaction, where the platform itself becomes the primary mediator of experiences and transactions. While this offers exciting possibilities, it also raises concerns about privacy, data ownership, and the concentration of power in the digital age. Apple’s focus on empowering creators with tools and a secure platform offers an alternative vision, one where individuals retain control over their creations and data.

By highlighting this commitment to empowerment and privacy, Apple can differentiate itself in the eyes of consumers and creators alike. This differentiation is not merely a matter of marketing but reflects a deeper philosophical stance on the role of technology in society. Apple’s approach suggests a future where technology serves to amplify human potential, creativity, and freedom, rather than constraining them within a single, corporate-owned digital ecosystem.

This contrast with competitors like Facebook not only strengthens Apple’s brand identity but also resonates with a growing segment of the population concerned about privacy, data sovereignty, and the ethical implications of digital platforms. In embracing this role, Apple not only sets itself apart in the competitive landscape but also lays the groundwork for a sustainable, creator-driven future that could significantly contribute to its goal of reaching a $4 trillion valuation.

Legacy of Innovation

Apple’s journey towards becoming a $4 trillion creative platform is deeply intertwined with the legacy of its co-founder, Steve Jobs, whose vision for technology was always about empowering individuals to make a dent in the universe. Jobs’ philosophy of innovation and creativity remains a guiding light for Apple, emphasizing the potential of technology to empower people to create, innovate, and express themselves in unprecedented ways. By invoking this legacy, Apple can position itself not just as a technology company, but as a champion of a creator-owned future, where individuals are encouraged and enabled to own their creations and innovations.

Steve Jobs famously said, “Technology alone is not enough—it’s technology married with liberal arts, married with the humanities, that yields us the result that makes our heart sing.” This perspective is more relevant today than ever as Apple stands on the brink of unifying its ecosystem to support creators across various domains. By providing tools that integrate seamlessly across devices and platforms, Apple empowers creators with the freedom to bring their visions to life, from the initial spark of an idea to the final masterpiece.

Apple’s commitment to this vision is evident in its continuous innovation in hardware and software, making sophisticated technology accessible and user-friendly. This approach democratizes content creation, making it possible for more people to participate in the creative process, regardless of their technical expertise. By doing so, Apple not only honors Steve Jobs’ legacy but also builds upon it, pushing the boundaries of what technology can achieve in service of creativity and innovation.

Championing a creator-owned future, Apple sends a powerful message about the value of intellectual property and the importance of supporting creators in maintaining control over their work. This stance not only differentiates Apple from its competitors but also aligns with a broader cultural shift towards recognizing and valuing the contributions of creators in the digital age.

In essence, Apple’s legacy of innovation, inspired by Steve Jobs, provides a solid foundation for its vision of a unified creative platform. By empowering individuals through technology, Apple can inspire generations of creators to innovate and own their futures. This commitment not only pays homage to Jobs’ vision but also propels Apple towards a future where technology continues to make our hearts sing, driving the company towards its $4 trillion valuation with creativity and innovation at its core.

in the screen recording i’m selecting the screenplay.json and the media directory where i have generated images and audio clips based on the same screenplay data.

how we got here?

The journey here didn’t start with Blender as target for building prototype, but after i’d had enough fiddling with Xcode to FCPX (Final Cut Pro X) via a macOS extension that never worked, nor did the SDK provide any useful feedback. after digging into the Console logs, i encountered a dreaded old friend Mobile Gestalt..

That was enough for me to really consider abandoning my effort to force our prototype into the FCP workflow. i would have used Motion to build my plugin or generator, but i need to call APIs from the timeline and i couldn’t find an option other than what seems like a janky for Apple option that requires side loading an SDK from Jan 2022 just to enable the macOS extension in Xcode. then once all setup it still failed. i tried. i really did and i even started to reach out to my network for support. i really wanted it to work because of my own history with FCP, but i was forcing it and without proper support it was a blocker.

I really can’t believe they’re still charging people for FCP, especially given obvious lack of support, which is no more apparent than the chunky User Guide(s) that used to ship with the installer disks.

The world has come a long way since this was a luxury, but it seems like FCP has zero plans to invest in innovations and building an ecosystem. they seem to be stuck in harvest mode. forget about feeding these monster PDFs into a GPT, u can’t get it to complete a single response without throwing an error.

Once i accepted the shortcomings of FCPX and it wasn’t my FCP, i came to my senses and looked to Blender and was able to make significant progress as demonstrated in video above. Blender has a library called bpy which is core to the application as it provides programatic access to everything possible in Blender.

What next?

This is just the beginning of our journey as a studio as build and develop tools that address our needs to support content creation. Now that we’re essentially unblocked on building GUI that can interact with the Video Editor timeline, we’re cooking with gas and can proceed as planned.

As we proceed, we’ll continue to post updates.

Pulpy Fictional is a fanart recreation of Pulp Fiction to exercise our components of our screenplay engine.

Leveraging applied ML, screenplay analysis and software automation, we’re converting the Pulp Fiction screenplay to blocked out video.

Here is the original segment of text from the screenplay we’re using as input to create this scene.

INT. COFFEE SHOP - MORNING

A normal Denny’s, Spires-like coffee shop in Los Angeles. It’s about 9:00 in the morning. While the place isn’t jammed, there’s a healthy number of people drinking coffee, munching on bacon and eating eggs.

Two of these people are a YOUNG MAN and a YOUNG WOMAN. The Young Man has a slight working-class English accent and, like his fellow countryman, smokes cigarettes like they’re going out of style.

It is impossible to tell where the Young Woman is from or how old she is; everything she does contradicts something she did. The boy and girl sit in a booth. Their dialogue is to be said in a rapid pace “HIS GIRL FRIDAY” fashion.

YOUNG MAN No, forget it, it’s too risky. I’m through doin’ that shit.

YOUNG WOMAN You always say that, the same thing every time: never again, I’m through, too dangerous.

YOUNG MAN I know that’s what I always say. I’m always right too, but –

YOUNG WOMAN – but you forget about it in a day or two -

YOUNG MAN – yeah, well, the days of me forgittin’ are over, and the days of me rememberin’ have just begun.

YOUNG WOMAN When you go on like this, you know what you sound like?

YOUNG MAN I sound like a sensible fucking man, is what I sound like.

YOUNG WOMAN You sound like a duck. (imitates a duck) Quack, quack, quack, quack, quack, quack, quack…

YOUNG MAN Well take heart, ‘cause you’re never gonna hafta hear it again. Because since I’m never gonna do it again, you’re never gonna hafta hear me quack about how I’m never gonna do it again.

YOUNG WOMAN After tonight.

The boy and girl laugh, their laughter putting a pause in there, back and forth.

YOUNG MAN (with a smile) Correct. I got all tonight to quack.

A WAITRESS comes by with a pot of coffee.

WAITRESS Can I get anybody anymore coffee?

YOUNG WOMAN Oh yes, thank you.

The Waitress pours the Young Woman’s coffee. The Young Man lights up another cigarette.

YOUNG MAN I’m doin’ fine.

The Waitress leaves. The Young Man takes a drag off of his smoke. The Young Woman pours a ton of cream and sugar into her coffee. The Young Man goes right back into it.

YOUNG MAN I mean the way it is now, you’re takin’ the same fuckin’ risk as when you rob a bank. You take more of a risk. Banks are easier! Federal banks aren’t supposed to stop you anyway, during a robbery. They’re insured, why should they care? You don’t even need a gun in a federal bank. I heard about this guy, walked into a federal bank with a portable phone, handed the phone to the teller, the guy on the other end of the phone said: “We got this guy’s little girl, and if you don’t give him all your money, we’re gonna kill ‘er.”

YOUNG WOMAN Did it work?

YOUNG MAN Fuckin’ A it worked, that’s what I’m talkin’ about! Knucklehead walks in a bank with a telephone, not a pistol, not a shotgun, but a fuckin’ phone, cleans the place out, and they don’t lift a fuckin’ finger.

YOUNG WOMAN Did they hurt the little girl?

YOUNG MAN I don’t know. There probably never was a little girl – the point of the story isn’t the little girl. The point of the story is they robbed the bank with a telephone.

YOUNG WOMAN You wanna rob banks?

YOUNG MAN I’m not sayin’ I wanna rob banks, I’m just illustrating that if we did, it would be easier than what we been doin’.

YOUNG WOMAN So you don’t want to be a bank robber?

YOUNG MAN Naw, all those guys are goin’ down the same road, either dead or servin’ twenty.

YOUNG WOMAN And no more liquor stores?

YOUNG MAN What have we been talking about? Yeah, no more-liquor-stores. Besides, it ain’t the giggle it usta be. Too many foreigners own liquor stores. Vietnamese, Koreans, they can’t fuckin’ speak English. You tell ‘em: “Empty out the register,” and they don’t know what it fuckin’ means. They make it too personal. We keep on, one of those gook motherfuckers’ gonna make us kill ‘em.

YOUNG WOMAN I’m not gonna kill anybody.

YOUNG MAN I don’t wanna kill anybody either. But they’ll probably put us in a situation where it’s us of them. And if it’s not the gooks, it these old Jews who’ve owned the store for fifteen fuckin’ generations. Ya got Grandpa Irving sittin’ behind the counter with a fuckin’ Magnum. Try walkin’ into one of those stores with nothin’ but a telephone, see how far it gets you. Fuck it, forget it, we’re out of it.

YOUNG WOMAN Well, what else is there, day jobs?

YOUNG MAN (laughing) Not this life.

YOUNG WOMAN Well what then?

He calls to the Waitress.

YOUNG MAN Garcon! Coffee!

Then looks to his girl.

YOUNG MAN This place.

The Waitress comes by, pouring him some more.

WAITRESS (snotty) “Garcon” means boy.

She splits.

YOUNG WOMAN Here? It’s a coffee shop.

YOUNG MAN What’s wrong with that? People never rob restaurants, why not? Bars, liquor stores, gas stations, you get your head blown off stickin’ up one of them. Restaurants, on the other hand, you catch with their pants down. They’re not expecting to get robbed, or not as expecting.

YOUNG WOMAN (taking to idea) I bet in places like this you could cut down on the hero factor.

YOUNG MAN Correct. Just like banks, these places are insured. The managers don’t give a fuck, they’re just tryin’ to get ya out the door before you start pluggin’ diners. Waitresses, forget it, they ain’t takin’ a bullet for the register. Busboys, some wetback gettin’ paid a dollar fifty a hour gonna really give a fuck you’re stealin’ from the owner. Customers are sittin’ there with food in their mouths, they don’t know what’s goin’ on. One minute they’re havin’ a Denver omelet, next minute somebody’s stickin’ a gun in their face.

The Young Woman visibly takes in the idea. The Young Man continues in a low voice.

YOUNG MAN See, I got the idea last liquor store we stuck up. ‘Member all those customers kept comin’ in?

YOUNG WOMAN Yeah.

YOUNG MAN Then you got the idea to take everybody’s wallet.

YOUNG WOMAN Uh-huh.

YOUNG MAN That was a good idea.

YOUNG WOMAN Thanks.

YOUNG MAN We made more from the wallets then we did the register.

YOUNG WOMAN Yes we did.

YOUNG MAN A lot of people go to restaurants.

YOUNG WOMAN A lot of wallets.

YOUNG MAN Pretty smart, huh?

The Young Woman scans the restaurant with this new information. She sees all the PATRONS eating, lost in conversations. The tired WAITRESS, taking orders. The BUSBOYS going through the motions, collecting dishes. The MANAGER complaining to the COOK about something. A smiles breaks out on the Young Woman’s face.

YOUNG WOMAN Pretty smart. (into it) I’m ready, let’s go, right here, right now.

YOUNG MAN Remember, same as before, you’re crowd control, I handle the employees.

YOUNG WOMAN Got it.

They both take out their .32-caliber pistols and lay them on the table. He looks at her and she back at him.

YOUNG WOMAN I love you, Pumpkin.

YOUNG MAN I love you, Honey Bunny.

And with that, Pumpkin and Honey Bunny grab their weapons, stand up and rob the restaurant. Pumpkin’s robbery persona is that of the in-control professional. Honey Bunny’s is that of the psychopathic, hair-triggered, loose cannon.

PUMPKIN (yelling to all) Everybody be cool this is a robbery!

HONEY BUNNY Any of you fuckin’ pricks move and I’ll execute every one of you motherfuckers! Got that?

“PULP FICTION”

				Posted				

02/11/2024

				in				

Animation

				by				

khaos

				Tags:				

AI, DALL-E, FCPXML, Final Cut Pro, Machine Learning, ML, Pulpy Fictional

below are a few screenshots of log output from our agents as they are reading the screenplay text and extracting insights about the story, characters, location, dialogues and more.