S6E05: Absolutely Flawless

Our technology does the magic. - Hang on a second, this is AI doing the acting night. - Workflow is significantly more important. - Which really changes how you approach the entire process. - Avid's reputation of building professional grade workflows aligns very well with what we're seeing as professional grade innovation. - A completely different part of the time, if you like, for the experience that people have is quite short. - Everything is cut on an avenue, everything. (soft music) - Hi everyone, welcome back to Making the Media. I'm Ray Thompson, and I'm here with Craig Wilson, and we're excited about this next episode where we're gonna focus on flawless, which is a very, very exciting AI driven video-diving tool set. - Yeah, that's right, Ray. And on this episode, as you say, it's focusing on some really revolutionary technology from from flawless, and we're joined by their SVP of customer success, Peter Bush. Now Ray, before we hear from Peter, tell us a little bit more of it flawless. - Absolutely, we found a flawless, because Peter actually reached out to me on LinkedIn, and he sent me a link to their site, and I was just so impressed. Right out of the gate, when I looked at what they were trying to do, it was immediate to have a discussion with them. And then of course, Peter's been fantastic, as has the entire company. And what they do is using AI, offer a way to do video-diving in a very innovative way, that drives super high efficiency, and can save significant money on the back end, in both reshoots, as well as localization, right? Getting content done in various different languages, and delivered in a much faster fashion, right? It's very revolutionary, and it's super exciting. And it's so well done, in terms of its panel or extension, inside of BDCoposer. It's really just awesome to watch it happen. For the user, the experience is great. And I know we're gonna dig right into that Peter, but I have to say this is one of my favorite integrations so far. - Yeah, this kind of stuff is really so, so cool. It's quite amazing. So let's hear now from Peter. And as he discusses, he is being in the world of artificial intelligence for quite a while. (bells chiming) - Yeah, I've had the good fortune of being in AI since 2008. And so I got my roots and start in animation, worked at Warner Brothers in Paramount on feature 2D and 3D animation, and then ended up working for a AI technology company that essentially could use actors' performances to animate faster. So I got really comfortable in using and building tools, bringing those to market, and joined flawless about three years ago when they were working on AI back technology that aims to impact filmmaking. And so it was a much wider industry to go after than just purely character animation and facial animation. So I've been very comfortable in how new technology is brought to market and through key strategic partnerships like what we've got with you guys. - Yeah, so for people who don't know, maybe tell us a little bit about flawless, because the company's been around for a little while. It's an idea that's been around for a little while, but it's really been brought to market over the last year or so. - Yeah, I mean, I think the origin story matters. We were founded by a Hollywood filmmaker of a DGA, WGA member, a guy named Scott Mann. And so Scott had built his career directing a lot of independent films and understanding the compromises that indie filmmakers have to make and how content ultimately gets translated. He had worked on a film back in 2015. This is kind of the origin story of flawless, called Heist with Robert De Niro. And the both of them worked on the script extensively. And then when they saw the German dubbed version of that film, they realized how destructive it was to the original writers intent in English. And so he said, "Serly, there's a better way to translate content and make this process better." And so long story short, we found that he founded the company based on those ideas of how do we solve fundamental problems filmmakers have had in telling their original story. But then ultimately, how do we get that to a global audience? And so over the last six years, about $100 million has been invested into the core technology to essentially change dialogue after you've shot and do it in a very foundational and fundamental way that's ethically aware of the impact it has to the editorial process, to the acting process. But ultimately, we've built a set of tools in post-production and localization to really impact the film industry in a transformative way. Yeah, we'll come back a bit to talk about that. Because I do think the ethics question is a really interesting one. But just from a practical reality perspective, I think all of us have seen dubbed films, whether from the language we understand into another language or from another language into the language that we understand. But you talk there about the creative process. And in some respects, the people who had that creative vision, feeling that the vision was in some ways impacted by going through that dubbing process. So from your perspective, how does flaws solve that issue? So fundamentally, you have to first look at the workflow that how films are created, right? Of how do we sort of understand what compromises are made and say the original creative edit? Which for decades, editors are long limited to what footage they've shot, right? And if they haven't shot it, they then need to do course corrections with ADR or reshoot or something like that. And so what we did is take a look at that post-production workflow from Scott's lens and from an editorial lens and say, well, what would editors want? They would want the ability to change dialogue more easily and have the flexibility of using inputs like ADR. So you could use audio, basically, as an input to our technology to change a performance on screen. And when I say change, you're changing a line of dialogue. So the process with our deep editor integration directly into media composers, you're going to scan through your timeline and find a moment, a line of dialogue you want to change is probably the easiest example to use. And so you'll find footage of whether the original line was delivered. And then you're going to find what we call driving data or a new performance. And a new performance can be just from audio, right? For an ADR source. And so you line up the new audio to the original performance so that it's timed correctly. And then the workflow is you're just marking in and out points on sort of this is the start of this line. Here's the end of this line. And you pass that to our deep editor product. And we essentially-- the best way to think about it is we reanimating and re-articulating the way that performance is delivered. But we do that in a very smart way. So it feels very natural. We learn how the original actor essentially delivered that line and how they emoted and expressed. And then when we create the new motion and the new line of dialogue, we're remaining faithful to that actor. And so it feels very natural. And so we're changing a line of dialogue on what the actor said. But it looks like something they would have originally delivered on camera. So it's incredibly nuanced. But it gives this powerful sort of ability to filmmakers that they just simply didn't have in live action filmmaking. And so the way we've streamlined this into the editorial workflow and media composer is very natural. I mean, you're just marking in and out points. And now you're sort of picking and choosing where to create what we call a VUB or a Visual Dub. And the process of VUBbing is really sort of how do you sort of get the story exactly the way you want it in using the inputs that you already have access to in your workflow. So is it fair to say that in some ways that you really see as it's saving time, saving that potential reshoot that people possibly have to do, which sometimes is even possible. But it's remaining true to the creative vision if you like. Yeah, there's a lot of really great applications. I mean, I think every editor knows the dilemma of why they would need to change a line of dialogue. Maybe the story's not working. Maybe it wasn't shot. Maybe they got a note from an audience screening or from a studio. Lots of reasons why lines of dialogue need to be changed. And so the way in which we've manifested that is at any point during the editorial process, you can use VUBbing to essentially correct these things and get the narrative correct, get the story right. And that's where we've seen such a great response from the editorial community of being able to just back into a natural editing workflow, right? But giving the power to change lines of dialogue without doing a reshoot, right? Or without cutting in ADR over the shoulder or cutting away. And all these compromises that you're making are just fundamentally gone, right? And so now you're looking at your edit in an entirely different way with this capability, which has been fun from my point of view to sort of sit back and watch how editors, sort of the light bulb goes on, because they've been making these compromises for decades. And you get really good at it. But fundamentally, that's not the right way to tell the right story. Now Ray, Peter spoke about the media composer, an integration of this is something you've been heavily involved in. So do you want to talk a little bit about the partnership? For where? Yeah. So Peter actually reached out-- man, must be about a year, a year and a half ago, I'd say. About a year and a half. And he explained what flawless was doing. And it sounded extremely not only pertinent at the time, but also exciting, right? Because all of the things that Peter was describing, we saw as real benefits, right? So if we could figure out a way to have a more seamless workflow between flawless and media composer, it would be even more beneficial than it already is, standalone outside the editor. So that was sort of the genesis of the discussion, and it's sort of evolved from there. So we've had, I think-- And it's pretty exciting. I think it's fairly simple now, right? I mean, in terms of the number of clicks needed in order to execute this operation have. with flawless deep editor panel running a media composer is minimal and it's only getting better. So I don't know if you agree Peter, but I mean that's sort of what I've seen so far. Yeah, what's been great to see? I mean we started talking about a year and a half ago, but since then you guys have really kicked off this great initiative within Media Composer to give more access to providers like us. Right? We know the studios are asking for it. They're wanting users to remain entirely in Media Composer for their whole workflow. And so users no longer now have to go out and access and learn a new product. It's super streamlined like I was describing. You're just lining up media sources on your timeline marking in an outpoints and then our technology does the magic and returns that VUB back to you. And so from our point of view it just gives us incredible access to the Media Composer user base, which nearly every major film or show is cut on an avid. Right? So it gives us incredible access, but it also gives us really nice streamlined manner on how to do things and ultimately impact those creative decisions. And so without this workflow it just was very disjointed. And we ultimately were not going to get the adoption we were looking for is incredible as our technology is. One big thing I've learned in my career is workflow is significantly more important than the innovation. Right? Because if we remain in people's workflows, then people don't feel like they're learning new things or have to learn new things. There's already enough challenges and not enough time, not enough resource. And so what we've seen in terms of people embracing the workflow is this feels natural. Right? And to your point Ray, like we have made this very streamlined. I mean there's just not too much to learn other than where you use VUB's and you're at it. And that's the fun part of it. Like that's the new creative expansion is enabling this capability. And then what we've seen is just leaving you know train people on the basics. Here's the core workflows and then just let people go and edit. Right? And then they find other ways to use our technology, which I find fascinating is just that creative mind and how you bring you know technology and creativity together. And it's it's been really streamlined. And I think that's large in part to this initiative that you guys have to give more access to media composer that historical. Yeah, opening it up has been a lot of fun and watching what people are doing with it has been exciting. The other thing too, I think is interesting too about all this is the way in which you know dubbing used to have to work. And sort of what you guys are sort of reinventing right through your technology. And like you said it not only gives the creative more ways in which they can take advantage of the technology to not only the technology. To not only save money time drive higher efficiencies, but like you said get more into the edit and think of new ways to really take advantage of what's possible now. Which really changes how you approach I think the entire process right and the ways in which it's getting used and Peter you can correct me if I'm wrong here. But there's obviously the use of flawless to fix maybe some swearing that might have happened right during the original script to then. You know make it more palatable for television or for another region where that's not acceptable or you know even localization right obviously for for changing language so. I don't know if you want to talk a little bit more about that to Peter like how people are using it now and then ultimately how you envision it being used in the future. Yeah talk about censorship we just were part of a Amazon production called play date and so play date was originally shot as an R rating from director Luke Greenfield and when Amazon acquired the rights to the picture they said hey we really want to release this wider as a PG 13 release. And so they had gone through and identified five key scenes and this was on Kevin James Allen Richardson and East La Fisher on core lines of dialogue where they were swearing. And so in a traditional censorship at it you're just going to cut out that word and cut over the shoulder cut out the scene and make compromise but with with the predator what they were able to do is essentially keep the edit intact. And then rewrite the entire line that the character was saying and when they re wrote it they were able to be faithful to the comedy be faithful to the character but do it in a way that basically get you the rating but preserves all of that. And so in these five critical moments they they literally couldn't reshoot all the actors had gone off around the world to do other things so they logistically couldn't get that done would have been cost prohibitive. So in these five critical moments they rewrote the lines cut in the vubs and then the story plays out great Amazon's happy this is going to get a wider release for them as filmmakers. The other big use case rate that you talked about is localization and so we have a product called true sync which does end and conversion of one language to another so when you're watching the audience is watching they believe that this was shot natively in their own language. So when we change dialogue on actors we can do that for every speaking part what was interesting that I'll talk about is related to editorial is so our first feature film came out earlier this year earlier last year called Watch the Skies. So Watch the Skies was originally a Swedish film and then they translated the whole thing into English using our technology so it was visually dubbed and now it's been theatrically released and now it's out on Amazon Prime. But one of the fun interesting stories that I learned not even till the premiere from the original filmmaker is I asked him I said did you guys stay locked to the Swedish at it and he said actually know when we learned what your technology could do. When we went back into the edit we reopened about 50 scenes and rewrote the dialogue from compromises we had made in the original Swedish at it. So it got to the point now where everything looks in English but it tells a more faithful story to what his original intent was to the point where he now prefers and a lot of the main cast prefer the English edit right which was shot in Swedish so like this is kind of the level of immersion that we can get to. So he's rewrote the scene to tell a more faithful English version of his story but now to the audience they're watching an English film and the audiences have reacted greatly I mean it did double its opening weekend. It's hit in the top five movies on Amazon Prime like in the reality is it's because it's a good film it's not a technology showcase and so that's where I think it speaks to the principles of what we've been founded on. Yeah, great. I think that's a really interesting point there Peter and I mean I think the fact that people are reviewing it and they're viewing it as an English film they're not going in thinking it's something that was in Swedish dubbed into English I think it's quite extraordinary. That's a completely different paradigm if you like for the experience that people have if they're watching a film watching a movie. I mean that's the goal right is you're suspending belief already but you know one of the biggest challenges in dubbing or even ADR right is is how do you get it to look like it was naturally shot footage right and and blend all those sort of our artistries together to for the audience to get immersed in that story. And so you know what we take great pride in is the innovation around getting to the other side of the uncanny valley right if you're not familiar with the uncanny valley it's the idea of these digital performances the closer they get to reality the more creepy it looks and this is where digital humans that I've been involved with my entire career have long been sort of plagued with this idea that it looks uncanny looks creepy. And when you get on the other side of the uncanny valley it just feels natural and so nobody's talking about it so there's this great technological advancement and achievement but the reality is nobody's talking about it because they just thought that's what was shot originally and so it's this kind of interesting moment in time for for me is that everybody just believes this is what shot but I know the incredible effort that had to go under the scenes to to match into real footage right that's what we're talking about here and so it's it's an exciting time for us to be able to bring all these things. And so it's a great time for us to be able to bring all these capabilities to market and it's been fascinating internally on our side to see all the incredible science that had to be built for this. I mentioned earlier on about the ethics question so there might be some people who are you know sitting here listening or watching this going well hang on a second this is this is AI doing the acting now. And there's something I know that that flawless is done a lot of work around in terms of making sure the industry understands what they're doing and also ensuring that there's buy in you know from the industry so so he talk a little bit about that side of things because I know that something that the company really cares about. I mean look AI is very scary obviously everybody is kind of woken up to the impact that it's going to have right and there's plenty of use cases in other areas that are truly impactful but not really respecting the ethics of how to bring this to market and really what we stand and put our flag in the ground on is we're built for filmmakers by filmmakers and that's the industry that we're going after. And so we have the capability to have anybody say anything in any language and we don't take that lightly and so what we have to put in place our safeguards and guard rails and ethics so all of our data was ethically sourced right we we we built models that not built on copyright and material we have license and access to be able to use those we can. Actors that were shot to create our data sets were consent to they knew what they were doing they believe this is impactful positively for filmmakers and so we worked extensively with two main guilds. Right because this is the heart of what we're doing one is the screen actors guild right and so they've been recognizing us as AI done right we have a consent flow built into our tool so that if there's a fundamental change in dialogue that the actor didn't deliver on camera they have to consent to that change. And because it looks so natural we're seeing actors get behind what we're doing because they see us as a path to get their best performance on screen. And so Duncan Crabtree Ireland whose their chief negotiator has been out talking about flawless which we take. in high regard because it's really well. Well, now the other side of the equation is who's our end users are editors. And so we worked with the Motion Picture Editor's Guild and the Emerging Tech Committee. And in that Tech Committee is 15 members that basically put us through the ringer. They ultimately generated a 60 page report. What is the company? What are they doing? What is the workflow? What is the impact on labor? Who's actually doing this? How do cost approvals go in place? How do you know the actor approvals get in place? So we've built this really extensive, recommended workflow and guidance in collaboration with the guild to basically get recognized as, hey, this is the right way to do this. So we have all the safeguards in place. Because of a revolutionary of capability that we enable, we need to work with the labor side of things to make sure that the workforce is impacted positively. And the biggest win out of all of that is the Impag Board. They have 55 people on their board. I have no idea how they get anything done. But they voted basically to say, deep editor, especially with the avid integration, we want all of our members, all of our 8,000 members trained on this workflow because of the transformative nature of it, because it's ethically forward, all of these things that we've been working so hard on are now getting recognized and adopted in the industry. Yeah, one question that came up to me earlier, and I didn't ask it earlier on, but I'll maybe ask now, is that why was it important for flawless, from a flawless perspective, if you like, to have an integration with media composer? What media composer represents is the backbone of professional filmmaking, right, in film and TV, right? Everything is cut on an avid, everything, right? And so at scale, this gives us a channel to be able to access this editorial community and meet them where they're at. They're already editing a media composer. This just becomes a killer new feature for them. And then it's training at scale. And so, you know, avid's reputation of building professional grade workflows aligns very well with what we're seeing as professional grade innovation. And so those two things come together to tell a really powerful story. And that's where I've seen the heart of the partnership is that studios have been asking for tools like this, right? But media composer was the only choice to start. You guys have been the gold standard in professional editing environments. And so it's, to the surprise of no one, when we tell people about this partnership, they get very excited, right? And talking to ACE and all of their members, and we spoke it at it fast-slashed year, there's just a very warm response. 'Cause they love what we're doing. People get impressed by the use cases. But it kind of goes then, well, then what? Do I need to learn something new? And it's like, no, you're gonna stay a media composer and you're creative at it. You're marking in and out points in hitting process, right? It is a very straightforward process to learn. And when the goal, like you can imagine a world, this is kind of my aspiration, is imagining a world where technology doesn't replace creativity. It restores time and focus for editors to be creative again. And when technology disappears in the background, it just becomes seamless. And that's to me the goal of any technology in the creative field is to just have people think differently again. And it just challenges them to tell the right story and why else would we go anywhere else other than media composer to start? - As we all know in this industry, nothing stays still. Nothing stops. There's always new things to do. And I guess there's more stuff that you're looking to do. So can you give us some idea about some of the other workflows and other innovations that you're thinking about? - Yeah, I mean, this is ultimately a beachhead moment for us to get started, right? And to show what these assistive AI technologies can do, right, changing lines of dialogue. There's a whole host of things that you can change in your edit that you've been making compromises for decades on. And so where we've decided to focus is really in the dialogue and the script and the story, right? And that happens in post-production. Most stories are crafted, not on set. They're crafted in the edit, right? And that's kind of the secret to no one. But when we can embrace that craft to about, people tell better stories. And then the extension of that is on the localization side. It's like, well, now you've told the right story in your native language. Well, let's get better reach, right? So can we reduce hours in editorial to tell the right story, right? Reduce these compromises and these things where you can be proud of your work again. But then can we get this out to a global audience? Because if we can get a global audience, we get more eyeballs on it. That means more money in the system. That's good for everyone. And so the symbiotic relationships, kind of what I talked about with "Watch the Skies" is like they were able to reopen that edit. But now the Swedish filmmakers are able to access an English market that they otherwise couldn't. And that just, it's good for everyone involved in that production to be proud of, right? From on set, the gaffers, lighters, and now to in the editorial and everything that was done in post. And so it's this incredibly impactful time that AI is representing and we love being at the tip of the spear of that. And there's plenty more exciting innovations to come. (bell ringing) - Yeah, it's a really amazing stuff. They are an obviously more to come in the future. And I really wonder if we could use flawless to debuzz into lots of different languages. It could probably open up a whole new market for us. - Absolutely, absolutely. And we could do it on the media composer. And it's very cool. And again, I think this is why panels were created, right? To extend workflows that otherwise, would just simply not be possible, right? Without these types of integrations, be much more cumbersome, certainly. And so yeah, it's just a very, very cool one. And I really appreciate Peter's perspective having come from both the production universe, right? Prior to his time moving into the tech universe. So I love hearing what he has to say because of that. So it's great to talk to him. - Yeah, that's right. So if you want to find out more about flawless or other partners who've developed, you know, panel extensions for media composer, scan the QR code. You can see on the screen just now or follow the links in the show notes for more information. There's lots of good stuff there. But what's up next for us, Ray? - Thanks, Craig. Yes, so MovieLapse is up next. If you're not familiar with MovieLapse, just go ahead and Google it. It's basically an initiative that was created by the major studios to more or less look at technology and make recommendations for really all tech providers and for, you know, content producers to sort of follow as a roadmap for how to basically create secure environments, move to the cloud, and basically introduce no trust environments and so on and so forth. So it's really all encompassing. They've published some amazing work so far. I would encourage you to read it if you haven't. But we're really looking forward to this next episode. We're really gonna dig in on all the various different things they've already done and some of the key recommendations and what they're seeing in terms of progress, right? And the industry is a release to those. Yeah, so let's take a sneak peek at what's coming up. - I think it's something which is really very, very powerful and enabling. So API is definitely leading the key to the cloud element of this is the single source of truth. But with APIs comes the question, are we talking the same language? Lots of great information to come, so fingers crossed, you'll join us for that episode when it's out soon. For now, thanks again to Peter for walking us through all of the workout flawless. Thanks again to the production team for the podcast. But from Ray and for me, it's goodbye for now and join us next time here on Making the Media. [BLANK_AUDIO]