Ex-DOAC Employee Explains Why Thumbnail Testing Just Isn't Enough Anymore
33m 22s
The core of success at Diary of a CEO was a data-driven, experimental culture centered on pre-predicting performance before going live. Rather than relying on intuition, the team used analytics to evaluate guest potential, topic relevance, and content elements like thumbnails, identifying patterns that led to consistent growth. Experiments were tightly aligned with key performance indicators such as click-through rate, ensuring focus and measurable outcomes. A structured approach—hypothesizing, designing, measuring, and sharing results—was adopted, with tools like guest profiling and AI translation enabling innovation and early industry leadership. The experimentation culture extended beyond production, influencing guest booking and content strategy through pre-testing and trend forecasting. Even small teams could adopt these practices by testing one variable at a time and fostering accountability through shared learning. This culture was sustained by leadership, accountability mechanisms like the "Experimenter of the Week" award, and team-wide commitment to learning from both successes and failures. The approach not only boosted performance but also inspired broader industry practices, demonstrating that experimentation—when guided by clear goals and data—can drive sustainable growth across media and beyond.
One of the biggest things for us was pre-predicting as much as we could before anything went live.
How the heck do you identify whether a guest is going to perform or not?
How formal is like your experimentation process? Do you like write out a hypothesis?
And for me, like that was the most amazing quality in Alita because. Grace, my first question for you is just around this incredible,
unique title that you had working with the Diary of a CEO.
What were your main responsibilities as head of failure and experimentation?
Yes, it is quite a unique title.
If we wanted to experiment on our production and how we were editing the episode, for example,
there was two parts. The first was like, how do we improve everything we're already doing?
So that was looking at like, what are we doing on YouTube and how do we improve that current performance?
Or how does the episode look or the guests and how do we improve that?
And then the other side of it was looking at what are the new opportunities that we could have
and we could experiment on to get ahead of the industry?
When I think of experiment, I think it could go off the rails very, very quickly.
So how do you have the right constraints on the experiment to know that it's worth the time and money and energy to put into it?
This is a massive question because it's something we consider for a long time as well.
At the start, when I was in the experimentation team, we looked at all these ideas and we're like,
let's try everything and see what works.
And it actually became like made us so stretched thin of what we could try.
And also then the results were stretched thin because we couldn't delve deeper.
So what we ended up doing was looking at what our company KPIs,
or show KPIs were and going, what are the main goals we want to achieve?
And how do we experiment towards those goals?
So instead of just running experiments,
because you could run experiments on a million different things within a year,
but we were like, what are the three or five KPIs we really want to achieve?
And how do we make sure everything we're doing is working towards them?
And that really helped us align on actually focusing experiments.
Say, for example, we really wanted to improve click-through rate.
So click-through rate could be in the middle of our diagram.
And we run experiments on a million different things.
And we run experiments all around that to try and improve.
So it has that one overarching goal rather than before.
It might have been, there's one experiment for click-through rate,
one experiment for an average view duration,
and they were all running around each other.
Okay, so if you were to try to improve click-through rate,
you mentioned that there was a bunch of different ways of doing that.
Let's say that that is the KPI that you're tracking and it's day one.
What does day one look like to get to the point where eventually
the CTR is higher?
What we would do is go and brainstorm in a session.
How many ways can we try and improve our click-through rate?
Also, click-through rate is probably the worst example.
I'd say other ones like average view duration are better.
But say we're doing it on click-through rate.
We'd sit down, brainstorm, like,
what are all the ways that we could improve it?
Working off real-life analytics.
So we're not just like thinking of things.
We're looking at what has performed well before,
what hasn't, and why.
So we're using, we used to call it killing the guesswork.
Using the data to go what will work and what won't in the future.
So we're almost trying to, before we've even started the experiment,
pre-predict with the previous data what's going to work.
The way we would test it is not all at the same time, obviously,
because you want one variable.
You don't want to test a million different ideas.
Were you thinking at a level where you were testing more than just the thumbnail to get CTR?
Were you thinking about, well, what if we tested it on Facebook first
and see with paid ads, those kinds of things?
Yeah, we started doing that.
Four years ago, maybe it was in my first year that I started there.
We started thumbnail testing or we call it pre-testing ahead of time.
So we used to pre-test all of our thumbnails before they went on YouTube.
And obviously at the time, YouTube didn't have the thumbnail testing tool,
which the one they have now is obviously the best because it's based on the real algorithm.
The closer we could get to knowing what would work before is why over the last few years,
we've had such insane growth of the show.
Is there like a go-to example?
Where something that came out of your department exponentially changed the way that you operate at Diary of a CEO?
Well, I'll share a few.
The first one was we didn't use to have Steve's face on the thumbnail, which sounds so little.
But one day we were like, let's try his face.
And this was years ago.
And we're like, let's try his face and see what happens.
And we added it and it won every single time.
And we were almost like, why did we not do that before?
Right.
The other one is definitely translations.
Like it is the coolest.
Part of just before I left, like what I had done was helping to build that.
Because I think I didn't realize when I started how big of a like area it would be,
especially three years ago, the industry wasn't really like Instagram didn't have the translation feature.
YouTube had only brought it out more recently.
And so for us, we were like, let's try it.
Let's see what happens.
At the start, we're like, OK, this is not really working.
And we thought, like, do we park it?
Do we keep going?
But we just like kept trying to see, will it?
Some at some point pick up.
And it was at the start of last year that we were like, this is it.
And it just started to take off and skyrocket.
And I think the best part of that for me is like, if you were a normal team or a normal company where we had like KPIs to hit every year, we probably would have stopped that after the first year because it wasn't that big and it wasn't really taking off.
And we were probably investing more time and money than what it was worth, whereas we were allowed to continue over the years to keep trying.
And I think now it's taken off.
And now we were ahead of the industry because people are now trying to.
It was AI auto dubbed, by the way.
But we were ahead of the industry and it was just lucky that we thought this is worth trying and keep testing.
How do you actually implement what you did at Flight Story into a smaller team so that testing is actually happening and not getting brushed off for something else?
I get that question a lot.
And usually it's even like, I am a solo creator or solopreneur.
How do I have the time to experiment?
How do I find the time to experiment?
Especially in YouTube.
I think sometimes you can get caught up and this happened to us sometimes is you would build the episode, release it and then move on to the next one so fast and you don't actually get to sit there, review it, go, what actually happened?
You might watch like the first 24, 48 hours and then go, OK, on to the next.
Rather than actually going like, what has happened here throughout this whole episode?
And it's even harder when it's a two hour, three hour episode that like what has happened and what worked and what didn't work.
And I want to say it's even harder.
I think.
Some YouTube channels it would be amazing on because if the guest doesn't change, but for podcasts especially, it can be really hard because the guest is such a massive variable in a show.
But if you'd say a channel like a MrBeastDesk or Sidemen, those sort of channels where you can have kind of the same core like people in the episode, it's really fun to experiment with.
And like from a smaller team's perspective, I think it's about the accountability.
So when I said culture before, it's when you're reviewing those episodes or the video.
It's saying like, what experiments have we run this week?
Who ran an experiment?
Who didn't?
How do we come together and like keep that accountability up?
Because we know that's how we're going to grow.
I think some people are very data driven people and then others are not.
And it doesn't mean that the people who are not data driven can't experiment.
They definitely can.
Like one of our previous social media managers in our company who used to manage Steve Socials, she was like very creative driven rather than data.
And I was like, let's mix the two and become.
Still creative driven, but use the data to help us inform the creative and there's a whole debate of it of like, should it be brand or data led at the end of the day, like controversial, but I am very much saying led by the numbers rather than just a nice color or brand design.
But I think in terms of a team, the size should not matter.
It should be, what can we do to change one element or some elements within this piece of work?
So if you're always seeing like a drop off at 10 minutes, let's say, say you have a 30 minute episode.
And you're always seeing a drop off at 10 minutes.
And that's where your AVD changes.
What can you do just before that 10 minute mark to see if you can keep people going?
And again, not a perfect experiment because there's so many variables, but we want to try like as much as we can to try and experiment within it.
So if we said, okay, we're going to put something on screen, that's like this really interesting thing is coming in two minutes.
And we put that at nine minutes and see if people stay an extra two minutes on the AVD, like does it work?
And I think, don't get me wrong, it's not going to work for every channel, but I think it's worth those little things of like, what can you test?
And also it doesn't have to be massive.
Like they can be one percents and little things.
And I think that's what experimentation can scare people because they go like, I am a solo creator.
I can't, don't have time to test anything.
And it's like, well, if you're making the content, let's at least try something new or tweak something slightly every time.
Do you like?
Do you write out a hypothesis and like have a budget and things like that?
Yeah, usually budget wise, it's pretty flexible.
So we would have, well, we had like overall yearly budgets recently, but before it was really flexible, like we would say, hey, this is the experiment.
So the actual steps we had, like we had multiple steps of like, this is what we need to fulfill for an experiment.
So definitely a hypothesis.
Then we would write out like, what is the experiment actually going to be?
And then the last one was the determinant.
So what actually deemed the experiment a success or a failure?
Yeah.
And so we would run that for every experiment.
And also it's evolved even more now.
Like that was when I started.
Now it's starting to grow and evolve and into even more like different team and experiment and procedures as well.
But I think the most important part in it was people actually sharing the successes and the learnings and all the failures, because there's no point in me going and running, running an experiment and then keeping the results to myself and not sharing it with everyone because we're going to learn faster from a hundred people rather than just one.
just one person.
how did people share them so there's a few different ways we had a slack channel so people
could share straight away updates in there then we had like a database of all the experiments that
have ever run so that people could then go in there and see like what was run on this date or
on this channel what was the performance when do we need to retest it so for example the thumbnails
we started adding images to the diary of a ceo ones and that was tested because we thought oh
we haven't tested anything on thumbnails in a while let's keep testing and seeing what works
how many people are on the team overall and like you said you had a team under you doing experiments
and like how does that play out throughout the team yeah it's a good question because it's
something we had debated for a long time of whether experimentation should sit in its own team within
flight story as the parent company or should there be an experimenter in each team so in like an
experiment in the social team or the production team and we ended up having it right now as a
separate team that works with every team
which is like actually worked out for the better because it then gives
the people in those teams the not only autonomy but the responsibility to experiment themselves
whereas if there was an experiment in their team they'd probably be relying on that person to
experiment rather than like taking the experimentation culture and ingraining it into
their work and that was the most important part to be honest was people actually experimenting
because they wanted to not because they had to and it was really like hard to
start because everyone was super busy as they are in every company and they were like i don't have
time for this i don't have time to try this new thing but the whole point is if you just keep doing
the same thing you're usually not going to grow you need to keep innovating or testing new things
every single episode and like the best person who ever said this is mr beast obviously work out what
didn't didn't work and keep improving and so within our teams we were like you take the full
autonomy to go and run these experiments we are here to help so like help them write up the
experiment procedure and help them write results and we also used to be able to like give out an
experimenter of the week trophy which was really fun because it ingrained yeah it was like it became
part of the culture and i think the most special part was i so i left the company five weeks ago
and what was really special is since i've left i've seen people posting on linkedin that they
wanted the experimenter of the week trophy and i'm like that is so nice to know like i had a little
bit of impact on the culture of that company and that they were able to do it and i think that's
the trophy that we started is now still going even after i've left i felt that i was able to
then go and do my own thing now because the team so i had um three people within the experimentation
team when i left which was big for us because we're only a team of 100 but i had only a team
of 100 that's like the biggest team we've had on the show but we are we do have lots of shows it's
not just the dire of a ceo what is like the turnaround time for a video between the time
you record with the guests and publish it like how many weeks is that and then like you experimented
on this day and it took this long to get back to you every guest was different sometimes like this
may sound crazy but sometimes we had a guest record on say a saturday night and it came out
monday morning whoa oh my gosh yeah i i can't even explain how little preparation sometimes
there was a long time but sometimes it was very tight turnarounds and like when we used to do the
we had to wait for the full episode to be ready and say the full episode so we used to release
um like 6 a.m and 8 a.m on monday morning sometimes the full episode was ready at midnight on monday
which meant we had to translate between midnight and 8 a.m going live a lot of like even before we
even got to the episode our guest booking like or um production or pre-production teams used to
pre-test guests where possible as well so they would do heaps of research and testing on like
will this guest perform will this guest perform will this guest perform will this guest perform
will this topic perform is the topic even interesting and are people searching it
and so there was things that happened before even the recording a lot of the time do you think of
diary of a ceo as a podcast or as a youtube show we stopped calling it a podcast maybe in my the
end of my first year and we called it a show because number one podcasts are not well they're
seen as like audio only even though they're not anymore and so we wanted it to be a show because
that was like four years ago we started calling it um a show but also because there are so many
elements that are not just a podcast anymore like the visual elements on screen the props it's like
definitely a show not a podcast and i think everyone should call themselves a show now
if they're doing video as well one thing that you mentioned that kind of like piqued my interest was
you're pre-testing guests and you're pre-testing topics and when i think about youtube you know
outlier theory is very very known and popular and that makes a lot of sense for talking head
creators or you know any kind of podcast or whatever you want to call it but i think it's
but for a podcast slash show i i haven't seen many people do that kind of research and then
go and find the guests is that what you're saying happened you would you would identify
how the heck do you identify whether a guest is going to perform or not so it used to be very
manual also to preface as i said before nothing is ever perfect 100 at the time because you can
see a guest on one show be an outlier and perform 10 times better than their usual views and then
other show completely like bomb out and not even get the average views for that show guests are
very dependent on the audience of the show as well not just that that guest is an amazing guest it's
like there's so many nuances to performance but again killing the guesswork i don't know how many
times i will say that it was like it was a massive part of like how can we find as much data as
possible before we even like send the guest as an option to that host and then one day um we were
like why don't we make this an actual show and then we were like why don't we make this an actual show
platform that we can use and so our data team built that but one of the platforms yeah one of
the platforms that our data team ended up building was a profile page of every single person that's
been on shows on youtube and say you wanted a guest on women's health you could type in women's
health and it would come up with heaps of different options of potential guests for that topic or you
could go even more niche let's say like diabetes i'm making it up but like diabetes and you wanted
someone that's an expert on talking about that you could type in this specific topic would come up
with all of the guests and then it would say here's all the videos they've been on they have
been an outlier on these many videos so say they'd been on 10 shows and they had overperformed every
single time on those shows that person would have like a percentage score that was way higher than
others and it also was not relative like say they'd been on a massive show that didn't matter
it was like did they overperform for the show size and so it was like a lot of people were like
it would rank it on that and we could see like does the the guest perform before they've even
come on a show and not to say like some guests like for especially for diary of a ceo when it
was like health experts some of them have never been on podcasts so it could have been the first
time so it wasn't the holy grail but it was helping us able to like pre-analyze anything
beforehand of people who had been on shows and and then there was so much other data that came
into it and testing sometimes we would pre-test topics or things that we didn't know about and
potentially like lines from not lines that they would say lines of topic and beforehand to see
like is there interest in that topic from an audience and but all of it like every single
guest i would say was different and i'm sure our research and guest booking team will attest to
that is like some of it was relying on like for example a health expert if they didn't have a
profile we obviously couldn't use the platform that would analyze their previous performance
but we would then use other data to go like is their topic
about to take place and we would then use other data to go like is their topic about to take place
off and it kind of is like gambling in a way of like the risk taking of saying like okay we think
this topic is about to take off like i guess glp ones before they took off we think a topic is
about to take off how can we pre-predict that and get it at the right time when it is going to
skyrocket um on search and volume okay this is blowing my mind right now on our show you know
we're kind of scouring the internet and finding people that we think would make really interesting
guests so what you've just described has made me rethink how we're going to make our show
which is
which is how do we find really really interesting guests but also what's coming up in the zeitgeist
in social media and can we find someone who perfectly matches with that topic so that when
it comes out it hits the internet at the perfect timing yeah who's what are the topics areas you're
thinking of now a future guest i i'm always looking for people who work for youtube channels
or shows that have i don't know i would kind of set the benchmark at like 10 million subscribers
or more i learned it from becky who
is previously the head of youtube from ali abdal's team and when she joined ali abdal's team she said
you know ali had six million subscribers at the time that she joined and she was like if i want
to learn this platform really really well i need to be speaking with people who have 10 million
subscribers you know so that i i level up experimentations and failures are things that
i'm intrigued about it's something that i'm admittedly not doing enough of and so that
makes you a really interesting guest for this show and as long as i'm really really interested
i think it would make a great episode but now now i'm thinking what's going on here i'm not going to
be thinking what is coming up in like november and december around experimentation and failure
or something like that you know yeah there's so many like i was even thinking as you were saying
that i met a guy in another podcast i did and we were talking about instagram i know different
audience like instagram trial reels and he was testing like he had a red bull can in one and
then changed the color of the red bull can then did no red bull can and he was testing like tiny
tiny elements that maybe a team can do and he was like oh my god i don't know what to do with this
team with 10 million subscribers might not even think to bother doing anymore and i was like it's
interesting I think even hearing depending on like I think can you predict their success and
their growth but like get catching people as they're about to take off because they're still
in that element of like the testing like deep deep testing and learning to see what works and
how do they find that like moment and this rocket moment that catapults them forward but I just
thought it was so interesting I was like sometimes you can learn you learn a lot from people who've
already done it for sure but then also there's people coming up and Steve always used to say
he used to say this is um we want to make sure that the 18 year old in their bedroom or the 16
year old in their bedroom that's learning and just deep in YouTube like the old Mr Beast when
he was younger he will overtake us one day so how do we make sure we are always out working
out experimenting that 16 year old sat in their bedroom learning at the fastest rate possible
and that was like always the driver for me um and that's why I find it so fascinating to learn from
for me I'm like I want to learn from every level whether you've got like 10 million or like
10 subscribers at the time to see like what are you doing to get from 10 to the next 10 and keep
growing you brought up um Steve who was your employer for a very long time I'm really curious
what it was like to work alongside someone who is you know one of the most respected business
owners hosts in the creator economy what are some things you learned about excellence from working
so closely with Stephen Bartlett um you have to sacrifice a lot whether it is time whether it is
like missing out on something or whether it's like you know like you know like you know like
a lot of things to because Steve would always spend as much time as he could learning and
honing his craft and my favorite story is him and Mr Beast when they were recording the driver
CEO episode Steve said he is the only person he's met that was felt so much harder working than him
which I thought was fascinating because to me he was the hardest working person I'd ever met
and I was like if Mr Beast is harder working than Steve that is a different level but Steve
particularly like not only is he the most hard-working person but he's also the most
hard-working person I've ever met but he also cares so much like and when I say care not just
cares about the people he cares about the work so much to the point where like the one percent
it has been so ingrained in me and I think this is why he's been like Grace you can stay here for
as long as you want like at the start I used to in a polite way if there was something wrong with
something that was live whether it was a spelling error or even like an alignment issue on a
thumbnail whatever we would tell each other like I would say because I cared about the performance
and the overall assets and I would tell each other like I would say because I cared about the
performance and I think that's why Steve used to say thank you so much for feeling comfortable to
share and speak out loud when something's not right that's why I think I've been able to like
have such an amazing career at that podcast is because I've like really embodied that part of
him which was caring so much and like whether things blowing up and going amazingly or whether
something had underperformed like we would still care just as much on any performance of an episode
and I don't think it was just Steve I think at the start every single time I heard him say
when I started there was seven of us and every single person in that team at the start cared
so deeply and I it's quite rare to find that I think people who will do whatever it takes
for that episode to go live we interviewed the producer of Chris Williamson's show Modern Wisdom
and he said a lot of similar things that you just described Chris being exceptionally hard-working
and he said Chris really cares about the minutiae the one percent and Chris will take note of like
the top pinned comments and the top pinned comments and the top pinned comments and the top pinned
comment in a YouTube thing and he'll he'll make adjustments for the next time so it's interesting
this might be like a opening a huge can of worms but how how is like Stephen um or even you
personally like using AI yeah we've had um like multiple AI competitions within our team of like
having stints of where we will build things and then run them so like mid last year
we had our first AI competition and I had maybe used Chachi for
and this is so naive and but I'm not embarrassed to say I had zero AI experience and I'd used
Chachi and then for this AI stint we I was like okay I'm going to challenge myself to really
really build something and I built this TikTok translator that took our social clips that had
performed well on our English content uploaded them to be translated then wrote a copy for them
and put on the translated text then it would upload them to a Spanish TikTok channel and
then it would go live and it was basically this rotating thing and I built the whole thing myself
I had no idea it took me so long but it was the best challenge of like actually trying to use AI
and I think if people are still using it for like note-taking fine or like asking questions like the
most basic form but it's still a good first step at least you're using it but I think if you can
like having that test where you go like okay for the next two weeks whoever builds the most
innovative or most valuable thing in AI
you're going to be able to do that and then you're going to be able to do that and then you're
like okay for our team we will do this for you and I like it really changed what we did because
since that point of testing myself to use the AI I now use it across everything like whether it's
something super simple with like granola where you've told someone you're recording them and it
like can pre-look up who the person is so you can jump onto a meeting you don't even have to do
research it already says all of your previous context or about the guest or the person you're
speaking to then it can like take all of the summary of that meeting if you needed to write
an email it can help you write the email it can help you write the email it can help you write
the email like that is such a basic form of it let alone like we have been experimenting with
like the AI translated languages so like there's so such different scopes but everyone across our
team used AI because it's such an important thing like if I can take my job and get AI to do 100%
of my job for me then I can use all of that time to go and do other things for the company still
but go and do other work and like for Steve never saw that he was never like if you replace yourself
with AI he was never like if you replace yourself with AI he was never like if you replace yourself
with AI we will probably promote you it's like the concept of how I would see it it was never
feared it was always so encouraged what are you building now what are you experimenting with now
yeah it's um pretty scary because I only left five weeks ago so it's still very fresh and we ask why
you left or is yeah I yeah of course I um basically before I even joined the diary of a CEO I had
always wanted to start my own company which is why Steve obviously inspired me so much and then
when I moved to London I saw the job and I was like before I start my company I'm going to apply
for this and see if this works out and it did and I was like okay either way like amazing and so
Steve had known like even in my first meeting with him I was like I want to be an entrepreneur
one day and four and a half years later obviously it was the most amazing journey and I wasn't ready
to leave any sooner before that but I'd had so much self-confidence and like an itch to
do my own thing and basically long story short I had been doing talks about experimenting and
failing on stages for the last two years around the world like have done some in America and some
in Australia or kind of in different spots and I'd realized there was such a gap in the market
for people needing to know more about experimenting not just in the media industry like across every
industry and now like I've spoken at medical companies like in their away days to teach them
how do you experiment
more in a highly regulated industry versus like creators which is really fun to experiment with
but I kind of saw that and I was like there's this really big gap in the market and I don't
want to miss out on an opportunity to go and be that person to help educate people around the
world of how they can try new things and not fear failure and I was like now is the right time
because I'd just been given so many opportunities maybe manifested maybe the universe working in
a way that I didn't know how to do it and I was like okay I'm going to do it and I'm going to do it
and I'm going to do it and I'm going to do it and I'm going to do it and I'm going to do it and I'm
going to do it and I'm going to do it and I'm going to do it and I'm going to do it and I'm going to do it
and I'm going to do it and I'm going to do it and I'm going to do it and I'm going to do it and I'm going to do it
which is helping businesses shows teaching them how to experiment more and also helping them
through that experimentation which is like my favorite things are like production marketing
how do you experiment more within those areas in the company and then I also very crazy started a
YouTube channel because I could not leave YouTube and never be involved with it again so I've been
and this is the best learning journey for me because as talking about experimentation I'm now
like I have to experiment on this channel as much as possible and like test and learn and see what
works but I've been recording every day since I left because I really wanted to be honest and
show like what is it actually like when you leave a job and I think a lot of people especially in
the creator world say I left my job and now I have a million followers and I'm getting paid
heaps of money and this is what it's like and I'm like it's not really like that when you leave you
leave and you're like what am I doing on my first day I have 12 hours of more in the day of just
sitting here doing what and so I wanted to create that like real reality of what is it like when you
quit a job is it just you doing all these things yeah it is just me like my brother helps with the
YouTube editing sometimes and things but I love it I'm like it's so fun to work on all these things
so I don't want to leave anything and then I have another idea for an app but that's a whole
different conversation it sounds like that hard-working attitude from Stephen has rubbed
off on to you because that's a lot of things to be taking care of all at once yeah it is and it's
but I also am like at the stage where I'm like I want to I think for me over my career the thing
that has got me so far is saying yes to opportunities and so I don't want to say no yet
until I get to a point where I have to say no because of time and I I will say yes like someone
asked me do you want to come to Paris tomorrow and I was like yes so I'm going to Paris tomorrow
but I was like saying yes has got me so
So far and like being friends and connected with people, like even talking about
experiments the reason we thought of translations was because i was chatting with other people in
the youtube world which is why this podcast is so amazing is hearing from other people in the
industry is where you get further it really is it's not just like sitting alone in a room trying
to think of things that is one way but like actually brainstorming with people and sharing
and like i was at um one billion followers summit in dubai in january 100 go to it is the best i'm
not paid to say this it is the best conference ever because we were it was everyone in the
youtube creator sphere talking about what they're doing right now and i was like that is and like
talk about when you're saying like oh these are the people ahead you're right like they are and
it was fascinating to hear like what are they talking about translations was a massive topic
especially ai translations and like hearing about what they were talking about and being involved in
that to be like okay we should do this um or we should test this was really really valuable so
yeah those connections
you
but saying yes i need to probably say no more but i'm still on the saying yes screen
and you'll be at vid summit yes yeah yeah are you guys going we'll be there
yeah conferences are really important to my year because like you said like you're meeting all
these different people who you wouldn't see otherwise it's good for finding out what's
right and what's wrong and what people are talking about so i'm a big advocate of conferences too
and i hope to go to the one in dubai in january we'll see who is a creator that's on your radar
right now that's worthy of a shout out
oh i know exactly who this is it's actually a show it's called the art of the brand her name is
camille she's the host i think she's from canada actually oh let's go okay yeah and it is the most
amazing show because number one is because of timeliness like they release content for example
coachella week they released the podcast within the weeks of coachella about the coachella and
everything that was happening and number two is they're actually experts they're not people who
have just decided to launch a show about branding they're actual experts like they go and do
branding with nike and those sort of like big companies they have the expertise and also they
have the most hilarious personalities like it's a man and camille and they argue live on the show
and i'm sat there like i don't know if i should be listening to this so it's almost like a
combination of like expertise being super relevant and some like comedy like bts-esque in
there and it allows me to be smarter that's what i always think about in shows and i'd like i either
want to be entertained or i want to become smarter as you said before and so for me they are the most
amazing show that i'm watching right now do you know them have you met them no i haven't met them
in person but i have chatted to them before where can people connect with you if they want to
continue the conversation yeah i am grace experiments on instagram and then grace miller
on linkedin and both of those on youtube but if you want to see the most real version of me with
all of the failures
it's definitely youtube thank you so much for watching or listening to this episode if you're
new here please consider subscribing or leaving us a review on the audio platforms it helps us as
a very new show if you want to check out another episode just look through our library it's starting
to build up we're at like 40 episodes now and we have a new one coming out every single thursday
so thank you for being here and i will see you then
Podcast Summary
Key Points:
Pre-predicting performance through data-driven experimentation allows teams to identify high-potential guests and topics before production begins.
Experiments are guided by clear KPIs, focusing efforts on key goals like click-through rate rather than spreading resources across unrelated variables.
The team used pre-testing—such as thumbnail and guest performance analysis—to reduce guesswork and increase success rates, especially in early stages.
A dedicated experimentation team helped embed a culture of testing and learning, with accountability through shared results and the "Experimenter of the Week" recognition.
Key innovations like guest profiling and AI-powered translation allowed the show to stay ahead of industry trends and scale content growth.
Small teams can experiment effectively by testing one variable at a time, even with limited resources, focusing on measurable improvements.
Success came from combining creative intuition with data, ensuring experiments were both innovative and grounded in performance trends.
The culture of continuous learning and embracing failure was driven by leadership, accountability, and cross-team collaboration.
Summary:
The core of success at Diary of a CEO was a data-driven, experimental culture centered on pre-predicting performance before going live. Rather than relying on intuition, the team used analytics to evaluate guest potential, topic relevance, and content elements like thumbnails, identifying patterns that led to consistent growth. Experiments were tightly aligned with key performance indicators such as click-through rate, ensuring focus and measurable outcomes.
A structured approach—hypothesizing, designing, measuring, and sharing results—was adopted, with tools like guest profiling and AI translation enabling innovation and early industry leadership. The experimentation culture extended beyond production, influencing guest booking and content strategy through pre-testing and trend forecasting. Even small teams could adopt these practices by testing one variable at a time and fostering accountability through shared learning.
This culture was sustained by leadership, accountability mechanisms like the "Experimenter of the Week" award, and team-wide commitment to learning from both successes and failures. The approach not only boosted performance but also inspired broader industry practices, demonstrating that experimentation—when guided by clear goals and data—can drive sustainable growth across media and beyond.
FAQs
We use data to pre-analyze guests' past performance on YouTube, including their outlier history and topic relevance. A guest profile page ranks potential guests based on performance metrics, helping us predict success before booking.
The role involves designing, running, and analyzing experiments to improve performance metrics like click-through rate and average view duration, while ensuring all efforts align with key business goals.
We focus experiments on key performance indicators (KPIs) like click-through rate. All experiments are aligned with specific goals, ensuring efficiency and measurable impact rather than testing random ideas.
Yes, even solo creators can experiment by testing small changes—like thumbnails or transitions—on a single element. The key is consistency and reviewing performance to learn and improve over time.
We pre-test thumbnails and topics using real data from past episodes. We also use a guest and topic analysis platform that ranks potential guests based on performance history and topic trends.
Introducing translations early allowed the show to expand its reach globally. Over time, it grew beyond industry standards, especially as AI auto-dubbing became viable and audience engagement spiked.
Chat with AI
Loading...
Pro features
Go deeper with this episode
Unlock creator-grade tools that turn any transcript into show notes and subtitle files.