Tired of database limitations and architectures that break when you scale,
maybe it's time to think outside Rosen columns.
MongoDB is the database built for developers by developers.
It's ACID compliant, enterprise ready, and fluent in AI.
Start building faster at mongodb.com/build.
[MUSIC]
>> Hello, everyone, and welcome back to the Stack Overflow Podcast,
a place to talk all things software and technology.
I am your host, and today we're talking about AI,
but everybody knows not everybody's getting it right on the first try.
Sometimes you have to retrench,
rethink, and my guest today is going to be talking to me with that.
I'd like to welcome Asaf Elavik to the program,
head of AI at monday.com. Welcome to the show.
>> Hey, happy to be here.
>> Before we get into the AI stuff, let's talk about you.
We like to get to know our guests when we talk to them.
How did you get into software and technology?
>> Wow, so that was, I think over 13 years ago,
when I started my computer science degree,
and it was obvious for me since I was
fixed in computers for my neighbors as a kid,
that this is the direction I would want to go to.
Specifically AI, I think,
sparked me for the first time I chatted with a chatbot somewhere around 2014.
When you feel even just the sense of it understanding you was truly magical,
and it was obvious that this is what I want to invest
the next period of my life and the rest is history.
>> It definitely since what was in November 2022,
it was a lot of surprise, a lot of magic,
and a lot of people trying to find the business value in it.
You all tried a chatbot with monday.com.
Can you talk a little bit about what you were trying to do
and why it didn't quite work out the first time?
>> Let's take a step back and think about how we got to
build a chatbot and why we were so excited about it.
As we recall, two years ago,
we probably saw the first introduction of co-pilots,
which we started around engineering,
but then also around other professional workflows.
We felt that there was a big opportunity for
monday.com building a co-pilot because really,
there is just so much work data,
project management, dependencies,
documents, conversations happening across
monday within a specific project that we
believed that if there was a co-pilot in the world,
that you could just ask it any question you want,
or ask it to take any action you
want across a different projects,
it would most likely be very beneficial for users in
the sense that it would save
hours of manual work or manual navigation across different projects.
This was the thesis that got us to
build what we called back then, Monday Expert,
which basically, as the name goes,
it would be your expert across Monday,
and we spent many months building out this first version of it.
Because there were just so many challenges about
connecting it to across Monday information
and making sure that it's structured,
and when we released the product,
we were very excited.
But when we released it,
we actually learned that churn was very high,
obviously retention was very low,
and even more surprisingly,
while in the lab tests,
we were around 80 percent accuracy,
or accuracy I mean that 80 percent of the times you ask it a question,
it understands you and gives you the right results.
In practice, it was around 20 to 30 percent when we reached production.
Now, this required a very significant retrospect,
not only about the product outcome,
which was pretty much questions everything that we define as a thesis,
but it also questioned the processes.
How could we as a company build a product and reach 80 percent
in lab environment and then reach production, see 20 percent?
I think it's interesting to touch both.
We go forward and so I think,
but I'll just give you one example.
So one example is that when you build an open-ended conversation,
and basically this is what the Monday Expert was,
it's just an open-ended conversation,
you come in and you can ask anything.
When you build that kind of product,
it's very hard to create a golden dataset or evaluation ahead of time,
because what users could ask is infinite.
The irony is that when you define what is the correct answer in a limited way,
you're actually by definition creating bias towards a product
because you're optimized just for that specific use cases.
We use Monday here too,
and it's interesting that in the product,
you really conceptualize these processes at work.
You make them very literal,
and then you have this open-ended chatbot that tries to talk to it.
Was there a disconnect there between the formalizing processes
in the product and the open-endedness of the chatbot?
Maybe the answer that I'll give an example of one thing that we did.
I think we thought about it correctly,
but we didn't execute correctly.
So one of the main minds that I like to drive open-ended conversation
product development with is that we need to plan for the 99%
of conversations we will not know how to handle.
So basically, when you build an open-ended conversation,
from day one, you should assume that no matter how much we build,
in the end, we're not going to know how to take action,
99% is with a lot of big numbers, right?
What we did was we said,
"If I don't know how to do this action,
let's build this knowledge-based connector,
so that if I don't know how to do it as a Monday expert agent,
at least I can guide the user to how to do it."
So in theory, we thought about building this correctly,
but once we reached production,
we just realized that what we thought users would want to do
is completely different from what we thought we want to do.
So it just hit that knowledge base all the time,
and users eventually became frustrated.
So I think if you go to production and let's say that 70% of times,
users ask and you give them what they wanted,
and then 30% of times you tell them either,
"Hey, I don't know how to do that,"
or, "Here's how you can do that,"
that might be fine and be a good user experience.
But when it's 80% guiding the user,
it just becomes a support bot,
then also I tell you how to do things,
and then you miss the whole point,
and you add the magic moments that you have with Georgia PT
and other products,
because we have some expectation to go up
in terms of what users expect from AI products today.
That's created a big frustration,
and that's where the turn came.
Users will constantly surprise you
in the ways that they go off-label with your product, right?
You mentioned the sort of disconnect,
or the two sort of ways that you could go wrong,
like mistaking what the user wants
and the sort of technical hiccups in implementing it.
Which do you think was the deciding factor here for you all?
Definitely it was the first,
because if you have the best technology in the world,
but users don't know how to use it,
then it's meaningless.
And I think this is something that's very special
and different about Monday generally,
and not just AI,
is that this is how we think.
This is how we try to think,
at least in most of the products that we build
is obviously user-centered,
but mostly we don't get excited by the technology.
We get excited by adoption, the culture.
So when we saw that users are confused,
when we saw that users don't really know
how to interact with a product,
that's when we realized that we have to stop immediately
and take a step back.
So I could say that the technology aspect to it
and the fact that its capabilities were not as powerful
or didn't know how to do as much as we hoped it would do,
that was minor.
That was something that we brought back into the tech team
to kind of research and kind of retrospect,
but the deciding factor eventually was user adoption and confusion.
When you're starting over,
I imagine you're not starting at zero.
You probably have some of the AI implementation,
connections to data,
various things sort of spun up already.
How much of a foundation did you have
when you were back to the drawing board?
I'm saying luckily because I think with AI,
there's so much that could go wrong as you kind of build out.
And even if you do get things right,
but you started building in 2023, 2024,
everything changed again.
This is why I'm saying lucky,
but we planned the building blocks correctly,
meaning that we saw a conversational interface
as an orchestrator.
So basically we have this LLM powered chat
that its main goal is to take one user input on one side,
look at the different tools and skills
that could be on the other side
and kind of find the perfect match and execute it.
In that sense, that didn't change, right?
That worked well.
But I think what's mostly interesting
is as we built out this new solution,
how much product experience was a huge factor
into the early success we're seeing with this new version.
It starts all the way from the name.
So Monday Expert was already creating the wrong promise.
Let me ask it this way,
if I told you you had a Monday Expert,
what would you expect it to be able to do?
I mean, tell me how to use Monday.
Right, I mean, I would even emphasize on that.
I was hoping what you say is that
it would probably be the best go-to solution
that knows about Monday because we called it an expert.
Even the small things like what is the name of the product,
huge expectations and I think it's something
that we kind of learned in the process
is that we don't want to call something
that is very far from an expert.
So we changed the name to Monday Sidekick,
which already kind of creates this expectation
that you're in charge, you're the leader.
Sidekick is here to help you kind of complement
the kind of work you're trying to do,
but he's not the expert in any way, right?
So that's like from the small marketing branding side of it.
But what really made a huge difference
is that originally the way you would trigger the Monday Expert
was from the very top level nav bar on the platform.
And when we kind of thought about it from scratch,
we realized that what if that entry point
was on a board item or on a specific task and not high level?
Because what happened was is that when they open it
from a top level kind of context,
then again, users are kind of expecting it
to do anything they want.
But if I can open it from a specific task item,
then that already creates the kind of expectation
that we're talking in the context of this specific task.
So I can help you with this specific task that you open
and not broadly Monday.com as a platform.
It's interesting, we published something on our blog
about AI being a really good UI,
but I think the question has been for last year's UI to what?
Whether it's this conversation or it's agents,
it's a little thing that helps you do the things.
Like can you go at this agent and be like,
"Hey, post something on this other board automatically for me."
Yeah, and by the way, we are learning over time
that conversation is a means to an end
in the sense that it only gets you so far,
the kind of workflows or work tasks that you work on.
So I think what we're learning is a conversation
is great as a zero to one.
It's a great way of starting some kind of workflow
or task you're working on.
It's a great, very natural way of getting feedback
to a work that hasn't been done,
but eventually interface is critical.
So another thing that we've really changed
in this new iteration is add, exactly as you said,
we've added much more user experience and interface
into the actual conversation flow.
So for example, in the original version,
you can ask for an email draft.
And then it would just respond as a message,
just just like you have with charge APD.
Now, the problem with that is what we call
the last mile problem.
And the last mile problem is something I think
all conversational agents have in common.
And it's basically that I now have the draft,
but now I have to copy the draft.
I have to move some other interface.
I have to paste it there.
I have to complete the entire task,
and then I can send it.
And this creates two problems.
One is the contact switching.
So I might as well just go to Gemini
and ask him to do within the Gmail.
And I don't want to use Sidekick now.
And B is that for us as a company,
as long as we don't have a way of measure the last mile,
we can't really know if we were successful
in helping you complete the task.
So what we're doing now with Sidekick,
and as we release this new version,
built-in interfaces that help you complete tasks
end-to-end as part of the workflow.
So if you ask for a document, if you ask for an email,
you'd get that UI within the chat,
so you can finish the work and send out the email
instead of contact switching.
Yeah. Is this the LLM itself using tools,
or is it just saying here's a prompt
for you to use the tool yourself?
Definitely the LLM using the tools.
Yeah, and Monday has had a fair amount of automations
that you could set up in it.
And it sounds like this is the next level of automations.
Is that how you're thinking about it,
or do you see this as a different interface
to the Monday data you have?
So I think there's a few things.
So one, I think Monday, first of all,
in the very talk level, we see two main directions
that we are putting a lot of our AI investments.
And one is continue our already existing mission
of democratizing software.
So Monday has always been about bringing
the power of software to anyone.
And that's in the context of the workflow builder
that we've built, for example.
Is that anyone can create an animation before the AI era?
And with that, we've introduced Monday Vibe,
which is basically a new AI product
that allows you to build any enterprise-grade product
on top of your existing data.
So like lovable, but for enterprise, you could think of it.
The other side is our mission was to help you manage work.
We've now shifted to helping you do the work.
And this is where I want to kind of focus
the conversation on it.
So as we understand it, the AI has enabled us
to actually do work and not just help you manage work.
Now the question is, what does that mean for Monday?
And for that, we see two main paths,
which I think is also common for many
of the other companies in our sector.
One is a personal assistant.
We highly believe that there's a big comeback
into personal assistance.
We've seen them with Sierra and Alexa back in 2016.
And I think for many technical reasons,
they didn't really do a good job in helping you personally.
But we think that today with the current capabilities,
everyone should have a personal assistant.
It can be hyper-personalized to you, understand your context,
understand your work, understand your data,
and kind of really help you do your work better.
And for this, we've introduced the Monday Sidekick.
Now there's another kind of work, which
is the work that you've mentioned.
And let's take an example of a feedback collector.
So feedback collection is something
that many people across an organization
have as part of their role.
Usually, you want to call up customers
or people who have used your product or your service,
and you want to kind of get their feedback.
And this is a very tedious process.
And this is a kind of process that
doesn't require necessarily a personal assistant.
You probably want to have an agent
that you can kind of create an automation
that for every new user or some specific event
of a user who has completed some kind of workflow
within your product, you want to reach out
as a voice agent, call him up, gather the information,
give him a great experience around the conversation,
and then save that information back into a Monday board.
And for this, we recently introduced Monday Agents,
which is a completely new platform
where you can build any agent that can do the work for you
autonomously in the background.
Right.
I think a lot of the developers listening
will understand the sort of user research project of that,
where it's gathering unstructured data.
You have to do some sort of contact,
and you can automate that now.
And then you gather data, and you analyze it.
And all that can be done now with an agent, yeah?
100% I think generally the capability
of automating an understanding of unstructured data,
converting it into structured data,
I think is a commodity today with current technology.
And that is what LLMs have done great.
But I think the why now, for why, for example,
voice agents are so exciting for us.
And I'm sure you've probably used 11Labs,
or you've worked with other tools,
or have talked with Chagapiti real-time experience.
It's really groundbreaking.
Like, I think today for the first time,
like especially in the last 12 months,
we've reached a stage where voice agents can do
what we describe as a better job than the average human,
and I'll explain why.
Because we've actually tested out these voice agents
to kind of understand where the limitations are.
And we've realized two things.
One is that when I was trying to test it out,
I realized it understands language better than me.
Where I would not understand what someone is saying,
voice agents actually had better understanding
of what people were saying.
And I think the other thing why they're better is accent.
You could tailor-made a voice agent
with exactly the kind of accent and kind of voice tone
and as you want.
And I think that's rare at scale versus these cross-centers.
The voice agents, I think they are an amazing technology.
They're still a hill to climb.
I just had an incident this weekend,
had an AC guy over called a parts line
and he needed tech support.
Tech support was closed.
And slowly realized that this wasn't a real person
and it wasn't helpful.
And I think there is a bias
against this sort of automated voice agent.
How do you think we can develop agents
that we can overcome that bias?
When you say bias, you mean like the negativity
or impression I have as I talk with voice agents?
Yeah, I don't want to talk to a robot.
I want a person who knows what I'm talking about.
I can explain the situation.
I think in that case,
what you do is you kind of build out the agent.
And I think a lot of is actually prompt engineering
in the way that we kind of build those.
So the way it works today is just for context,
for listeners, technically,
it's the same way you would define a charge APT kind of,
define the tools, the system prompt, the messages.
And then you kind of connect that as an output
using some kind of voice API,
either live in labs or charge APT.
And I think what you want to make sure
is that the most important thing
is for the users to have a good experience.
Let's say you call up and you've got a voice agent
and you're like, no, no, I want to speak to a human.
Don't be aggressive.
Help the user reach a human.
Because what will happen over time
is that you're going to start to get at those experiences.
We're like, well, in the outcome, it did what I wanted.
Like I want to talk to a human.
And I think where we see frustration
is where the opposite happens,
is where you think that maybe what's good
is trying to be more aggressive.
I'm like, well, let's try to keep you before I actually go to a human.
Then you get more frustrated, you lose patience,
and then you kind of lose trust.
And I think a lot of it is around the conversation experience
because there's no other way around it.
I think a lot of folks, when they call up,
they want a pretty simple repeated answer, right?
And I wonder if the sort of solution here is,
like you said, a prompt engineering
to make it a much more narrow experience.
Don't try to have it be a person.
The first thing that we did and we've learned
around voice agents is introduce yourself as an AI.
This is almost, I think, for us, obvious at this point.
The last thing you want to do
is you want to leverage the fact that
these voice agents can sound superhuman
and treat them as such as they reach out to users.
I think that is maybe one of the best examples
of how to lose trust with your users.
And even more so is kind of set that expectation up front.
And I think it goes back to the start of our conversation.
I think what we've really learned,
especially, is how important setting expectation is.
So the thing about this way with interfaces,
you see that the expectation is created
just by looking at the interface
because you can see the boundaries.
You see what kind of pages you can navigate,
and when you have someone calling
or opening the conversation, there is no boundaries.
So just as we said, what to do with those chat conversations
is the same as with voice.
So, for example, when the voice agent calls you up,
make sure that right at the beginning it's like,
"Hey, I'm Linda, the Monday AI voice agent,
and I can help you with deep technical questions around
or help you ABC."
Like start with expectations before users
have the chance to ask things that you don't know how to do.
What's the sort of engineering effort
beyond just connecting the LLMs and the data?
Did you use any sort of standards like MCP or agent to agent,
or did you build it all yourself?
So, first of all, definitely we've leveraged LLMs
and existing voice models.
So, for example, 11 Labs and other LLM providers
to kind of complement the conversation experience.
What we've learned is, so there's two main challenges
that we've invested in as we kind of built this out.
So number one is back to what we said about user adoption.
You know, voice agent is something scary,
and even more generally, background agent is something scary
because you have to trust this system to go out on your behalf
or on your company's behalf and take actions for you.
So, we've really thought deeply about how do we put the control
in the user's hands.
So, not just about how do you feel comfortable
creating the agent in the first place,
it's also about what happens the minute after you publish it,
and it goes out.
What kind of information do I care about?
How do I want to create this kind of feedback loop
so that when I see the voice agent in action,
I can review the logs of what happened in that interaction,
and I can feed that, my feedback back to the agent
for their future conversations.
But one really important aspect just to summarize
was how do users feel they have full control
over the outcomes of this agent,
which I think it's always cool to show like,
oh look, I built this agent,
it does all these things by itself.
And I think this is where a good product comes in,
is in the end, someone needs to feel
that they can trust and control those agents.
And this is something that we've really put into place
across the product and where you put a lot of investment in.
And then secondly, is how do you build this agent builder
as a platform that can support any kind of agent?
So, we talked a lot with voice agents.
In the same platform that we've built,
you can also create email agents,
which have a completely different type of interaction.
So, for example, you can connect an email
and connect some knowledge.
And then you could now use that email on your website
and anyone interacting with that email
will start to converse with a support agent
that can help them with tickets
and you can connect tools,
as we described earlier,
so they can kind of like route to humans
or escalate or update your back CRM
about the user's complaints or user's feedback.
And how do you build a platform that knows
how to create any kind of different channels
and different integrations?
I think that was another challenging part
in ways I think unique about this platform.
You mentioned the control you give to the users.
How much control do they have?
Can they go in and write their own code?
Can they see the code of the agents?
Can they plug in their own agent
that when a new ticket comes up,
it posts a message in their favorite multimedia app
or in a game or something?
Like how much control do you give them?
The question of how much control,
I don't think there is a limit.
I think you want to maximize control.
So if I could give them 100% control, I would do it.
I think the challenge is that there is a certain percentage
of the agent actually taking an action,
autonomously, that you can't control.
So the mindset is not how much control,
but how do I maximize control
around the specific product experience
that I'm building with AI?
And do it safely, I'm sure.
Safety is a key of everything we're building around AI.
But to your question,
so control is anything from what kind of voice I want,
what kind of tone, the exact system prompt
and instructions that I will want to instruct
the voice agent as it creates the call,
you could test it.
So we put a lot of emphasis into the testing stage.
So we have this really cool feature
where you can have the agent call you,
the agent you created call you.
And then as you're speaking with it,
you can give it feedback, verbal feedback,
and it will update behind the scenes the system prompt.
So imagine it's really cool experience
where you're talking with your voice agent
and you're giving it feedback like you'd give a real human.
Like, look, you should not start with this,
you should say this more often, don't say that.
And it kind of learns as it's talking with you,
and then you can go publish it.
That creates that sense of trust and control, right?
So it's anything from how do I give full control
into the very specific kind of voice agent
I want to create all the way to as discussed,
how do I monitor and evaluate
after it was conducting its voice calls?
What's the future of AI on Monday look like for y'all?
We want to help you do as much of the work
as AI can do for you so any business can scale up infinitely.
And so we believe in the future
where obviously AI is here to augment
and help you be a 10x, 100x yourself,
which is what we discussed as a personal assistant.
But then we also want to continuously go
and help you become a 100x business.
And I think it's what's really exciting for us
is seeing the value that we're already seeing in SMBs,
for example.
We're seeing SMBs as one of the highest adopters for AI
just because we can see the value,
we can see how specific users are already doing 10x,
what they could have done before using AI on our platform.
And this is the kind of future that we're trying to build.
All right, it's that time of the show
where we shout out somebody who came on to Stack Overflow,
drop some knowledge, share some curiosity,
earn themselves a badge.
And today we're shouting out a populist badge winner,
somebody who dropped an answer.
They're so good, it outscored the accepted answer.
So congrats to Wilkes for answering
how do you provide a default type for generics?
If you're curious about that,
we'll have the answer for you in the show notes.
I am Ryan Donovan.
I edit the blog, host the podcast here at Stack Overflow.
If you have questions, concerns, topics to cover in the future,
et cetera, et cetera, et cetera.
Email me at
[email protected].
And if you want to reach out to me directly,
you can find me on LinkedIn.
So I'm Asalph Elovik, head of AIAtLondon.com.
It was great to be here.
Thank you so much for the time.
And if anyone has any further questions
about the kind of challenges that we discussed today,
you can feel free to reach out on LinkedIn or Twitter.
Thank you for listening, everyone,
and we'll talk to you next time.
(upbeat music)
PodcastTranscript AI