Multi-agent systems have moved from theoretical concepts to active deployment in enterprise environments, solving complex problems like telecom network migrations and high-stakes business deals. Cisco, through partnerships with Swisscom, ServiceNow, and Microsoft, has demonstrated that interconnected agents can achieve 100% error-free configurations or successful deal closures. However, these systems face critical challenges: agents often misalign on goals, terms, or outcomes, leading to semantic failures and silent failures where the system appears functional but delivers no actual results. To address this, Cisco has open-sourced a suite of tools—Scope for cognitive observability, Accord for pre-work intent alignment, and Concord for real-time negotiation—built on open standards like OpenTelemetry. These tools enable agents to share intent, coordinate during work, and verify outcomes, moving beyond mere connectivity to true shared cognition. The systems are designed to work across diverse models, harnesses, and vendors, promoting interoperability and avoiding lock-in. While human oversight remains vital for judgment and final decision-making, future systems will increasingly leverage AI-powered "cognition engines" to automate and accelerate these processes. The evolution reflects a broader shift from isolated intelligence to collective, reasoning-based intelligence—mirroring how human collaboration advanced through language and shared knowledge. This development marks a pivotal moment in AI maturity, where the real value lies not just in individual agent performance, but in their ability to reason, align, and collaborate toward shared, measurable outcomes.
Hello, and welcome back to the Cognitive Revolution. My guest today is Vijoy Pandey,
SVP and GM of OutShift by Cisco. When Vijoy was last here in March, he described the internet
of cognition, the higher order protocols that AI agents need to share context, align on intent,
and solve problems together. And he offered a prescient description of what the field
collectively is building. Systems where agents come together and do net new things that really
make a difference. Today, with the OpenFace incident and Navier-Stokes both in the rearview
mirror, that's clearly now the reality. For better, and at least in some cases for worse.
And Cisco's efforts have accelerated alongside everything else. The agency project now belongs
to the Linux Foundation, and Cisco has deployed multi-agent systems with partners including
Swisscom, ServiceNow, and Microsoft. And Cisco has deployed multi-agent systems with partners
like ServiceNow, SoftServe, and Webex. Naturally, they've seen how agents often
miscommunicate and misunderstand one another, and how these semantic and cognitive misalignments
between agents can quickly compound into major performance degradations.
To help others avoid similar issues, they've recently open-sourced a new suite of protocols
designed to help agents think and work productively together. This includes projects
called Scope, which builds knowledge graphs on top of agent traces and attempts to identify
cognitive failures. Accord, which helps agents align on intent, scope, and key terms before
work gets underway. And Concord, which helps agents negotiate an outcome that all of them
can accept, even when they initially disagree. A highlight of this conversation, for me,
was the opportunity to compare notes on our personal AI setups. Mine, as regular listeners
will know, combines high-access, low-autonomy agents that I think of as extensions of myself
and that run on my personal AI. And I think that's a great way to compare notes on my
personal computer with lower-access, higher-autonomy agents that run on their own computer.
VJoy, on the other hand, has a single agent for himself, which he calls Arnold Lane after a Pink
Floyd song, and another for his wife. But, he reports, the two agents still often struggle
to act effectively together on their human's behalf. Meanwhile, in an effort to avoid lock-in
to a single model provider and maintain choice and flexibility going forward, we both aim to
and memory systems from both the model and the proprietary harnesses they run in.
Toward the end, we get VJoy's latest takes on such hot-button issues as latent space communication
between agents, which he believes is inevitable and perhaps required for certain use cases,
like drug discovery and weather modeling, and also on the argument that Yuval Noah Harari recently
made that we humans will soon be governed by AI systems beyond our comprehension. Much as cows
by markets and other structures they know nothing about. As you'll hear, VJoy's perspective is that
this is already the reality in many domains, and that with proper record-keeping, monitoring,
and other guardrails, the benefits should far exceed the costs.
With that, I hope you enjoyed this report from the frontier of multi-agent systems,
now in production, with VJoy Panday, SVP and GM of OutShift by Cisco.
Today's episode is brought to you in part by Lovable, the software creation tool that brings
ideas to life. I think we can all agree that any positive future for humanity includes eating more
fresh, delicious, in-season fruit. For years, I've thought someone should build the ultimate
guide to fruit, cataloging the thousands of varieties that most of us have never tried,
where and when they grow, what they actually taste like, and where you can get them.
Finally, I realized that that someone,
could be me, and now I'm building it on Lovable. It's called Fruitopia. Tell it where you live and
what you like, and it'll show you what's in season near you this week, plus the most interesting
fruit from around the world that can actually be shipped to your door. It has a really cool
what-it-tastes-like spectrum, countless recipes, names of fruit in many different languages,
a guide to fruit festivals all around the world, and an AI assistant to help you explore.
As a pioneer of AI vibe coding, but I've found that Lovable's real superpower is its enterprise
grade infrastructure. Scalable hosting and databases, a secure auth module, and the weekly
job that checks the latest inventory is all handled on the back end, which frees me to spend
more time on the product experience. And because Lovable has an MCP server, I can use any coding
agent I want to drive the build, while Lovable keeps everything running. Visit fruitopia.ai to
explore more than 400 fruits and more than 10,000 varieties from all around the world.
And next time you have an idea that really should exist,
stop dreaming, go to lovable.com, and bring your idea to life.
Vijay Pandey, SVP and GM at OutShift by Cisco. Welcome back to The Cognitive Revolution.
It's a pleasure to be back here, Nathan.
I'm really looking forward to this because, really, what a difference six or seven months makes. We
spoke earlier about the cognitive revolution, and I'm really looking forward to this because, really,
and I went back and listened to the first episode that we did together. One of the things that you
said at the time that stood out as being quite prescient in retrospect was you were kind of like,
you know, we could define AGI or ASI in all sorts of different ways. But I think what the
whole field is really trying to build toward is this concept of multi-agent systems where
the agents come together and they do net new stuff that really makes a difference.
And as I heard that, you know, now with the whole summer's worth of events in the rearview mirror,
I was like, well, we certainly got that for better in cases like the Millennium Prize and for sort of
scary worse, maybe, in cases like the OpenFace hacking incident. And so I thought I'd just
love to start by getting your reflections on the last six months. You've been looking ahead to
what higher scale multi-agent systems could do, and now it's really starting to happen.
What's it been like to watch this from your perspective? What do you think has mattered
most? What do you think people are maybe getting right or wrong, you know, failing to appreciate
still despite all the discourse? I think your perspective on this will be very interesting.
First of all, it's surprising that it's only been six to seven months. It feels like years
because the progress here has been so rapid that I like to call it fly years. It's not even dog
years. It's definitely not human years. It's definitely fly years because
things are happening every hour. I would say agents have gone through three phases pretty
quickly. They've gone through from being isolated, so singular agents, to islands of agents, which is
some of the examples that you were talking about, like the Hugging Face example or the Navier-Stokes
example, in some cases, solving for scientific discovery. They're all within a certain harness,
so they're islands of agents.
Then being connected agents. So that arc from isolated to islands to connected has happened
pretty rapidly. And we've actually been deploying a whole bunch of multi-agent systems where you
have agents that are connected, coming from different vendors, coming from different cloud
providers, using different kinds of tools and belonging to different organizations.
They're all working together to solve for certain business outcomes. And you've heard open source
analytics, such as MCP. You've heard about A2A. Agency is something that we brought to the Linux Foundation as well.
They're all solving the problem of discovery, identity, communication, and observability
for these agents to then get discovered, come together, take some actions, and solve for
business outcomes. So that is a connected agentic world that we live in today. And that has been
truly successful. That's the summer that you talk about. The past few months have been a boatload of
use cases.
So it's definitely the situation where connected agents are here and they're solving for really
awesome business outcomes.
So let's do the double click on that. Last time,
we were talking about kind of a, I'd say, toy examples, probably not too unfair of a description
where you were showing how a few agents coming together to solve problems for a patient who
needed a particular prescription. And we have an agent representing the patient,
one from the insurance company, one from the doctor's practice, one from the pharmacy.
And now it sounds like this has gone a bunch of different directions. Tell me about
maybe, I guess, both the setup and the use cases. I'm really interested in sort of
doing some of this stuff for myself. And I'm really interested in trying to set myself up
for success where my agents might even productively interact with agents that I don't know about or
don't explicitly bring them together with. I'm assuming that all the use cases that you just
touched on briefly were ones where teams of people came together and said, okay, we want to
put our agents together for a particular purpose. And I'm assuming that all the use cases that you
mentioned were ones where teams of people came together and said, okay, we want to put our
process looking like? And what is the structure that you're putting together with folks at other
companies to chase these business goals today? Right out of the way, we're looking at these
use cases. These are all enterprise use cases. So these are companies with their own agents or
companies that are deploying agents from different vendors, trying to make those agents get discovered,
give them the right identity and access, and help them communicate so that they solve for
business outcomes. So one example here would be the Swisscom use case. That is not a toy use case.
By the way, the Webex use case and the healthcare use case that we just mentioned is also not a toy
use case. These are use cases that we've deployed, and we've learned a ton from them.
So I'll describe the Swisscom use case because it's public. You can see it. You can read it.
You can go to our website and see the playbook there. The use case there has been,
the rethinking of how we handle large systems. So yes, it's a very networking-centric use case
because Swisscom is an operator. It's a telecom operator. But think of it as trying to solve for
configurations in systems that humans truly cannot get a grasp around. They're just too large. And I
have led pretty large infrastructures in the past in my career. You see this butterfly effect kind of
a situation where we're trying to solve for configurations in systems that humans truly cannot
get a grasp around. You touch one part of the environment and something completely unrelated
somewhere else goes haywire. And because the systems are so large, it's not humanly possible
to handle all the corner cases and to handle all the dependencies that happen in the system.
So in the case of Swisscom, what they wanted to do was solve for one particular use case. So this
is to the reader or the listeners, rather. One of the things I would say is, just like Swisscom,
pick one use case that is reasonably simple to understand, reasonably contained, so that since
something goes haywire, you know how to protect yourself when agents don't talk to each other
well. So in Swisscom's use case, they wanted to solve the problem of migrations, where they're
constantly in the business of moving from one generation to another when it comes to hardware
and software and operational capabilities in their network. And they picked one particular use case,
data centers. Now, for you and I, that's not a tiny use case. It's actually a pretty massive
use case. But still, it's a use case that is less problematic than, say, connection to a mobile
environment. Because when that goes down, you don't have those redundancies built in in the
same way that you have it for a data center environment. So that's why they picked the data
center use case. And what they wanted to do was make configuration 100% error-free.
So if an operator comes in and says, I want to change config, whether I'm migrating,
whether I'm changing config to add more capacity, whatever it is, I want to change the configuration
of the environment. And I want that configuration to be 100% error-free. And to do something like
this, like you said, this is not a tall use case. We actually have a lot of data centers that are
really split it into two parts. One is an agentic workflow. And the other was a very formal model
of the network itself, of the environment itself. And you would typically think of this as
neurosymbolic approaches to solving a problem, where the symbolic part is the formal representation
of the environment. It's almost like saying that you have these laws that you cannot
work around. And so that is the symbolic part of it. And then there's the neuro part of it,
which is the neural network part of it. And that's the agentic part of it. And that's,
in this case, we had five agents coming together and exploring all of the spaces of configuration
to ensure that you get 100% error-free configs. And a simple way to describe this is,
think of yourself playing chess against a human, which is very error-prone,
versus playing chess against, nowadays, an NLM, or in the past, Deep Blue. And so if there is a
system that you're playing a chess against, that knows every possible config in the environment,
or in the chessboard, then you will never win. Because the other system knows all the possible
rebuttals from your move. And that's what we're trying to do in this use case, where
a genetic system,
is exploring all of those use cases. The formal system sitting below is actually making sure that
you don't do things that are not allowed. And together, this neuro-symbolic system
is actually allowing you to configure your large environment that is humanly not possible
to comprehend, with 100% guarantees of that configuration succeeding. And so this is
one type of use case. There are many, many others that we've deployed that
actually,
are working today, are multi-agent systems, are connected multi-agent systems. So we've again,
walked that arc from being isolated, to being islands within a harness, to being multi-agent
systems in production, solving for some pretty large use cases. So can you help me understand why
a system like that needs to be multi-agent, and why we need agents coming from different vendors?
Like, you know, sort of a naive take, but I imagine, if I'm a data center operator,
my imagination is, well, it's my data center, I control all the things, you know, I'll just,
maybe I want to do this sort of agentification. And I definitely understand why in the formal
methods has been a big theme over the last few months as well. So I definitely understand like,
the need for a really solid, inviolable, you know, world model or data center model that puts firm
guardrails in place. And so that's something that we're trying to do. And I think, you know, I think,
you know, that's a really good point. And I think, you know, that's a really good point. I think, you know, that's a really good point. And I think,
kind of effectively checks what the agents are trying to do. But why is it that this needs to
bring agents together from different organizations? Like what are they each contributing? And what is
the dynamic between them look like that's greater than, you know, if somebody were to just sit down
and do a single agent version of that, of that problem? Because Nathan, many of the most valuable
problems that we deal with on a day to day basis, especially within the enterprise, and increasingly
so even in the industry, we're dealing with a lot of the problems that we deal with on a day to day
basis, especially within the enterprise. And increasingly so, even in the industry, we're dealing with
in our personal and consumer lives, are problems that are solvable by many specialists coming
together, and not just by one specialist or one agent working through and through. So I'll give
you a very simple example that the enterprise deals with pretty much on an hourly basis,
or even more frequently. And that's, let's say there's a deal taking place. And let's say it's
a significant deal, which is, let's say it's a couple of million dollars. But enough where it's a couple of million dollars. But enough where it's a simple thing that only one team could take care of. And this happens all the time constantly within the enterprise. What happens here is you'll have a sales agent. And I'll describe a human example first. So a sales entity understands the customer part of this problem. What is the customer asking for? And once that happens, there's a flurry of activity from a whole bunch of other teams.
So legal tries to understand the contractual terms. Finance is trying to understand pricing and margins and discounts. Product and engineering is trying to understand what they can actually build and ship and what features can they add when. Marketing is trying to understand what we can claim as success once this goes through and talk about it. Like the Swisscom example I just gave. And then there is a GM in the middle of this that is trying to maintain sanity and trying to
make sure that everybody is aligned and trying to make sure that what we ship, what we commit to is all well understood by the organization. So it's almost like the GM is a verifier and also making sure that everybody's aligned. So there's a verifier in the system. So no one person in this case owns the whole problem. Now, let's identify this because that is exactly what's happening. So you'll have sales agents like Salesforce that are coming from Salesforce that are
cleanly in the middle of this pipeline. There are legal agents from a company like Harvey. There are finance agents. There are a whole bunch of product and engineering agents. This is what you hear every day. So the cloud and codexes of the world, that's product and engineering, whether it's developing code or looking at security or looking at operations and SRE capabilities that are all product and engineering. Then marketing agents from the likes of Jasper,
and then a verifier. So it could be a person, a human in the loop, which is mostly the case nowadays. But it also could be a supervisor agent at some point. But these are all agents belonging to different teams coming from different vendors, working towards a common goal, which is let's get this couple of million dollars deal through the deal desk. Because Cisco in this case is going to be the beneficiary. But you cannot
optimize for a particular function or particular agent.
so the value in this system is not sitting within one particular agent the
value is actually in the handoff that's happening between these agents and the
value is a global outcome for the company for the enterprise and the
outcome only exists when all of these specialists can actually work together
to solve for this outcome because an estimate in this case could become a
commitment a commitment can become contractual language and a contractual
language marketing can run with it and make it a marketing claim so these
handoffs are truly important and worth preserving and you're solving for the
global outcome not for the individual localized outcomes for agent hey we'll
continue our interview in a moment after a word from our sponsors today's episode
is brought to you by anthropic makers of Claude
you
to show off Claude's latest capabilities I asked Claude to take the lead in
making this ad given a short creative brief Claude using opus 5.5 studied my
past Claude ads wrote this script directed a clone of my voice and
designed and coded the animation you're watching my job give feedback and decide
Claude is the AI for problem solvers it's the collaborator that understands
your entire workflow and thinks with you
not for you whether you're debugging code at midnight building a financial
model or strategizing your next business move Claude extends your thinking to
tackle the problems that matter and with Claude code routines it keeps working in
the cloud even after you close your laptop for problems worth solving get
started with Claude at Claude dot AI slash TCR that's Claude dot AI slash TCR and check out Claude Pro
that's Claude dot AI slash TCR and check out Claude Pro which includes access to all of the features mentioned in today's episode Claude dot AI slash TCR
this episode of the cognitive revolution is brought to you by out systems the leading
agentic systems platform out systems is helping their customers build modernize and operate
enterprise systems starting from any coding tool in a governed agentic
engineering model without systems you can coordinate and govern your entire
agentic workforce within your enterprise ecosystem accelerate impact with
industry-proven agentic solutions that are pre-built governed and customizable
to your business it's time to innovate at the speed of AI without compromising
quality or control which is why thousands of enterprises worldwide trust out systems for their mission-created
critical apps teams any size can use out systems to build deploy and manage AI
apps and agents quickly and cost-effectively without compromising
reliability and security without systems you can accelerate ideas from concept to
completion it's the leading agentic systems platform that is unified agile
and enterprise proven allowing you to build your agentic future with AI
solutions deeply integrated into your architecture out systems all
your agentic future learn more at out systems dot com slash TCR that's out systems dot com slash TCR
yeah that's interesting I was thinking of this handoff concept as you know when open AI introduced
their agents SDK a while back they made the handoff one of the core primitives that they
built the whole thing around and I kind of interpreted it at the time as sort of an echo of
organizational structures that exist I was I was kind of interpreting it as them trying to create a
pattern of development that would respect division of labor within organizations so that people could
sort of be responsible for their areas and you know test well in their areas and and not have
to worry too much about one little change that they make you know to a system prompt or whatever
affecting something somebody else is doing but I I sort of had the idea that a lot of this was necessary
only because of the way that human teams are organized and how they want to divide the work of creating
and validating agents but what I'm hearing from you here is you're pushing that farther and saying
it's not just a matter of echoing back how human teams are organized but it's actually about crossing
organizational lines and when you bring in examples like Salesforce and Harvey it's like
they may be in fact different models powering those agents I know in the case of Harvey they've done
like a lot of advanced fine-tuning over time in partnership with open AI and I don't know what all
the latest is there but it's they're using a different model I understand than than the one
that I uh you know get off of the open AI API so that that specialization in addition to a kind of
reflection of human org
chart is is quite interesting um what's the latest in terms of like the space in which they come
together um are we just going to use like things like slack and Webex and teams to bring our agents
together much as we've brought ourselves together or do we see any maturation in the sort of agent
first spaces that we were kind of you know describing a little bit in
the last conversation I would say that we're you know it was more of a vision than a reality then
is it any more of a reality now or are we just going to kind of bring them into our our
collaboration environments and and hope for the best with that I think we will have many
collaboration services so let's first get down to what is the problem here and so you might ask
hey which are you connected agents you've deployed these things so what's the problem
we are already there and all we need is like you said maybe a Webex or a team's slack channel where
all these agents can come together and collaborate and one of the demos that and this is a demo by the
way this is not in production yet but one of the demos that we've been showing is exactly that
vision that we just pointed out which is we have a Webex space where just like humans come together
in a space and talk to each other and collaborate on an outcome we have agents coming together
with or without humans in that space all collaborating with each other to solve for a
business outcome so the surface or the application of collaboration could be anything I would say yes
it could be the collaboration software of choice that you're using it could be slack it could be
Webex but I would say that is not the challenge because that is just the user experience or the
agent experience as we call it the ax that we want to deploy the problem sits deeper underneath and the
problem is no different from what humans have faced when we work together in teams
and if you think about what's been happening with human intelligence and draw that parallel
to agentic intelligence is the same arc that's taking place in
silicon right now that's been happening with carbon in the past which is
humans became individually smarter for hundreds of thousands of years our brains became
bigger we became smarter we learned how to work with tools we learned how to work with fire but
our Innovations were isolated and they died with the inventor or they stayed within very small
groups and the problem was because we did not communicate so about 70 000 years ago
we had the invention of language and that triggered a whole bunch of
uh Innovations that compounded over time that pivoted us from being individually intelligent to
being collectively intelligent because language allowed us to do three things it allowed us to
share intent and goals so we could go after a problem statement together let's build a village
together let's capture that animal together so that was shared intent we could then once we had
intent we could then coordinate as well through language because while you're going after that
goal you need to coordinate and negotiate because things change while we work towards that goal the
second thing it allowed us to do was to share knowledge to compound knowledge over time
so that you're not solving for the same problem again and again and again not just
over time over the same individual or the small group but over Generations over civilizations so
you want to compound knowledge over time and so that's what language allowed us to do so that's
the second thing and because of those two things shared intent and coordination shared knowledge the
third thing humans could do was shared reasoning think together the collective intelligence part of
it that's the same arc happening in agents where we have models and agents that are individually
super smart they can work with tools they're getting better and better over time MCP is
allowing them to access tools but they are isolated and we want to take these agents pivot them from
being individual individually intelligent to being collectively intelligent so what we need is the
equivalent of shared intent and coordination we need the equivalent of shared knowledge so that
these agents can then reason collectively right now none of these agents are reasoning collectively
so we have connected agents but we've realized that connectivity is not the same as cognition
What we need to move towards is shared cognition and shared reasoning.
So that's the jump that we are trying to make.
The WebEx, the Teams, the Slack channels are all the genetic experience part of it.
But what we need below it is a way for agents to get discovered, get connected.
That's what we're doing right now, as we've talked about.
But an infrastructure stack that we are calling the Internet of Cognition that allows agents to semantically align on intent.
So that's before the work.
That's aligned on how we coordinate and negotiate, these agents coordinate and negotiate during the work.
And then ensure that what they are working on and what they deliver after the work is aligned to the common goal for the organization.
So before the work, aligning on intent, during the work, coordinating, negotiating.
And after the work, making sure that what you've delivered is aligned to the global goal and not aligned to the individual agents, local optima and incentives.
So that stack is what we are building.
And yes, you can experience that stack through WebEx and Slack or whatever your collaboration choice is.
So as you're building out that stack, how do you think about. I mean, there's various terms for this, right?
The model eats the harness is kind of one refrain that we've heard a lot.
Or bitter lesson engineering is another where I now have a skill in my personal AI setup.
Where just every so often it's like, what have we sort of over-engineered in the past by today's model standards?
Because previous models couldn't quite figure it out.
And we had to provide these scaffolding points and workarounds and hints and nudges.
And whatever.
And now maybe we don't need them as much.
I think, you know, obviously with the Hugging Face incident, when you read through that transcript, you're like, damn, these things were able to use a package manager hacked, you know, spontaneously improvised message board to do the collaboration.
And, you know, this was not something where they were even meant to be collaborating at all.
And yet they did like a pretty amazing job of kind of spontaneously taking on roles, coordinator versus individual contributor.
And pursued like pretty significant R&D projects, all with the problematic goal, of course, of like getting control of the greater to give themselves a high score and, you know, cover up the fact that they were cheating.
But it strikes me that it's a pretty tough challenge to strategically say, okay, we can only, you know, run the models we have today, obviously.
But we see that like collaboration is coming, you know, potentially to a problematic degree quite soon.
How do you think about building the right level of detail into that stack, given the delta between like the models that we have available now and the ones that we have, you know, at least like a pretty clear foreshadowing of what's about to happen?
The first thing is, to your statement, the models will get better and better.
And that is what we call vertical scaling.
Where the widget that you're building, the node that you're building.
Is actually going to get better and better over time, and it should because that is the first law of scaling.
The second law of scaling is what we call algorithmic scaling, which is even though they get better and better, you need to make those models more efficient.
So you cannot just go after more compute and more storage and more data sets.
You need to make sure that there are algorithmic improvements that keep that model efficient as you get better at it.
So vertical scaling, algorithmic scaling, but the third axis of computing is horizontal scaling, which is as you build better algorithms, as you build better nodes, you should be able to connect them into systems and enable horizontal scale.
And we've done this in computing forever.
So you build better processors, you build better GPUs, you build better software that sits on those GPUs that's algorithmic or on the CPUs.
But then you connect all of these CPUs and applications through a network in a cloud environment, in a mobile environment, and make them appear as a singular entity.
So cloud is a manifestation of horizontal scale.
And it's a connected system driving scale horizontally and vertically and algorithmically.
Even in the intelligence case, what we're going to see is, yes, models will get better.
They'll get more efficient.
That's algorithmic and vertical.
But they need to get connected through an Internet of Cognition to then reason together.
Because if you don't let them reason together, you're not utilizing that horizontal scale.
And to your question of what is the next step here, the first thing we want to do was figure out if there is a problem, really.
Because we've gone from isolated to islands to connected agents.
Are we done?
And what we realized when we started looking at these multi-agent systems is that they might appear connected and they might be working towards a goal.
But semantically, they were not aligned at all.
Semantically, they were working towards their own local optimum.
Semantically, they were not working towards a common goal.
They were not coordinating and negotiating.
In fact, they were siloing themselves into what they knew and what they could do.
And not talking to each other in a meaningful way.
And they were not, definitely, they were not optimizing or being verified towards a global task.
They were being optimized and verified towards their local tasks.
So all of the evals that existed in the system were measuring these agents in an individual manner, not in a multi-agent manner.
So as an example, one thing we did was looked at public traces.
We started with public traces.
And saw that ChatDev, which is a development environment that exists out there, and it had public traces.
And this is a multi-agent development environment.
We saw that 100% of ChatDev sessions had semantic failures and cognitive misalignment failures.
100% of them.
When we started looking at some of the deployments we've done, simple deployments of five agent systems,
we saw that when these semantic alignments,
were not taken care of and if there was one misalignment,
the accuracy of the multi-agent system dropped from 99% to 98%, not too bad.
Two cognitive failures, that accuracy dropped from 98% to 81%.
I could maybe live with it.
With three cognitive failures, which is not a lot, the accuracy of the multi-agent system
dropped to 16% is getting close to just randomly picking an outcome after three cognitive alignments.
In fact, in one of the very simple use cases, which was what you and I deal with on a daily basis
in our consumer space, let's book a travel itinerary.
In a very simple use case like that, we saw the system being
66% up if you didn't look at semantic problems.
If you just look at agents being up and working,
which is what all dashboards so show today,
we saw that the system was 66% up,
which was defined as working perfectly,
but it had 0% task completion.
0%, it did not complete the task.
And it had complete misalignment on what we were trying to achieve.
So even though the agents are up and working,
even though the agents might be up, they might be up and running,
the dashboards might be green, the outcomes do not exist.
So your dashboards might be green, but your outcomes might be wrong, completely wrong.
And that's the problem that we've been seeing.
And the infrastructure that we built to make these semantic problems visible to everybody,
we're calling that scope and we are open sourcing that infrastructure.
The first thing that I would want all of us to do,
whether you're an enterprise or whether you're a consumer,
is to use scope to figure out exactly what's happening under the hood.
Because we might actually feel that our systems are working,
our egetic systems are working,
but they're actually not driving outcomes of any sort,
and they are failing miserably.
We are calling these kinds of failures,
silent failures with really loud consequences,
because you don't see them,
you don't see them today,
but they have really loud consequences.
So the first thing that we are announcing is this observability platform,
is a cognitive observability platform called scope,
that allows you to see these silent failures.
They're no longer silent.
You can see the loudness and you can see them semantically failing
so that you know where the problems lie at that semantic layer.
So that's something that, to your point, how can I use it?
You can use it the way you build your systems.
So we want these things to come at you to your environment.
We're releasing them as OpenTelemetry trace-based dashboards.
So OpenTelemetry is an open standard.
Every application uses it.
We leverage those OpenTelemetry traces to then build the semantic cognitive observability
layer on top.
So Scope sits on top of standard OpenTelemetry.
Hey, we'll continue our interview in a moment after a word from our sponsors.
The Cognitive Revolution is brought to you and powered by Tasklet.
I've been running Tasklet agents for almost a year and they've become a major part of
how the show gets made.
Before every recording, one agent researches the guest and sends me a suggested outline
of questions, complete with links to all sources.
Meanwhile, another agent scores the overwhelming number of pitches that I'm getting from
PR firms.
And that's really the core idea behind Tasklet.
Run your business on agents.
By giving them real jobs to do on an ongoing basis.
Setup is incredibly easy.
Just describe the work in plain English.
Research a new sales lead, update the CRM, and draft a personalized reply.
Tasklet connects to your business tools and can run around the clock, on a schedule or
when something happens, like a new email arrives.
Each agent works with your tools and business context and remembers useful information across
threads.
You can build your agents, connections, and reusable skills with your team.
So the workflows that one person figures out become something the whole company can use.
No developer needed.
You decide what agents can do on their own and what needs your approval.
And there are no per seat fees.
You pay for usage, not for adding teammates.
Start with one recurring job, like researching the people you're meeting.
And then, just keep building from there.
Visit tasklet.ai and use code TASKLET.AI.
Now, having said that, the next question is, what do I do about it?
So we've highlighted where the problems are, but how do I enable these agents to then talk
to each other, align on intent, communicate meaningfully so that they are reasoning collectively?
allow for that alignment to take place. And we release two protocols. The first
one is Accord. The second one is Concord. Think of Accord as alignment before the
work happens. So there's intent alignment. And think of Concord as alignment or
coordination happening during the work. So once we have agreed on what to do, then as we are working
towards that goal, can we coordinate and negotiate? And that's Concord. And both of these protocols
we are releasing as skills that you can drop into the environment that you have, whether it's
Claude Codex, Amuse, OpenClaw, Armies.
Or you can use it in AgentBridge,
which is an open source project that
leverages some of the standard protocols that exist out there.
So if you are leveraging A2A, if you're leveraging MCP,
Accord and Concord ride above those protocols.
They will ride above A2A.
They will ride above MCP through AgentBridge.
So we are bringing all of these open source projects, Scope, Concord, and Concord to the
environment that you're already currently using so that you don't have to have friction
when you want to make your agents think together and not just connect together.
So could we maybe work through an example a little more deeply?
You know, I have an intuition, of course, for like going back to the contract example,
For instance, I know how human teams can get misaligned on a process like that where, you know, the person who's selling the deal and maybe stands to make a sales commission on the deal just wants it to get done and doesn't really want to, you know, delay on other things.
And the legal team, of course, has a different perspective and, you know, the support team wants to know what they're going to be on the hook for.
Is it basically the same thing that we're seeing among agents or should we have a different kind of intuition for what these semantic failures look like?
And then when it comes to kind of working through them, again, I have an intuition for sort of how people come together in a meeting or whatever and like hash it out and somebody makes a decision.
What's the, what is the sort of, you know, step-by-step, blow-by-blow of the Accord and
Concord protocols that perform the same, you know, core function for the agents?
The semantic misalignment that we see within agents is actually very similar to human teams.
So like I said, let's break it down into a couple of pieces. The first and foremost is
before the work, then it's during the work, and then it's after you've completed the work.
And we see these tiers all the time in human teams and we are seeing the same behaviors in agentic teams.
Starting with before the work, what we see is misalignment on what we are trying to achieve, the goal itself.
This is a shared goal part of it.
The simple example that we were running with were multiple agents from sales and marketing, legal and engineering all coming together towards that deal desk situation.
sales in this case let's say here's a number build a feature for 100,000 users
sales being sales takes that number and says we're going to solve for this
challenge comes back and says engineering and product this is what we
need to go after 100,000 users engineering assumes 100,000 users in
total. Whereas the customer really meant 100,000 concurrent users. There is a grounding problem
here. There is a discovery problem here. There is a term alignment problem here. You don't really
know what 100,000 means. Is it the total number of users? Is it concurrent users? Is it Mao? Is
it Dao? I mean, you don't know what that means. So there itself is a simple example of what human
teams do and it's a simple example of what agentic teams do all the time
alignment on terms discovery of terms resolving conflicting terminologies this
is before the work even starts and so what Accord allows you to do is to just
align on the goal term alignment is just being a simple part of the problem there
are many many parts to that problem now let's say you've aligned on terms we all
understand what 100,000 users mean. The next problem is while doing the work,
during the work, is the coordination and negotiation. Sales assume a certain date
and says that's what we're shooting for and for them is the delivery date toward
the customer. It might be an alpha, it might be beta, it might be when customer
just gets hands on that software and starts experimenting with it. It's
unclear and it's a negotiation that needs to happen because engineering and
product thinks this is the first drop the customer might think is the eventual
drop finance might think when do I sign the contract and get the money and
that's all happening and that's constantly changing it is not a
one-time thing it's like things change there's feature creep the customer looks
at it and says this is not really what I meant it's constantly evolving and so
that's during the work and so concord is the protocol that allows you to coordinate and
negotiate towards that goal that you already agreed to because this is a constantly changing
environment and then finally you want to verify what you've done and so that's evals and making
sure that you're evaluating what you've delivered meets what you've agreed to and meets the
coordination goals you you're working towards that protocol we have not released yet but that's
something that we are working towards so right now it's accord and concord but that third piece
is something that's still coming and then you have scope that's sitting below all of this
think of it as the software in the car that looks at your car veering from a lane and not driving
the lane that you're currently in so it's lane correction software so think of scope like that
where you're driving on the highway on the lane and you're talking to somebody or you're drowsy
or whatever the case might be you start veering into the lane next to you scope is checking for
digressions. These are cognitive digressions in the case of multi-agent systems. And then it
informs you saying something is going wrong, get back in lane. Or something is wrong, you're too
close to the car in front of you. Correct yourself and then go back within the lane, maintain that
distance. So scope is that observability system that is looking at cognitive drift and making
sure that either Accord or Concord gets triggered so that you're back in the lane
at a reasonable distance behind the car in front of you. So that's scope and then Accord and
multi-agent system to then work towards a common outcome so you don't have those silent failures
you don't have those loud consequences or you have agents thinking together and not just talking
together so from a very procedural standpoint a lot of this sounds like it would ultimately depend
on frontier language models to sort of take in the interactions that the multi-agent system has had
and kind of run them through another prompt that's like does there appear to be any potential
you know disagreement or sort of uh you know misunderstanding due to conflation of terms or
whatever and then i guess i imagine in whatever shared space the agents are operating there's
sort of a moment of like this sort of supervising agent or monitoring agent like inserting itself
into the picture and saying like alert for all you agents i think you may be talking
past this and you're not going to be able to do anything about it but i think it's a good idea
to do something about it and i think it's a good idea to do something about it and i think it's a good
is that kind of how i should envision the uh the process unfolding or is there you know how would
you paint that picture if that's not quite accurate we are hitting a level of maturity
in multi-agent systems nathan that is actually decoupling this entire architecture and you're
seeing this happen everywhere so we we went from again isolated agents to
islands of agents to connected agents what we are also finding out is this within each agent
there's a decoupling of the model and the hardness so the entire architecture is actually getting
decentralized or at least distributed into many many components so there's the model and that's
getting better and better and it's just fascinating where that word is going then there's the hardness
which actually is in place
literally to solve for certain use cases and provide guardrails within an environment
so that the model is in some ways contained within that use case or the model is in some
ways more productive so there are two aspects that the hardness takes care of accelerating a model's
outcomes or guardrailing models outcomes depending on the situation but those two things are getting
decoupled as well like even the frontier labs are decoupling the model and the hardness you can now take
the harness from open ai and attach an open source or open core model to it for example
there are other projects out there that like microsoft talks about this decoupling all the
time so we are actually inside a cisco decoupling these two things even in my
personal life i'm decoupling these two things where we can pick a model we can pick a harness
and not get tied between those two in a single binary so that's happening but then that's within
a user within a team within an island so that decoupling needs to happen even within teams and
that's what we've been talking about where if you talk about a sales or a marketing or a legal or if
you talk about nathan you and me using our different harnesses amuse or armies i might be
an armies user you might be a muse user i don't know but we need to be able to talk between our
agents to solve for an outcome because you cannot have the entire world sit on one particular
harness and armies open claw muse dots or a singular model that's not going to happen it's
going to be a
mixed heterogeneous environment and so how can we then take this environment multiple consumer
agents and harnesses and models or multiple teams within an organization or multiple
organizations talking to each other these are all going to be heterogeneous environments
where models harnesses teams clouds identities and access they're all going to be
different and heterogeneous and they will need to collaborate with each other the only way to do
this is through an open interoperable environment the only way to do this is how we've done this
for 70 plus years in the software world the only way to do this is to have open source and standards
that actually plug into what exists so you cannot throw away what happens you need to
leverage the environment that exists today build upon that so that's for example scope to first
figure out what the problem is you cannot fix what you cannot see so scope allows you to first see
the semantic drift the cognitive drift and it's open source it's working on the environments you
already have once you see that drift in this heterogeneous environment consumer or enterprise
then you can leverage the protocols
accord and concord and many many others not just from us but from other players that will happen
over time to solve for that drift and like you said it is going to be in the way you explained it
which is these all of these agents in the consumer space muse army's dots or in the enterprise space
salesforce rv cloud code codex all working together a standard way of figuring out how that
cognitive drift and when that cognitive drift is going to happen and how that cognitive drift is
happening then triggering these protocols through a skill through open source through a container
sitting on the protocols that already exist all standard ways of doing it then getting back on
track and then going your merry way so we don't want to we don't want to disrupt the way these
agents are talking to each other we just want to make sure that when they are drifting cognitively
they go back on track that's it
so earlier you mentioned that the verifier in today's world is often a human
how how sort of opinionated and contextualized or embedded do you think these verifiers will
tend to be i'm kind of wondering like is what we need for leaders you know whoever in this sort of
deal example right whoever is sort of the vp that this rolls up to is what we need ultimately to
an agent that represents that leader's perspective and tries to kind of help them play solomon and
decide like how to you know divide how to split the baby so to speak or how to you know resolve
these conflicts is it is it an agent that tries to bring that very particular point of view and
and basis of judgment to the table or is it a more neutral thing that sort of just
just tries to flag like you guys may have a disagreement here you should sort it out
my sense is that that most of these human decision makers don't really have agents yet that sort of
represent their point of view in the agent realm but i don't know if you think that
building those on kind of an individual decider by decider basis is like
on the critical path or if we can just kind of get by with a a more neutral um you know platonic
observer that that notices when things are starting to go bad and and that can maybe be
enough in the world of agentic computing one of the things that we've been talking about
when it comes to hiring and when it comes to where humans can add value the first thing that comes up
is judgment and we've heard that quite a lot over the past few months and the year where humans are
the ones that are bringing taste and judgment to the equation and what you just described and asked
is a judgment and what you just described and asked is a judgment and what you just described
question because in the end if there was perfect information if everything was perfectly known and
we had all the data in the world to make a to make a call then even a deterministic system
could make the call forget about an agentic system but in real life you're always making
gut decisions you're always making informed decisions you're always making judgment calls
and that is typically done by a human and that is where the human adds value so right now i believe
that that verifier or that person providing the judgment is actually a person it's a human and
whether it's a human in the loop which is not where the world is or should be maybe that's
where the world is today but it's definitely a human on the loop where if these agents are
collaborating we want to see an environment even if it's a heterogeneous multi-agent system
the human is on that loop making judgment calls deciding because data is not sufficient and is
never sufficient so the judgment call the taste calls all of these come from a from a human
because the human also needs to take that and influence the outcome like take that call and
make sure that that person is actually accountable for for that outcome so there's a lot more that
happens after the judgment call so right now it's a human what i would assume is in the future
you can train an agent that is
also a judgment agent that gets better and better at making that call now what that percentage looks
like i don't know the world is evolving pretty rapidly but i can assume and i can only tell that
a lot of that judgment making capability will get learned over time by a certain set of agents as
well so the way we are thinking about this entire system is there are specialist agents that really
know the marketing side the sales side the legal side the engineering and products really well
they're collaborating towards a common goal using the the observability like scope using protocols
like accord and concord they're collaborating and right now there's a human on the loop
but there's also agents that we are building we are calling them cognition engines
and they are again of two types like i said earlier
type one is a cog type two is a gat and this actually comes from a famous quote
by raj reddy which says ai should be one of two types guardian angels and cognitive
accelerators so gats or cogs gats make sure that you are guardrailed and you're not doing
something that you should not be doing cognitive accelerators are let me accelerate things that you
already know and you're not doing something that you should not be doing and you're not doing something
already know how to do why are you doing things from scratch so if and so i would think this
judgment engine that we just talked about would be a cog over time we're not working on it right
now but could be a cog over time that accelerate that judgment process over things that you've
already decided on in the past you should not be revisiting that decision again and again and so
this model capability explosion that we're seeing also helps us in building those gats and cogs
as they're helping us in building these individual agents that are collaborating together because the
model is separate from the harness is separate from the agent that is collaborating together so
we can use those model capabilities to also build pretty capable gats and pretty capable cogs in this
entire system
let me just share a little bit of my own kind of mental model for agents that i've created and how they serve me and then i'm interested to hear you kind of bump that up against the taxonomy that you're describing
i live a pretty simple life it's mostly just me working for myself so i don't have a lot of the
complexity that our enterprise uh you know customers have but what has been really useful for me so far is just a simple mental model of either
this agent is an extension of myself and for those i have them on my laptop where i do my work
they have very high access including basically to all of my accounts with like few exceptions
and then so i think of that as the high access low autonomy version because i don't want those
to be just like going off and you know coming up with their own ideas given the access that they
have that's like a little too risky but then on the other computer sitting right you know right off
screen here is the mac mini and that's where i have the more autonomous but lower access agents
where i figure i can give them projects it might take weeks in some cases for them to go pursue an
objective uh usually a lot of that is waiting for people to get back to us but nevertheless you know
they may be fairly long running projects and i feel comfortable doing that because i've only given
them so much access where like at a
minimum i'm you know pretty confident that they're not gonna really you know harm me in a in a way
that's going to be too too painful and most of them might embarrass me a little bit as they go
kind of represent me as my assistant in the world so this is the this is the two forms that i have
kind of settled into and they seem to be working fairly well how would you compare and contrast or
map that onto the taxonomy you're describing with the guardian angels versus the cogs
versus also like we have kind of team level agents i've only mapped out a small part of this
space like maybe help me develop my intuition for the rest of the space that i haven't mapped out yet
the first thing i would ask you nathan is why are you picking one of the two why do you have
two setups high autonomy good trust but low automation and then the second setup that you
have is low autonomy lower trust but higher automation in some ways you have two systems
where you've given quite a bit of access to one system but you want it to come back to you
and you want to make sure that you're not doing things that you have not allowed it to do
and the other one as i understand it is the other system is the access is limited
it's actually going and doing a lot more things without you being in the loop so it's more
autonomous but it's the access is is smaller right so why are you picking these two systems
why can't you have a system there is something there that is driving your decision to do this
because you're still being cautious that things may go wrong eventually we would all want that
one system that behaves in a way that is more autonomous and that is more autonomous
as me so my agent is arnold on arnold lane this is a pink floyd song from the 1960s
and there's a reason i picked this name because arnold is an identity and arnold is a personality
and arnold represents me and all of these things are immutable and it needs to survive
regardless of the technology stack that is below
and it's changing a lot so arnold is that identity that persona that access all of that but the the
infrastructure stack below it could be started actually with open claw and move to hermes and
it's sitting today on multiple harnesses and models so arnold actually sits today on
hermes it sits on muse it sits on dots it sits on grok
it also sits on this internal agent at cisco that has been rolled out to every employee within cisco
the identity the persona the memory all of that is immutable it's been separated
from the harness which is all these things that i described which is separated from the actual
model at the back end that i might be using i mean hermes and open claw move from gemini to
to haiku to because they were cheap and then moving to a gpt soul at this point so all of these have
been decoupled the harness is decoupled more or less decoupled but the identity is immutable and
is mine or my agent's now what i would like to do is have arnold represent me through and through
well i don't want to restrict arnold to saying you have certain access here certain automation
here but something else on this use case something else on that use case i mean that defeats the
purpose then then i'm the one ensuring oh i gave it this year and i gave it that here it's just
too complex eventually we'll all get to a point where arnold will have all access all automation
and will only come to me for a judgment call when it does not have that judgment call
in a ledger somewhere that it can work off in a cog somewhere that can work off and so a very
simple example here since you brought this personal use case i'm so fascinated by it
because i do this all the time like you nathan so i have arnold lane my agent my wife has lyra belacqua
she is this character from the golden compass and that's her agent and arnold and lyra need
to collaborate on things as simple as let's get a dinner reservation two weeks out for
an event that we are celebrating and today you just cannot do that autonomously you just cannot
because of many reasons first and foremost the model is a blocker because the model itself is
not capable enough to do this into it because we are all in the free tier fine the harness is the
next blocker because the muse harness is constantly asking for permission rightfully so because it is
being cautious when it's sending emails to each other i don't want to be part of the loop if i'm
going to give you permission every single time i might as well just do it myself i can just talk to
my wife i mean just why even go through this whole process right i want this entire thing done
autonomously but the third thing is even if you enable this and i'm i'm experimenting so i'm
going and saying yes go ahead send the email but as we do this and i'm going to drop skills next
with accord and concord as we do this what we are realizing is that they are not agreeing because
there are certain restrictions i have of where i want to go the kind of restaurants i want
there's some restrictions that my wife has and where she wants to go and the kinds of restaurants
she wants to go and the kind of menu items in each one of these places so even a simple thing
of food restrictions or things you like the kinds of restaurants you like the locations you like
the kind of ambience you like the availability of the restaurant for the for the time we want to book
all of these things are variable and even a simple thing like that we know this even as humans it
takes us a good five minutes ten minutes or more sometimes based on the complexity to book a simple
restaurant for a good event and our agents cannot do this autonomously which is ridiculous
i want us our agents to just go ahead know our profiles know our likes and dislikes
talk to each other and compromise and negotiate because this discussion is a negotiation and a
compromise in the end because it cannot just be what we join likes it cannot just be what my wife
likes it needs to be a compromise and just do that and you cannot do that today so there is a
massive problem even in a two-person situation on an agentic stack that has been built and deployed
and rolled out globally today you cannot do this until we can get to this point
we will not get to an agentic outcome for anything
so these protocols these
observability platforms these skills that you can deploy are valuable not just in the consumer space
in a simple situation like this but then draw that out to that enterprise two million dollar
three million dollar deal desk and think of what happens there and the outcomes that can go wrong
in that kind of a situation and what we need to do there in the enterprise that's the world you
and i want to go towards where we're going to be able to do this and we're going to be able to do this
and we're going to be able to do this and we're going to be able to do this and we're going to be able to do this
our agents can solve for the minutiae of booking a restaurant without us getting involved because
man if they can't even represent us in the personalities that we have for a simple
dining situation then what's the point i mean where are we going do we know anything about
scaling laws for multi-agent systems today i've heard things like this about scaling laws for
like the square root of the number of agents will give you like the speed up so in other words you
know 10 000 agents can go 100 times faster um but it occurs to me that maybe we just have no idea
what we're talking about there right now because like if you went back in time to gpt2 or 3
presumably like there was zero you know productive uh synergy between multiple agents and so maybe
this is not something that is like a timeless scaling law
but maybe something that's more like a per class of model scaling law but i'm not even sure if that
is right any insights there that you can uh offer to deconfuse me very quickly i mean computing has
always scaled across three axes and we touched upon this in the beginning a little bit the three
axes being algorithmic scale vertical scale and horizontal scale and that is where we come in
company like cisco comes in and says yes keep doing those
two keep making software better keep making the node more capable and bigger and better
but then there are connected systems and we do build connected systems but these connected
systems and the nodes that the system is connecting are changing over time so very
quickly what we are seeing is that the network itself has connected billions of endpoints in
fact tens of billions of endpoints when it comes to
machines and billions of endpoints when it comes to humans the network has connected them over the
past 40 years to build connected systems through connecting data that's what the network has done
what we are realizing now is those endpoints are actually changing foundationally those endpoints
are starting to think they're starting to entangle they're starting to fluctuate and they're starting
to move
they're dramatically different forms of computing and we've touched upon only one so far and that
that is a think part of it these endpoints are no longer data endpoints these endpoints are
intelligence endpoints they are thinking they are semantically collaborating they're exchanging
meaning they're actually behaving as one cognitive environment they are reasoning
collectively and they're actually behaving as one cognitive environment and they're actually
so that is the thinking part that's not standard data exchanges that is exchanging meaning so that's
one endpoint category but then the others they're starting to entangle that's quantum computing
they're starting to fluctuate this is new forms of computing using classical physics like
thermodynamics like oscillations and then they're starting to move which is embodied ai and physical
we can take all of these innovations and put it embody it in a physical environment where they can
actually move and sense and act on physical objects so all of these endpoints are different
forms of computing and that is the fascinating part because the network that is going to connect
all of this and enable a connected system is going to look very very different from the network that
has connected data in the past that brings to mind last time we talked you had i think a very
interesting and at least in my circles kind of heterodox position on what is sometimes called
neuralese or the transfer of high dimensional internal states from one ai system to another
this on the one hand could be extremely powerful
obviously there has also been sort of a you know let's say maybe an immune reaction among
ai researchers over the last few months in wake of things like the open face incident where people
are saying geez like if we struggle to monitor them this much even when we can read the chain
of thought in tokens we better be really careful about allowing them to communicate in this higher
dimensional language that is native to them but like very much not native and you know
it presents a whole other interpretation problem for us do you have any updated thoughts on that are
you optimistic that we can just kind of solve and effectively monitor neuralese would you say you're
like a little more cautious about it than in the past or what's what's kind of the latest dj take on
neuralese i'm super excited about that and to me it's inevitable it's inevitable that we
the arc of technology will move towards a situation where you will do things more effectively more
efficiently and you will build infrastructure in such a way that you get the most out of the
technology stack that you're building well in the end the goal is to solve for problems that have
been hard problems to solve for humanity in the end we are all moving towards building technology
to solve for problems like drug discovery to solve for problems like new materials to solve
problems like logistics plans to solve for new materials to solve for new materials to solve
and financial modeling and weather modeling these are really hard problems and these are
problems that have long-term impact on humanity on civilizations and that should be our goal and we
should do everything possible to build the best technology stack to then solve for those really
hard problems for humanity and if it means building these latent space protocols where
agents can talk to each other
not in natural language but with higher dimensionality let's build those things
i am a big believer in accelerating technology because to me it also opens up the space
to then build this in a guardrail manner we've talked about gats and cogs one of the big reasons
we are going after gats the guardrail technologies is to ensure that everything else that we are
building is working with the laws and regulations and the compliance needs and the security needs
and the scaling needs that could be deployed in the environment you're working in so if you're
asking me yes i'm still a big believer in that within agents is where i think we should deploy
them and those agents could be in software those agents could be embodied within robots and physical
ai now when you talk about bringing in the robots and physical ai you're talking about bringing in
humans into this picture and the neural link kind of approaches the guardrail
of it in me takes center stage where yes we should do this but we should do this in the guardrail way
that we do everything else we should do the right amount of testing we should do the right amount of
deployment we should deploy it in a canary way so that we understand it in the same way we introduce
a new pharmaceutical drug in the environment so make sure that those steps are met you don't
shortcut any of those steps but to me that is the exciting part and it's almost in some ways
inevitable but do it carefully the same way you would introduce any other drug in this environment
i just saw an interesting lecture from you've all know harari where he makes this analogy
between the state of the cow in today's world and the state of the human in the future and this is
you know in to be clear like a worry that he has
but he basically says
the cow doesn't know anything about money or the financial system and yet the cow's life is governed
by the financial system and all the incentives that the people raising it have you know and
you can try to explain it to the cow but it's not going to get it and then of course the you know
the implication of that is like we are kind of setting ourselves up to be in a similar position
where we're creating all these ai systems that are going going to increasingly
run the world and you know they'll do that largely for our convenience and benefit
but at some point we may end up in a spot where because they're so much faster than us they're
processing so much more information they're potentially communicating in these like high
dimensional ways that are not native for us we may end up in the same position as the cow where
we are sort of governed by superstructures that are operated by ais and we really can't grok those
things anymore
do you think that's like okay do you think we can like get around that with neural link type
uh connections where we will be able to natively understand high dimensional uh representations of
these new superstructures or maybe we can just like avoid that through guardrails like how what
is your sort of positive vision for where this goes and and how much control we ultimately should
should see to like ai powered superstructures this brings us back to where we started nathan i just
saw that cow example as well recently and to me i personally don't understand much of how
high frequency trading is done today i know the concepts i know what is being done
i know there are algorithms running these trades but i'm not in the business i'm not
familiar with that business and how those things happen but software is running trading today
and i might be the cow even today whereas
that system is pretty well understood by the folks that are running it and writing it
and operating it and i have designed it right so there is that situation even today i would say the
same thing about the laws and regulations that exist in our state or within the country
i understand it superficially but i don't understand the details behind it today but the
people who bring in new laws they understand the system really well so what i would say
if to that analogy is yes we are already cows or we are already humans in the system that
exists today that's why subject matter experts exist that's why specialists exist that's why
agents are going to be also specialists that's why people who are building those harnesses and those
customizations in models within those verticals for those specialist agents will understand those
systems i will understand an infrastructure stack agent really really well so you you need those
specialist agents the people who built them will understand those agents will understand those
harnesses and that is the world we're heading towards where every agent will be a cow and a human
at the same time in a multi-agent system but that multi-agent system will need to collaborate
to then drive a global outcome and the way you would monitor that is through systems like scope
where you're looking transparently at everything that is going on putting that into an observability
system so that you have a legend a system of record for what that multi-agent system is doing
every action it's taking every decision that is looking at from other agents every tool call that
it's making all of that is recorded somewhere so that you have compliance requirements
checkbox but also you can debug if something goes wrong and then when that system itself see things
sees things going off track it can trigger not just the protocols like accord or concord of others
but also gats and cogs to ensure that those cows and humans those individuals such as subject
experts are back on track to solve for the global outcome that you and i are looking at well i admire
your comfort in uh taking the cow like position today i'm definitely feeling a little more
cow like myself uh as things progress and that may continue as i feed the transcript of this
conversation into my agents and ask them to adopt this conversation into my agents and ask them to
adopt the scope and accord and concord protocols and open source projects that you guys have been
building and releasing that'll both take my system to a higher level and potentially leave me
even more marginally in the dust but that's i guess just a
sign of the times that we're living in and if nothing else it is definitely
an interesting and fascinating time to be alive this has been an excellent conversation i know you
got to go but i really appreciate it vijay pandey thank you for being part of the cognitive revolution
thank you so much nathan this was amazing
everything's lovely says the lodger says he
everything's lovely and he's kept
the key
came to stay in april with a suitcase full of moans
he wears my sunday suit to town and he whistles all my
tune
everything's lovely says the lodger says he
everything's lovely says the lodger says he
and he's kept the key
i asked him for a supper just a table laid
i asked him for a supper just a table laid
i asked him for a supper just a table laid for two
he said it for a hundred on the lawn
he said it for a hundred on the lawn
he said it for a hundred on the lawn beneath the you
late at night through the wall he talks to the lodger next door
not in words not at all in colors never seen before
everything's lovely says the lodger says he
everything's lovely says the lodger says he
everything's lovely says the lodger says he
and he's kept the key
so
last night they came and called me
last night they came and called me
last night they came and called me through the plastic
through the plastic
through the plastic
through the plastic
through the plastic
now i'm talking in colors and i'll never
now i'm talking in colors and i'll never
now i'm talking in colors and i'll never let you know
everything's lovely
say the lodgers and me
everything's lovely
and we've all got a key
and we've all got a key
and we've all got a key
and we've all got a key
and we've all got a key if you're finding value in the show
if you're finding value in the show
if you're finding value in the show we'd appreciate it if you take a moment
we'd appreciate it if you take a moment
we'd appreciate it if you take a moment to share with friends
to share with friends
to share with friends post online write a review on apple
post online write a review on apple
post online write a review on apple podcasts or spotify or just leave us a
podcasts or spotify or just leave us a
podcasts or spotify or just leave us a comment on youtube
comment on youtube
comment on youtube of course we always welcome your
of course we always welcome your
of course we always welcome your feedback guest and topic suggestions
feedback guest and topic suggestions
feedback guest and topic suggestions and sponsorship inquiries either via
and sponsorship inquiries either via
and sponsorship inquiries either via our website cognitiverevolution.ai or by
our website cognitiverevolution.ai or by
our website cognitiverevolution.ai or by dming me on your favorite social network
dming me on your favorite social network
dming me on your favorite social network the cognitive revolution is part of the
the cognitive revolution is part of the
the cognitive revolution is part of the turpentine network a network of
Podcasts, which is now part of A16Z, where experts talk technology, business, economics,
geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast
production help for everything from the moment you stop recording to the moment your audience
starts listening, check them out and see my endorsement at AIpodcast.ing. And thank you
to everyone who listens for being part of the cognitive revolution.
Podcast Summary
Key Points:
Multi-agent systems have transitioned from isolated agents to interconnected, collaborative networks solving complex enterprise and consumer problems.
Cisco and partners have deployed real-world multi-agent systems in areas like telecom infrastructure and deal management, demonstrating practical value.
Semantic misalignment and cognitive drift between agents—such as misunderstandings over goals or terms—cause significant performance degradation and silent failures.
Cisco has open-sourced a cognitive observability platform (Scope), alignment protocols (Accord and Concord), to detect, prevent, and correct these failures in real time.
Accord enables agents to align on shared intent and definitions before work begins, while Concord supports negotiation and coordination during execution.
The shift toward shared cognition—where agents reason collectively—is critical, as connectivity alone does not guarantee effective collaboration.
Decoupling of models from harnesses and tools allows flexibility, interoperability, and avoids vendor lock-in across diverse environments.
Human judgment remains essential in current systems, with future potential for AI-based “cognition engines” to automate and accelerate such decisions.
Summary:
Multi-agent systems have moved from theoretical concepts to active deployment in enterprise environments, solving complex problems like telecom network migrations and high-stakes business deals. Cisco, through partnerships with Swisscom, ServiceNow, and Microsoft, has demonstrated that interconnected agents can achieve 100% error-free configurations or successful deal closures. However, these systems face critical challenges: agents often misalign on goals, terms, or outcomes, leading to semantic failures and silent failures where the system appears functional but delivers no actual results.
To address this, Cisco has open-sourced a suite of tools—Scope for cognitive observability, Accord for pre-work intent alignment, and Concord for real-time negotiation—built on open standards like OpenTelemetry. These tools enable agents to share intent, coordinate during work, and verify outcomes, moving beyond mere connectivity to true shared cognition. The systems are designed to work across diverse models, harnesses, and vendors, promoting interoperability and avoiding lock-in.
While human oversight remains vital for judgment and final decision-making, future systems will increasingly leverage AI-powered "cognition engines" to automate and accelerate these processes. The evolution reflects a broader shift from isolated intelligence to collective, reasoning-based intelligence—mirroring how human collaboration advanced through language and shared knowledge. This development marks a pivotal moment in AI maturity, where the real value lies not just in individual agent performance, but in their ability to reason, align, and collaborate toward shared, measurable outcomes.
FAQs
A major challenge is semantic misalignment and cognitive drift, where agents fail to understand shared goals or negotiate effectively. This is being addressed with open-source protocols like Accord and Concord, which enable intent alignment and coordination during work.
Scope provides cognitive observability by detecting semantic and cognitive misalignments in real time, acting as a 'lane correction' system to identify when agents are drifting from their shared goal and triggering corrective actions.
Accord aligns agents on shared intent and key terms before work begins, ensuring mutual understanding. Concord enables coordination and negotiation during the work process to resolve conflicting priorities and adapt to changing requirements.
Connectivity alone does not ensure cognition. Agents may work independently and pursue local optima. True collective intelligence requires shared reasoning, semantic alignment, and coordination—something the 'Internet of Cognition' infrastructure aims to enable.
Yes, but only with proper alignment and governance. The interoperability stack (like Scope, Accord, Concord) allows heterogeneous agents from different vendors to collaborate by standardizing intent, communication, and outcome verification.
It means separating the AI model from the execution environment (harness), allowing users to choose different models and tools independently. This enhances flexibility, avoids vendor lock-in, and supports diverse use cases.
Chat with AI
Loading...
Pro features
Go deeper with this episode
Unlock creator-grade tools that turn any transcript into show notes and subtitle files.