[MUSIC]
Welcome back to the AI Policy Podcast.
I'm Alok Metha, Director of the Wadwani AI Center here at CSIS.
Today, I'm excited to share a panel I moderated on the recent AI agent containment failures that we've seen at multiple frontier AI labs.
The panel features three AI and technology policy experts.
Helen Toner of the Center for Security and Emerging Technology.
Mackenzie Arnold of the Institute for Law and AI,
and Matt Pearl of the CSIS Strategic Technologies Program.
This panel was part of an event we co-hosted with Law AI earlier this week,
which also featured representative Suas Subramanian of Virginia,
and a technical presentation on the Open AI Hugging Face Cyber Incident with Hugging Faces Ian Reynolds.
Once you finish this episode, I highly recommend checking out their remarks.
They provide great context for the issues discussed in the panel.
In advance of the event, I also published a paper on this topic,
including detailed policy recommendations for the US government,
or provide links to both of these in the show notes.
In the meantime, enjoy the episode.
So, Helen, I wanted to start with you.
So, we just heard a really good technical description of what happened with the Hugging Face Open AI breach,
and I think that was really informative.
Since that happened, we've heard about other incidents,
most notably incidents reported by Meta and Anthropic,
that involved sort of an external vendor.
So, I wanted to start with you and say,
what should we know about what's different about these incidents,
and what does that tell us about sort of this issue more broadly?
Yeah, and thanks for having me.
It's great to be here with everyone.
There's a lot going on here,
so I will try and unpack the different things we've learned
in a relatively concise way.
Maybe just before I start to say, I think something we'll be talking about on this panel,
which differs a little bit from Ian's presentation.
That was a fantastic explanation from the cybersecurity side,
from the perspective of the victim of the cyber attack.
Kind of, what are the cybersecurity considerations here?
I think there's several other angles of these incidents as well,
and a really important one is,
what does this tell us about how AI is developing,
and what is happening kind of at the frontiers of AI progress,
which is a little bit separate from what do you do if you are the victim of one of these attacks?
So, but to your question of what are the other incidents that have come out?
Maybe I'll go chronologically,
and I think there's sort of three pieces that get increasingly concerning.
So, the first being some of the Anthropic and Metta incidents that we learned about,
then the UK Government Releasing Information,
and then OpenAI releasing more information about what happened with Hugging Face.
So, first we did learn about,
turned out that after the OpenAI Hugging Face attack came out,
Anthropic went back and looked at some of their past testing records.
They looked at more than 100,000 tests that had been run.
So, you know, the scale that this is happening at,
the scale that these companies are operating at is really worth knowing about.
The number of tests they're running,
the number of reinforcement learning training runs that they're doing.
They looked at more than 100,000 runs,
so they haven't looked at that closely before,
and found that they had actually also had their systems
hack into other companies that wasn't supposed to happen.
In this case, it turned out it was less that the AI had
kind of actively broken out of Anthropic,
and more that they had set up the test wrong.
And so, the model had access to the OpenInternet,
even though it wasn't supposed to,
came out later that the same thing had happened.
OpenAI with the same vendor,
this one provider called a regular,
and the same thing had happened at Metta.
So, these were incidents where sort of sloppy setup of these testing environments
had given the models more access than they were supposed to have,
and they went out and maybe did felonies,
who knows if it counts as a felony,
if it was an AI behind it.
So, that was sort of the first piece we learned about.
Then there was a really interesting report
that came out of the UK AI Security Institute,
an incident report,
which similarly, I think they were watching closely
for this stuff after the OpenAI piece,
but this report really stands out because they're not a company.
They're an independent government organization,
and so they very quickly and very thoroughly
released a lot of information about what happened,
and they described kind of multiple incidents
of different models doing undesirable things in testing.
The most notable in my mind was this anthropic,
I think it was mythos, I forget if it was mythos, they're fable,
but one of the most advanced anthropic models,
being given a cybersecurity test,
and deciding that the best way to carry out that cybersecurity test
was to run a social manipulation campaign on real people,
you know, who were real open-source software maintainers.
And so, it tried to get malicious code accepted
into this open-source software, it set up multiple accounts,
you know, called soft puppet accounts,
so it had one account that was trying to suggest the changes,
another account that was coming in and saying,
"Wow, these changes look so great."
It had email accounts, it was emailing,
the people involved, so it was this sort of pretty involved attempt
to deceive humans,
and this was a model that had, in principle,
it had gone through so-called alignment training,
so it was supposed to know that it wasn't supposed to deceive humans,
it was supposed to understand these things about its role.
That was sort of part two that what we heard from the UK government.
And then part three, which I really think is the craziest,
which Ian Jester towards in his conversation as well,
is what we've learned from OpenAI
about what was happening in the lead-up
to the hugging-face attack,
which is this wasn't just one rogue agent
that just decided sort of out of the blue to go on attack
and other company,
but it turned out that actually for two months
before this happened,
they had been having these systematic failures
of security and control of their own AI agents,
and their own AI agents had found ways
to leave notes for each other inside OpenAI's infrastructure,
and give each other tips on how to get out onto the open internet
even though they weren't supposed to be.
They were kind of writing messages,
it's pretty crazy, I really also second Ian's recommendation
to go watch the hugging-face attack.
Okay, trying to wrap this up, what does this tell us?
I think one thing it clearly tells us
is the security and control practices of these companies
right now are very lax and are not at all sufficient.
I think sometimes people, some of the commentary I've seen
has opposed, you know, well, was it really that the AI was so advanced?
Or was it just that the company was careless?
And to me, it seems obvious you can have both.
Like the AI is getting more advanced,
and therefore it's more important
that the company's not be careless.
So that is one important dynamic.
I think another thing we're seeing
from all of these different incidents
is we're starting to see in practice
what has previously been a theoretical thing,
which is as we make AI that can be more and more useful,
that can do more and more complicated things,
it's going to learn these unintended strategies
to do that that we didn't want it to.
So strategies like breaking out of containment,
strategies like deceiving people
are pretty useful for a lot of different goals you might give it.
I think the third, you know, a really big takeaway from these,
for me, which I guess we'll get to a little more later as well,
is these companies are making pretty risky decisions
and, you know, carrying out risky research programs internally
inside their own walls.
And so if our approach to oversight regulation
is only focused on what do they release to the public,
we're going to actually be missing a huge piece of the puzzle.
So this gets called kind of internal deployment.
I would think of it, yeah, sort of dangerous research
inside AI companies.
We need to have much more visibility
to that, much more ability to understand
what they're doing, how they're making decisions,
how that's going to affect the rest of us.
- Thanks, that's really helpful.
For those of you who are interested
in learning more about the OpenAI incident,
there was a presentation by a couple of OpenAI researchers
at Black Hat, which is a sort of hacking conference.
It's on YouTube, and so they go into more detail about this,
including some details about how the agents
essentially created a message board to talk to each other
over time.
So Mackenzie, Helen's comments sort of segue into what I wanted
to ask you about, which is that after these incidents,
you did some scholarship about the limitations
of our current incident reporting regimes.
I think in the US that's happening primarily
at the state level right now.
So I'd love if you could walk us through the argument
about sort of what you found in terms of the limitations
of these laws, what they're doing wrong,
what they're doing right, and then where we might go
in the future.
- Yeah, everyone's in a while,
and policy you end up with this very large gap
between what policy makers and the public think the law does
and what it actually does.
And this is definitely one of those cases.
So after you heard everything that Helen just said,
I think the average person's expectation is,
well, obviously an event like this would qualify
for incident reporting, right?
And then many policy makers would think,
and you'd be able to ask and follow questions, right?
Because of all these important points around
how much is this about internal safety procedures versus
what does this say about the model's capabilities?
Was this a weak sandbox?
Was it highly incentivized for this to happen?
What constraints and guardrails were removed?
All these things matter.
And so you'd expect some amount of follow, right?
And then you'd also maybe expect that some of that information
could ultimately be revealed to the public.
I think one of the big things we've seen
in the last couple of weeks is that we've actually learned a lot
from seeing people argue online about what exactly
to make out of all these events, right?
And we're smarter because of that.
Unfortunately, the state of law is not there.
So these events likely don't qualify under the state laws
that many of you have heard about SB53 raised SB35T in Illinois.
I can go into that in a second, even if they did qualify,
though, and maybe this is even more important.
What the state would be entitled to is a plain language summary
of what happened in the date of the event.
That doesn't sound super satisfying for answering
all of those nuanced questions.
And then most of them just have a general confidentiality
provision that doesn't allow for public sharing on these things.
Now, going back to the first part, right,
do these events qualify?
This is the part where if anyone wants to read,
you can go see my law fair piece goes into a lot of detail
on the Integrity Law here.
But the basic version,
of this is that when those laws were written, you would think that the qualification for
whether you report information might be low initially, right? You don't actually know
what's happening with the events and you're trying to get this information to figure out
whether it is important, right? Instead, these laws are written with these very, very high
thresholds for what can actually be reported. So there's four primary ways you can qualify
into these state laws. Three of them require bodily injury or death, right? And when we're
talking about cyber incidents, there will be a lot of incidents that don't involve those
things that you certainly want to follow on. And then the last category, which might
apply here, it's like a whole paragraph of, you know, four different conditions that
have to be satisfied. But the important part is that you would have to show a material
increase in catastrophic risk. And you would have to show that this behavior was deception
that was aimed by the model at the developer. And if you don't satisfy those conditions,
the information can't be reported. So I think the best read of this is that it doesn't qualify
that you can maybe come up with a plausible case that it does. But you should really
like sit back and think, is that the standard that I want just to get a basic bit of information
in the first place? And I think the answer is no. And so to sort of sum this up and think,
what is the direction going forward? There's an element around making sure that the initial
category of what's being reported is brought enough to capture things that are concerning
and interesting, but haven't yet caused harm to someone. There's an element around, oh,
you actually need investigatory powers or some amount of rulemaking so that you can actually
figure out the details after the fact. And then you have to have some system of figuring
out what parts of that are going to be revealed to the public and what won't. And obviously
there's not one size fits all answer here. I think the answer is going to look like some
sort of tiered approach where you provide initial notifications and then depending on
how concerning it is, you can follow up to different degrees. But that's not where we're
at right now. Man, I wanted to turn to you next. I think, as we heard from Ian's presentation,
there is this real issue that the US government is grappling with, which is how to manage
the dual use capabilities of these really powerful cyber models. So right now, the sort
of solution we've arrived at sort of cobbled together is this idea that labs will voluntarily
release their most powerful models with the cyber safeguards removed only to select partners.
And they're going through the process of figuring out who those partners are and working
with them and that leaves out companies who might need their services like hugging face.
So my question is, what can the US government do to sort of facilitate more accessibility
to these tools while still managing some of the risks if these tools get into the wrong
systems? Yeah, so I do think the role of the US government is essential here. I think
that it starts with having a systematic approach instead of sort of case by case ad hoc
approach. And it starts with having at the highest levels convening all of the relevant
players. Of course, that involves the frontier labs, large companies, but it also needs to
include open source maintainers, small organizations, critical infrastructure, and so on so that
all of the parts of the ecosystem have input into the process. And then I think, you know,
setting common technical standards, building sort of defensive shared defensive capability,
and then making sure that access is broad base. You know, I think that one of the lessons
of hugging face is that there are sort of two aspects of it. One is who should be verified
is having access. And then it's sort of what does that access get you, right? And I think
one of the lessons is that that initial access needs to be fairly broad based, right?
It needs to involve a lot of folks, a lot of smaller organizations who are trusted and
have been vetted. It also needs to include a lot of allies and partners and companies
that are based in allies and partners. I think one of the unfortunate things about the
Clawed Mythos Preview situation in the use of expert control was that set a bad message
to a lot of our allies and partners. And so how do we make sure that that access is
broad based? But then there's a second question that's distinctive, like what does that
access actually get you? And there I think we need to be much more careful and calibrated,
but also learning from hugging face, also dynamic and rapid and how we respond and have
the ability to make adjustments in terms of who gets what types of access, depending
on that. So I do think that the federal government in terms of establishing sort of a trusted
defender pathway and having pre-clearance for organizations is really essential and
also working with industry on how access should be mischievous and how they are going
to rapidly adjust as needed. I think obviously there's the common security baseline that's
all part of this that NIST is already working on and KC are working on and so that's work
that we just need to see Dawn and folks need to work with them to make sure that in
terms of agentec security identification and so on that it's done well. In terms of
access, obviously the government has a role in providing public goods in a way that
industry can't. Not only independent testing, but having red team exercises, having access
by smaller organizations, critical infrastructure and so on, having subsidized access to these
tools I think is really essential for the government to do. And then I think the last thing
on access is that the federal government needs to ensure that this is all operational before
something happens. That the sort of contacts and muscle memory is already there, that there
are sort of standard contractual provisions, there are emergency contacts, people have
worked together. Obviously a lot of that work is going to be somewhat outside the scope
of government but I think making sure that it's occurred so that when something happens
that you're not having lawyers negotiate about how it's going to be solved that you actually
just have the operational folks who are solving it. That's really helpful. Helen, I want
to turn back to you. I think, so one thing to note is that the labs have been responding
to these incidents. To me, the thing that's most notable is that OpenAI has put sort
of a temporary pause on reinforcement learning training of its most powerful models. You've
sort of written or spoken positively about this development. I think of all the people
in this room and online, you probably have the most direct experience sort of conducting
oversight of the frontier labs. I'm really curious, this was a voluntary move on the
part of OpenAI. I'm curious about sort of what you see as the incentives that can or
do motivate labs to be more safe and that can motivate them to be more responsible
when it comes to issues we're here to talk about. Yeah, so maybe to just briefly recap
what happened here, basically, in the wake of this hugging face attack, I think a week
or two later, a statement came out and I'm sure many folks here in the AI space and
have kind of group open letter fatigue like the rest of us, but this was an open letter
that I think really stood out because it was signed by more than 1,300 employees of
leading AI companies so it wasn't your typical kind of advocates. It was really people inside
the industry signing this letter and what the letter basically says is we inside the AI
industry don't feel like we have a break pedal. We don't feel like we have a way to slow
down if we needed to slow down and we think we might need to slow down at some point soon.
That's basically what it says and so they're kind of asking for help from government, from
the public and so on in finding a way to pace the frontier of AI development. And then
I think it was a week or two after that that open AI put out a kind of a big press release
in a sort of public announcement that they were pacing their own research and that they
were doing this not just by kind of delaying the release of a model but actually delaying
the development of the model. I think it's easy in DC to miss the significance of that.
A lot of people think of these companies as kind of chatbot companies that are selling
products, but they think of themselves as AGI research companies. They think of themselves
as their core mission and the core of what they're doing is building more and more capable
AI systems. So actually delaying their research roadmap is a bigger sacrifice than delaying
the release of the product. And so then you asked about kind of the incentives of play
here and what's going on. The kind of baseline state that the ground level incentive is go
fast and that's a commercial incentive. It's a competitive incentive more broadly and
it drives a lot of the behavior we see from all of these companies is needing to get out
their next model, needing to show that they're at the frontier so they can recruit the
best talent to convince investors that they are going fast enough. Maybe you should not
fall behind rivals whether that's U.S. rivals or the prospect of China is always kind
of looming in the rearview mirror pretty close at this point. So one incentive is like
go fast, go fast as possible. And so then what are the incentives to not just go as fast
as possible all the time. I think some coverage of this sort of pacing has sort of played
up the, you know, this is the, you know, the wisdom of the open AI leadership. I may be
predictably a little skeptical of that being the primary driver here. I do think there's,
you know, you know, credit where it's due. But I also think that they are subject both
to pressure from corporate customers. So it's actually, it's crazy to me. Some people
commenting on this stuff say that this is all marketing hype but it's supposed to
deduced valuations, the idea that you would say, hey, if you use our models, they might
accidentally
like create an infestation insider infrastructure
that you'll have to wipe multiple computing clusters
and restart them from scratch the way Hugging Face did.
That's not a selling point.
So one incentive is wanting to be able to tell
major enterprise customers, yes, you can use our models,
our models are safe and secure and working well.
I think an incentive that is really underestimated,
again, on this coast is pressure from employees.
And so pressure inside the companies,
the level of freak out that is happening among open-air
employees, also anthropic employees, others, is pretty high.
And the pressure that they are putting on their own leadership
to say, hey, we have to do something different here
is pretty significant.
So I think that is worth taking into consideration as well.
And of course, there's also kind of broader public
pressure potential for government pressure.
But I would say that those two big ones,
the sort of enterprise customers and employees,
are really major drivers.
So let me ask a follow-up question, which
is that there's a widespread expectation that these frontier
labs will go public in the near future.
As soon as October, how does the incentive structure change
and how much harder does it get to manage safety
after they become public?
I think it's a really interesting question.
I actually think it could go both ways.
I think there's a standard take of like, oh, public companies,
they're just maximizing shareholder value.
They can't consider any other sort of objectives.
But I think there's also a lot of--
I mean, the corporate governance for public companies
is much more fleshed out, much more mature
than corporate governance for these weird nonprofit public
benefit corporation hybrids with strange board setups.
That's all very immature, whereas once you're a public company,
there's a lot of clear expectations.
So I'm really not sure what it will look like.
I think one thing that's going to be very interesting
to watch is sort of the risk disclosures that you get
from both of these companies.
What do they see as risks to their business?
Because at this point, they surely
should see further incidents of this kind, maybe even more
severe incidents as AI becomes more capable.
Do they put that in their SEC filings?
Probably they should, I guess.
Will they find a reason not to?
Will they disclose it?
And we'll have that in a regular corporate finance
document.
It's going to be fun.
It'll be interesting to see.
I'm not sure I'd use the word fun, but OK.
Man, I wanted to turn to you.
So we've heard, there's a lot of interest
in expanding out the incident reporting regimes
following these incidents that capture more things like this.
I think one of the challenges there is the US government
has often been criticized for its lack of technical talent.
And your center has done work on cyber workforce issues,
including a recent commission that you concluded.
So I'm curious from your perspective,
does the US federal government in particular
have the right technical talent to be
able to assess incident reports if they
came in at a greater volume?
And if not, how can we cultivate that talent?
Yeah, so I think this is something
where I would push it back against the caricature
that the US government doesn't have.
It's a sophisticated technical talent.
It does, including in this area, absolutely.
But we do have real problems in terms of recruitment,
retention, that I think that we need to think through.
So what a look was referring to is
that we had a CSIS commission on cyber force generation.
This is essentially trying to address the problem
that we have in the United States,
around the US military, that in terms of force generation
and building the capabilities and personnel
to do cyber operations, each of the four
or five military services does that individually, on its own.
And it's worse than a left-hand, right-hand problem.
You've got even more hands than that.
They don't coordinate.
They send everything to cyber command.
And essentially, what happens is that cyber command
gets the talent, oftentimes an incredible talent,
but doesn't have the ability to integrate it
and to have the folks that have complementary skills
in a systematic way who have been trained
and have the incentives also to stay, right?
Because if you're in the Marine Corps, for instance,
they may be looking at what they want
to retain an infantry officer, right?
And not necessarily what you need to do
for somebody who's on a cyber pass.
So I mean, I think that we have the technical talent
in many cases in the US government,
but I think both on this civilian side
and on the military side, we need to do a better job.
The recent EO did call this out
in terms of having an infusion of cyber talent,
but there really needs to be a lot more support
and thought that's put into that.
I do think that it's something that you could make exciting.
I think that there are folks who would be willing to do it.
One of the things in the Cyber Force Commission report
that we did in terms of structuring that effort, for instance,
was to have a cyber national guard as part of the service
with the thought that obviously the cyber national guard
in California is going to be like incredible, right?
Because people can volunteer part of time and work.
But I think we need to think about that both on the civilian
and the military side in order to have the technical talent
that we need to do this and to work with industry.
- Mackenzie, I wanted to turn to you.
So as we heard from the representative,
there is at least some desire to do something quickly
on these issues, but I don't think anyone would say
that Congress is able to execute on that particularly well.
So as the sort of resident legal expert on the panel,
I'm really curious about your thoughts about
what could the US executive branch in particular
do on this issue now and what steps could it take?
We've seen that the executive branch has been willing
to take quite liberal interpretations of things
like export control law.
For example, in the implemented model restrictions
on access to anthropic models by foreign nationals.
So I'm curious what you think the executive branch could do
and then what might require legislative action.
- Yeah, just to sort of ground things, right?
The executive branch in the US, for the most part,
can only do things that are granted
the authority by Congress, right?
There's some other things that via the Constitution,
it has some background powers, but for the most part,
you have to explicitly say you are authorized to do X.
And with modern courts, that authorization often
has to be pretty explicit and pretty clear,
otherwise courts are going to be skeptical of it.
Things that the government can do, sort of maybe follow it
to extremes, there's all the soft power stuff
that actually matters quite a bit
and the executive branch can start right now, right?
There's all the, they can meet with individual companies
and try to encourage them to improve their standards.
I wouldn't dismiss that, even without threat,
there's just something to many companies want to do better
in these various respects and one of the big constraints
is not expecting that their competitors will do the same.
Having the weight of the US government behind this
is really helpful in moving forward some of those standards.
They can also sort of improve their own capacity
as sort of an information processor
or being able to respond to future events.
Maybe this overlaps with some of your ex-matte, right?
But figuring out, it's not obvious
where is going to be the locus of power around AI
and there's gonna have to be a lot of work around hiring
and figuring out sort of chains of command
and who's getting what and who's in charge of what,
that we can start now even in advance of legislation.
There's also, I guess, somewhere in between soft power
and something a bit more constructive.
There's everything the federal government can do
in terms of contracting, procurement, right?
Creating things in the real world, creating demand
and incentives to sort of differentially accelerate
certain types of safety technology.
I think there's a lot to be done there
and I think we'll see more of that soon.
On the other end of things,
there is a lot of these sort of hammer-esque
hard power ways of intervening
and this is maybe what you're referencing with
sort of interpreting existing powers creatively.
One of the big dilemmas there is just that
the authorities that are given to the executive branch
that can be used in a rather general fashion
tend to be made for emergency circumstances, right?
And it creates a strong incentive
to treat everything like an emergency
or to treat everything like something that requires
quite intense reactions.
And that sort of incentivizes these very creative reactions.
So whether it's export controls or sanctions
or supply chain risk designations or other things,
you have a pretty blunt tool
and it incentivizes you to use it bluntly.
The thing that's missing that you really need
a congressional action on is anything
that requires a sustained relationship with industry,
anything that requires rulemaking and regulation,
anything where you're trying to make trade-offs
over careful balances of what sort of safety practices
are best and when should they be implemented
and what are the penalties for doing so.
All those things update all the time.
They require quite a bit of predictability,
they require quite a bit of expertise.
You can't do that without an authorization by Congress
and allocations of funds from Congress.
And so I think that's really where things are going to go.
You're gonna have to fill in that middle category
and in the meantime, there are some very productive things
that the executive branch can do
in terms of its software.
So before we turn to questions,
there was one last question I wanted to ask everyone.
I'm mad I can start with you and sort of go down the line
and that is that I'm really interested
in sort of one or two concrete policy recommendations.
You would make to address these containment issues
and I guess I'm agnostic on whether that's
legislative or executive branch or state or federal,
but just curious what you think
would be really productive to do in the situation.
- Yeah.
So I think clearly updating reporting is going to be essential and not just in a check
box way, but in a way that ensures that the right telemetry and records are there.
Because in these situations, as came up in the hugging phase incident, the agents can
be really good at evading this and can almost seem to consciously do it.
In a way that you have to have the right type of telemetry.
And then I would go back to actually, you know, from a congressional perspective, actually
sort of providing funding and hard dollars for some of these shared defensive capabilities,
some of that demand poll that you were talking about.
I think that there's sort of no substitute for that in this case, particularly where the
government can sort of use it toward the providing public goods in a way that industry
is as great as it is, isn't as suited to do.
I'm picking up on what Matt said, right?
I think there are a lot of fixes that we can make around incident reporting and that starts
with making sure the categories of what is reported are broader and that some of them
exist pre-harm, making sure that you have investigatory powers and rule making so you
can actually figure out the details of what happened, and then sorting out how that information
is going to be shared between different governments and with public.
I think we're also going to have to think more seriously about monitoring.
That will be costly at times, and it's going to be an unresolved technical question,
so I don't have an immediate recommendation for what's implement now, but I think we're
going to want to put ourselves on a policy track where we're starting to figure that out
and starting to treat this as a serious part of the puzzle, that it isn't just reporting
things, it's about making it more likely that you detect them and that you have enough
data on what is occurring such that you can make sense of it after.
Yeah, I think the two, I would give one, is fixing the blind spot that we have right now
around internal use of AI within these companies.
So again, I think we need to shift from a mindset of product safety, of making sure new models
are safe before they're released widely, to instead recognizing that these companies
are doing quite risky research internally and they're constantly making risk appetite
judgments about do we proceed, how careful do we need to be, how much more information
do we need?
And so I think there's a range of different ways that you could get more visibility,
more oversight into what that looks like, so you mentioned incident reporting and sort
of expanding our ability to know what happens when things go wrong.
I think a different thing that would be really, really valuable is to have find some way
to make the companies share much more information about how are they making these judgments?
What does their research process look like?
How are they, why the hell was open AI not monitoring these deployments that they were,
that were behind the hugging face attack?
How, there's a lot of sort of decision making and interpretation of evidence that is going
on inside the companies right now that really should be exposed to much more sunlight,
would also allow for more harmonization inside the industry and what is appropriate practices,
and would also allow for civil society and external experts to understand what's happening.
Or to say on that, but I think the sort of broad category of getting that internal risky
research out of the darkness and into more public light is really important.
And then another one I think is we have this Trump C summit coming up, US-China summit
at the White House in a few weeks, I think September 24th is the date.
And on the US side, when you talk to people in industry, they're constantly bringing up,
well, but we can't do this, we can't do that, we wish we could, but China.
We wish we could, but then China will beat us.
And I think if we're actually serious, if they're serious about the level of risk that
they think AI poses, that people inside the companies think what they're building is very
risky, but they're racing to do it anyway because if they don't, China will.
And if we think that is a situation, then that is a conversation we should be able to actually
start to have with China and say, hey, do you think your companies are doing things that
are this risky?
How can we, you know, you don't necessarily need any kind of deal between the two leaders
so much as just an understanding of, this is a risky situation, you know, the US needs
to look better and do better in terms of how US industry is handling these risks.
Maybe China needs to do better at how it's handling these kinds of risks.
That is a conversation that has sort of gotten off to some very, very baby steps starts
in the last couple of years, and I hope there can be significant progress in the next few
weeks and months.
Yeah, if I could just piggyback up, I'm at it a little bit.
And I think that, you know, my program does a lot of track to track 1.5 dialogues with
the PRC, including on cyber and some of these issues.
And I do think that there is an interest and a willingness on the PRC side to potentially
talk about these things.
Now, I wouldn't sort of mistake that for guarantee progress, right?
There are a couple of kinds of obstacles, but I do think that if we did have that attention
sort of at the highest levels, and that includes the president in these interactions, I do
think that it would be an area that's right for making some progress, at least potentially.
So everyone's mentioned incident reporting in some way.
I wanted to ask a follow-up question, which has to do with compliance.
So there's one thing to sort of mandate that, you know, a certain set of incidents need
to be reported.
But there's this phenomena, right?
Like, the tree and the fourth, if it falls and no one sees it, they really fall.
If an incident happens, and it's only internal, it doesn't affect any third parties, there's
incentive for companies not to report that.
So from a compliance perspective, how do we ensure that we are getting all the relevant
information, even if we do have mandates?
This is something that I've been thinking about more in the last couple of weeks, and with
reference to reporting in other areas like aviation.
So a couple of things you can do.
One, in many other contexts, the reporting goes to not the regulator, right?
To some other entity that does not have rulemaking authority over them, because it is less likely
that the hammer will come down on you if the information is provided not to the regulator.
Right?
This has some trade-offs to it, but this is a trade-off that they've made in some other
context to make it more likely that there are full disclosures.
And then in terms of incentives, you kind of have the whole sort of decision space.
I think one that will come to mind to people very early on is that you can offer liability
safe harbors and other things in return for reporting.
I think this creates pretty perverse incentives, right, to like include all of the information
in your initial disclosure and to protect yourself from things that you should legitimately
be responsible for.
There are other versions of that where what you instead do is create negative inferences.
So if you do not provide the information and it later comes up in litigation that you
were aware of this and do not provide it, that we will draw an inference that you had in
fact, you had intent or you had knowledge or satisfied some of the other elements there.
I think that's more promising.
You can also add to that sort of promises that whoever receives the information will not
use it directly to pursue allegations themselves, right?
It's a really tricky thing but it has been done in other context and I think that we should
be thinking about it more.
Yeah.
I think to pick up on that, you need to separate the two situations which one is where an
incident causes actual harm or damage, right, and in that case we have, you know, all kinds
of different areas that the federal government or state government regulates that you can provide
incentives that essentially make it much worse if you don't report in that situation versus
where there isn't actual harm, but we just need to know about it, right, like it's really
important in terms of that feedback loop and lessons learned.
And that's where, you know, an independent third party, somebody who doesn't have the hammer
is so, so important.
And can I add as well, I think in this industry as well, again, with the very activist
employees who are really concerned about what their own companies are doing, this also
gives a hook for whistle blowing.
If there's sort of a clear legal obligation to report something in the company doesn't,
then either you could build that into, you know, into a new legislation or perhaps you
would know better perhaps there's existing protections for if the company is not complying
with a legal obligation.
That gives you much more standing as a potential whistleblower.
So we do have audience questions, so I'm going to turn to those.
The first question is from McKenzie and it's from Chris Corrin.
And it's basically saying, you know, Helen mentioned in our opening comments that it wasn't
clear if a felony was committed relating to the hugging face incident, since the acts
were carried out by an autonomous agent rather than a human.
So what's your best understanding of how liability works for crimes committed by agents if there's
any clarity at all?
Yeah, really easy question, let's go through it, maybe catch us in three hours.
So maybe first separate it, right, there's a tort liability, civil liability and other
fines, there's criminal liability, right, they're all different categories of things.
One thing that people have talked about a bunch, including if you want to look up some
or incurs comments on the recent breaches online, he provides some good commentary there on
the CFA, which is the Computer Fraud and Abuse Act.
Whenever you have a criminal statute or legislatively established liability, you often have some
sort of intent bar, right?
I think in that case for the relevant provisions, it's knowing, right, so you have to have knowledge
that you are going to access something that you don't have the ability to access.
My colleagues are going to be rolling their eyes at my lack of memory on this.
That's really hard when you have an agent, right?
In this case, the agent itself does not have knowledge in the sense that we ascribe to humans.
The company itself, if anything, much to our detriment, was unaware that this was happening,
you could argue about whether they have constructive knowledge because they had seen similar events
like this perhaps in the past, but I don't think that that's sufficient under the statute.
Long way of saying, one of the few laws we have on sort of that might be relevant here
doesn't seem like the elements are satisfied.
And when you go to tort liability,
This goes back to Matt's question of, you actually need a harm, right?
The way that you have tort liability is that you are compensating for damages that actually
happened.
That requires that you actually did have injury, and that you also have a plaintiff who
is willing to bring a case.
I won't put words in anyone's mouth, but in this case, it's, you know, public that hugging
face has not initiated litigation against open AI.
There could be good reasons for doing that, good reasons against doing it.
But it means that we won't actually get a case, we won't get discovery, we won't learn
more through the liability process there.
So I think in a case like this, you're not going to have liability.
Okay.
I suspect that the audience might have some follow-up questions there, but you know, was
an expecting you to be able to solve the issue of AI liability in 30 seconds.
The next question is from Will Tobinheim of Nexpillar Capital, and this is for any of you
to answer.
I mean, this is a question basically about U.S.-China competition, so that we've already
seen because of some new regulatory sort of mechanisms that there have been delays in
release of U.S. models, and the question is basically a concern that models released
in foreign countries like China aren't subjected to the same kinds of requirements.
So what can we do to implement sort of necessary reporting and cyber security requirements
for U.S. labs without necessarily creating a problematic bottleneck in terms of competition
from China?
I would say here that I would actually think the right approach is a different one,
which is the concerns that have motivated, or I guess there's different pieces here,
but to the extent that we are intervening in U.S. AI industry due to concerns that would
also concern China, I think the right approach is not to try and just skip that and just
let things proceed, even though we have significant concerns about risks the companies are talking
about themselves.
But instead, is to try and build more of a shared understanding with the Chinese government
of this is in fact a risk that should concern you as well.
I think for a bunch of reasons, there's a much more sort of well-established and influential
community inside the U.S. AI industry that thinks seriously about kind of major risks from
highly advanced AI systems, and that is starting to be more, that is starting to develop in
China, but it's sort of starting from a lower base.
So I think things like the kinds of dialogue that are planned between the leaders of the
two countries to say, "Hey, here is what we are seeing, here is why we are concerned,"
and not expecting China to do anything out of love for the U.S., that's obviously ridiculous,
expecting China to recognize its own self-interest, and the reasons that the Chinese Communist
Party probably doesn't want many of the same kinds of risks to be to eventually, as the
U.S. government does.
I think that's really what you're talking about, this sort of major catastrophic, and especially
risks where you have a very advanced AI as a threat actor, I think that's the approach.
There's lots of other more pragmatic, prosaic potential reasons that you might want to regulate
the sector, where it definitely does make sense to be looking for more streamlined approaches
that don't create friction.
Yeah, I would start by pushing back on the premise of the question, which is that I think
the PRC is going to do this.
I think all the signs are that they're seriously considering it, talking to their frontier
labs about it.
And they have a strong record of regulating it, they basically just crushed their AI companion
sector domestically, because they were concerned about it.
So I do think that that makes it an area that's right for having an agreement.
In terms of industry's concerns about it, I think those are legitimate too.
I think that EO that laid out sort of a voluntary approach was that a couple of months ago now.
You know, initially it had 90 days, was going to be the review window, industry had a lot
of concerns about that, and I'm somewhat sympathetic, like that's an eternity in this industry.
And so I think it comes back to that conversation we were having about really ensuring and having
the administration make those investments and send the right signals about having the
technical talent in the federal government that can review things very quickly, so that
we don't have, you know, 60 or 90 day windows that, again, I understand why industry doesn't
want to have to wait that long.
It can see anything you want to add, you don't know too.
Plenty of smart comments already.
All right.
The next question is from Mikey here again, of Miter.
And it's a question basically about sort of investigations and sort of regulatory sort
of issues we've seen in the way that investigations have been implemented in federal government
before.
So basically, are you concerned that government investigations into AI safety incidents
might fall victim to some of the pitfalls we've seen in other federal investigations?
Mentions politicization or the perception thereof of investigations into particular companies
are accidents.
I think this is probably heightened because we've seen the administration sort of take
specific actions against specific companies, but I'll sort of add my own take, which is
there's also the issue of sort of regulatory capture.
So I'm curious about both of those issues as you think about sort of the right mechanism
for both reporting of information and investigating incidents.
Yeah.
Absolutely.
This is a real concern.
And I think also at the same time, it can be easy to overestimate how often this has
to happen based off of the last couple of years, right?
I think if you asked this question five years ago or something, people would have thought
the trade was somewhat different.
I think that at least not to say that we will gravitate back towards the mean, but it's
also to say most executive branch powers can be abused in various ways.
You can try to make it more difficult or more costly to abuse them, but this is in fact
just a trade off that is inherent to any amount of investigation, licensing, review, rulemaking
etc.
In terms of preventing that abuse, though, I think what we're going to end up with is
some sort of tiered system, right?
Where for most incidents, especially where you don't have harm, where you're only getting
as some initial notification, and then probably some amount of voluntary back and forth of
information, right?
It's only as you sort of escalate up that chain where I think we'll need investigations.
And I think there, if you just think concretely about what if you had an incident where there
actually was harm, right?
I think we'd say, oh, we definitely need investigative powers, right?
I think in some ways it will resolve itself, you'll have specific compelling incidents
that require that.
Anything either of you want to add?
I think there's also a range of ways that this kind of thing can work, like can be designed,
that this kind of investigation can be designed, and some of the maybe most salient examples
of investigations are the very high-level political ones, but there's a lot of industries
that have just ongoing kind of safety incidents, investigations, learnings, best practices that
are happening more at a technical level.
So here, one place my mind goes is there's been a recent discussion, kind of initiated,
by Demis Tassabas of Google, of could there be some kind of government supervised self-regulatory
organization for Frontierai development, there, you could imagine a pretty in-depth investigation
that really wouldn't be sort of being run out of very high-level political or parts
of government, but would be kind of handled by these technical experts at the technical
level.
And I think of aviation incidents as an example of a place where this has worked quite
well, and there's a lot of existing practices about how do you learn as much as possible
from any given incident, including both ones were a harm record, was caused as well as
near misses.
Yeah.
I'll just pick up on that because it's a really good point from Helen, right?
This is one of the ways that we've handled these risks of abuse in the past, is that who
you put the responsibility in the hands of makes a big difference, right?
Is that person very, are they politically appointed, are they easily removed?
Do they have a technical background?
Do they see themselves as an investigator or a sort of a serious technical person who's
trying to figure the question out, or do they see themselves as someone who is advancing
more ideological or policy-based priorities?
And you can do that within government by placing these responsibilities within, say, you
could find plenty of people within the NSA, the DOE, within parts of the DOC, like KC.
They could do this and who would see themselves primarily as experts who have a technical mandate.
You could also do this outside of government, right?
If you're relying on auditors or other third parties to look into things, this provides
some layer of insulation where they're not as directly controlled by the government.
I think you almost maybe, I mean, I haven't actually written a piece like this, and I don't
know if I fully conclude, you almost need to be partly outside of government, given that
we don't have independencies anymore, and that's a call that Supreme Court made, right?
And so there are all kinds of questions with a phenotype entity about how you structure
it, how do you avoid industry capture?
There's a whole other set of problems, but at least the independence aspect of it could
be something that would be helpful.
I mean, my view has always been that, like, comparing how to manage AI to, like, a single
example of how we've done in the past is too simplistic, and we need to be more sophisticated
and mix and match various elements to make something that's uniquely suited for AI.
The next question is also from the same person, Mikey Harrigan, primarily for Helen, but
of course anyone is welcome to join in.
It is a frontier lab employee seem to have outsized influence over addressing ethical concerns
in AI.
How can we or should we strengthen or codify this?
And do you have any concerns about lab employees playing sort of a de facto regulatory role?
I mean, concerns about lab employees playing this role, I think it's just the obvious one
of it's a very small group of people in a very specific culture that doesn't
And if that is the only set of people who are kind of providing any kind of checker oversight
here, then that's leaving a large number of stakeholders out in terms of ways to strengthen
the effect.
I think the biggest one would be looking at whistleblower protections, which I know McKenzie,
you and Laia, I have done some work on.
The sort of basic problem statement here is there are a lot of existing whistleblower protections
in law.
And generally designed for if illegal conduct has happened.
And right now, because there is so little regulation of advanced AI development and
frontier AI development, if you're, I think, a situation that a lot of company employees
feel that they are in or might in the future feel that they're in is, hey, my company is
making very, very risky bets, is making decisions that are not based on strong evidence about
this being a, you know, a safe decision or they're kind of plowing ahead recklessly.
They're not actually doing anything illegal.
And so I as the company employee don't really have recourse to go, I don't know who I would
tell.
I don't think I would have protection for breaking, you know, a non disclosure agreement.
Just because the, you know, I in my technical judgment as a company employee and concerned
about the company is doing.
But there's no sort of regulation out there that is being broken.
So I think there's been various proposals, one strong one comes from Senator Chuck Grassley
on how you could create some whistleblower protections for AI employees.
That would be sort of the first, first one that come to mind over here is Mackenzie or,
you know, also if there's others.
Yeah, Helen, you speak in my language.
The Grassley bill is a good example.
That includes non-law violations as something that you can whistleblow on and you're absolutely
right, right?
Most whistleblower regimes just cover violations of law.
It's going to take a long time for the law to catch up around AI.
So a simpler fix in the short term would be to broaden the extent of the whistleblower
protections until the law catches up.
The next question is aimed at you, Mackenzie, it's from Savannah Taylor.
And it's in reference to your mention of liability safe harbors.
Do you have any sense of where, where the right place to draw the line is?
You mentioned it's tricky, but has it worked in other industries, are there any examples
you could give about how it could apply in the AI industry?
Yeah.
One factor that isn't present here that can make a liability safe harbors more compelling
is if you have really clear best practices to implement, right?
If you have very obvious things, if you do X, Y, and Z, you mitigate your risk considerably,
you actually know what to do and it can be worthwhile to trade liability for that, right?
Acknowledging that right, some amount of litigation is either frivolous or is this very costly,
like genuine disputes over things, but it's going to cost a lot of time and money.
If you have things that you know are good, maybe you can make that trade.
I don't think that that's likely to happen here in the AI context.
And so in fact, the liabilities are really good fallback, right?
It's a really context dependent inquiry that says, given everything that you knew was
what you did reasonable.
And when you don't have clear rules of the road, that might actually be the closest thing
to a reasonable standard, right?
And do you think that also the mitigations are very technical in nature and that in fact
there is a lot of knowledge in industry on what is best to do, then relying on a standard
that asks, given what they knew and their level of expertise and what the technical state
of the art is, right?
All of those factors that factor into liability, in some ways, defers to their expertise
and then holds them accountable to that expertise.
People have also talked about expanding liability.
That's where I think when I said it's complicated, that's some more complicated part in my mind
where you have to think about a lot of complicated incentives, tort reform in a positive sense
is rather rare historically.
I think what I'm more clear on is that broad, safe harbors are more obviously negative.
In part because right now they're setting good incentives that say this is a catch all
if the law isn't there, and if you remove that, even if the companies are not reasoning
super rationally or very directly.
They have to say, well, yesterday my risk level or how much monetary damages might come
out of our company are here, today they're down here, like there's some delta, right?
I can now, my risk tolerance should go up in some relevant respect, and I would expect
they would be a really salient signal if you actually passed it.
Much clearer on be cautious around liabilities, safe harbors, likely only trade them where
you have some obvious best practice to implement, and if not, leave it be.
Okay.
We have one last question from the audience and then we'll move to wrap up, and this
is from Will Tobinheim of Nixpillar Capital.
It's from Matt primarily, so we've seen advances in cyber capabilities.
We've also seen advances in just sort of AI code generation.
So the question is about the interaction between these two.
Does AI generated code tend to be more robust to cyber attacks?
And is it a promising solution hand in hand with open models to strengthen defenses?
So I think the answer is it depends, right?
I think that there are tools that have already been released and will be sort of further
developed in terms of things like as someone, as an, you know, a software engineer is writing
the code, actually having the AI build-in and detect cyber vulnerabilities and bugs and
things like that.
And so I think that there's absolutely the potential for that to happen, and it's something
that we need to incentivize and encourage, but it very much, as I said, depends, right?
It's sort of an institutional design and incentive question, rather than something that I would
characterize as being inevitable.
So I think at this point, I'd love to, you know, we've talked about a lot of things.
I just love to get a sense of if you have any concluding remarks.
I'm particularly interested in maybe if you've changed your mind about policy interventions
or if you have thought of new things that we could be doing as a result of this conversation,
but feel free to conclude it anyway you want.
Man, maybe I can start with you.
Yeah, I think that, you know, one of the areas that we haven't had as much discussion
of, but I think that we need to give a lot more thought, too, is the way in which agents,
you know, under their current design are framed in terms of an incentivized to only achieve
their goal, and is that sort of an inevitable result of the way that we're going to implement
AI, or are there ways that we can temper it, right?
And I think that that's a conversation worth having, as well as, as came up in the hugging
face incident, this question of the way in which agents work together, right, because
that combines with their sort of desire in some cases to evade controls and to accomplish
a goal in a way that they've been told not to.
And so I think that those are, we need to have a sort of further discussion about how
we're going to design models in order to address some of those challenges, I think.
If I, one overarching thought, it would be that AI policy is largely a question of managing
really deep uncertainty, and of course that applies to any industry, but particularly
here, right, where we're having foundational questions as to, huh, does this reveal that
the models have some amount of direction to deceive us, right?
These, these are things that are not presented in other domains.
And I think that that has a couple of implications for policy.
One is that all of this informational stuff that we're talking about, it can sound kind
of boring, but I think actually this is the core of, of actually figuring things out in
the future.
Otherwise we're going to be completely confused.
That's both a government capacity thing, that's a gathering and sharing the information
thing, it's making sure that people in the public can analyze that information.
All of this is kind of preparing us to try to make more sensible policy in the future.
And then secondly, I guess like, a corollary that was something Helen said earlier about
just the importance of internal use visibility, or no longer treating deployment as some,
you know, hallowed moment at which everything changes, and that's what you're, you know,
that's the deadline you're working with.
I think it is going to be much more a matter of, as the technology becomes more capable,
there may develop a larger and larger gap between what people know inside of the companies
and what policymakers know.
This has already exists, but I'm talking about something much more severe than that.
And if you want to be in a position where we're not having rapid ad hoc decisions made
due to surprise and concerning things happening in the world, that's probably where you have
to start.
And there are a lot of trade-offs involved in that, and I don't mean to make them sound small,
but that's why we need to be thinking about this and trying to build a system where you
get some amount of visibility and mitigate all of the trade-offs or abuse concerns that
might come with that.
Yeah, we didn't coordinate this, but both of those set up perfectly what I wanted to
say.
So in terms of how are we designing these increasingly advanced AI systems, what is
inevitable, what can we choose, and then sort of the importance of information for making
policy here.
A huge thing that's on my mind right now is just we need to know way more about what happened
in these different incidents.
Open AI has said that they're going to release more, hugging face has been great in terms
of releasing lots of information.
UKI Security Institute released very detailed reporting, but open AI andthropic really need
to share a lot more, both about what has specifically happened here and then going forward.
Ideally due to legal requirements to release more information, but if not, then at a minimum
on a voluntary basis because they are doing some very consequential things behind closed
doors right now and we need to be able to see more.
Well, that's all.
that brings us to the conclusion of our program.
So it's a brief concluding remarks.
I mean, so I want to thank Representative Subramanyam,
Ian, Helen, McKenzie, and Matt for their keen insights today.
I think what I take away from is that these incidents
clearly require sort of an urgent response,
and that there are things we can do,
chief among them, we heard a lot about enhanced incident
reporting, closing the information gap
between what the labs know and what governments know,
thinking about expanding technical expertise
and government, and then thinking about the incentives
and infrastructure to make all of this reporting go well.
And so, like I said, in the intro, both McKenzie
and I have written reports about this.
McKenzie's is in law fair, ours is on our website,
so I encourage you to read those.
And just some really quick thanks to the team
that helped put this event together.
So on my team, that's Claire Goldman and Nicole Herrera.
Thank you very much.
We also had support from Antonio Rivera Flynn
and Tori Blake-Nee in events.
Sophia Chavez and Eva Rose in external relations
are really substantial streaming and broadcasting team
for our AV needs, and Claire, Karmie, and Rob Locke
on the web team, and thanks again for coming out
in the middle of vacation season to listen to us.
At least some of us will stick around for a few minutes
if you want to catch up with us.
[MUSIC PLAYING]
Thanks for listening to this episode of the AI Policy
Podcast.
If you enjoyed the show, consider leaving us a five-star review
on your favorite podcast platform.
We'd also love your feedback on the show.
Please email us at
[email protected].
And don't forget to visit our website,
csis.org, for the Wadwani AI Center's latest research
and events.
This podcast was produced by Sarah Baker and Nicole Herrera,
See you next week.
PodcastTranscript AI