Go back

Databricks CEO: AGI Is Already Here, Why Companies Are Behind, and the Superintelligence Debate

1h 2m 6s

Databricks CEO: AGI Is Already Here, Why Companies Are Behind, and the Superintelligence Debate

Ali Ghodsi, CEO of Databricks, recounts how the company began in 2013 out of despair that industry was ignoring his UC Berkeley AMPLab team's research on combining data processing with machine learning. Early bets on cloud, AI, and open source were contrarian but proved decisive, and the company now runs at roughly a $7 billion run rate growing 70-80% year over year. Ghodsi argues we already have artificial general intelligence, yet most enterprises use AI only as glorified search or basic coding assistance. He outlines four barriers to real adoption: capturing organizational context, security and governance, rising token costs, and lock-in risk around IP and data sovereignty. Databricks' Genie product addresses the context problem by building an ontology of a company's people, meetings, documents, and decisions, which he demonstrates through HR survey analysis, org chart insights, engineering productivity metrics, and customer meeting reviews. On superintelligence, Ghodsi is skeptical of recursive self-improvement, citing four unmet conditions: faster training runs, lower resource use, generalized accuracy improvement, and a continuous improvement loop. He puts existential risk near zero but calls cyber risk real and significant, arguing AI code scanning will eventually secure the codebase. He also explains why Databricks remains private, citing stock volatility's distraction and demoralizing effect, and predicts AI's economic impact will take 10-15 years to materialize.

Transcription

13564 Words, 72420 Characters

English
Speaker 1So it was kind of out of despair that we started Databricks because no one else is listening to it. At that time, there was zero interest in AI. Now, I think we have all the intelligence we need to completely transform the planet.
Speaker 2It's not possible to have a thousand agents doing stuff realistically. Am I correct?
Speaker 1They're just using AI in very rudimentary ways. They're just using it as a glorified Google search. We're actually getting into like management money ball, aren't we? So if the token cost is going up exponentially, is your revenue at your company going up exponentially? But I actually think GDP will move very slowly and it's going to take decades.
Speaker 2Are they a little drunk on their own success?
Speaker 1Just because 98% of the code is being written by AI, we don't have RSI. If it automates the last 2%, who cares?
Speaker 2When are you going public? If you were building a global financial system from first principles today, you wouldn't build it on 50-year-old legacy rails. You'd build Airwallex, one AI-native platform for global accounts, cards, and payments. It's designed to make the entire world feel like a local market. Others are bolting AI onto broken infrastructure. But Airwallex was built for the intelligent era from day one. Stop paying the legacy tax and start building the future at airwallex.com slash all-in. Airwallex, built for the future. All right, everybody. Welcome back to the all-in interview. We're so lucky because all of these amazing leaders in the world, CEOs, politicians, scientists, they watch the show. They're part of the community. And Ali Ghatzi is one of those individuals. He, of course, is the CEO of Databricks, a private company that has been doing incredible. I don't know why it's not public yet. I think a lot of people are asking that question. We'll talk about it today. In 2015, they were doing $1.5 million in revenue. It's August, or the rumor I heard, August 2026, Ali, $7 billion run rate growing 80% year over year. And obviously, you've got a lot of competition, the snowflakes of the world, etc. But this is a tremendous business. And we're going to talk a lot about how you're using AI to run this incredible business. We're going to talk about recursive self-improvement. But welcome to the program. And then maybe you could tell the audience a little bit about what you do at Databricks and how it became so successful.
Speaker 1Yeah, happy to do that. Thanks for having me on. Excited for this. Yeah, so we started the business in 2013. But if we rewind a little bit before that, we were all at UC Berkeley. There were about 20 of us. And we were actually part of a lab called Amplab, which was, I think, the largest AI research lab in the world at that time. And we were trying to figure out how can we combine data processing with AI. And we came up with really interesting things, but the industry was just not ready for it. So we were trying to get the industry to adopt this technology, and they weren't. So it was kind of out of despair that we started Databricks because we thought, you know, if we don't start a company around this, no one else is listening. Actually, it was called machine learning at the time, and AI was just robotics. So that's kind of like the beginning of the story.
Speaker 2And for people who don't use the product day in and day out, who aren't in the industry, what's the goal of the Databricks product suite? What do you do? I know you released some new products today, so happy to talk about those as well. But conceptually, you got there so early. Obviously, machine learning. Google was built on that, Facebook and their ad network. A lot of interesting services, Uber and their routing algorithms. Maybe you could talk a little bit about where Databricks started and what it's become and how it's generating billions of dollars in revenue.
Speaker 1Yeah. So the first use case, actually, it was the first use case ever was at UC Berkeley when we wanted to help Netflix recommend better movies so that people would watch more movies. So it was like a recommendation algorithm, AI use case that started the whole thing. And then we thought, you know, more organizations, probably. Probably every organization will need a data science team that can do this kind of AI stuff. And it was kind of bold at the time. There's a lot of people that said, no, no, no. AI is just for ad tech. The ad companies need it. But we thought, no, no, this is going to be a bigger market. And actually, we had a lot of debates internally. We weren't sure it was going to explode. So we got into it. And then we realized when we started the company, we realized why they're not doing it, why it's so hard. Because basically, all the organizations out there in the world are bogged down. They have legacy software that they've had for 30, 40, 50 years. And some of them, you know, companies have been around for 100 years. You know, things are moving slow. They have a lot of processes. There's. You know, a lot of compliance, a lot of regulatory, a lot of security. But all these things have gone worse with generative AI in the recent years. So basically, what was missing from our research at UC Berkeley was that we didn't think about the security, the governance. So what we did the first bunch of years is that we figured out how do we take this technology that lets you process all your data in an enterprise. So, you know, organizations have a lot of data. The most important data they have, actually, people don't talk about it. The most important data any company has is the data on. The core product that they have. Like for you here, it's this podcast, you know, the data on that. Someone else might be selling insurance. Someone else might have, you know, a drug. It's the data that comes out of that. That data doesn't sit in CRM. It doesn't sit in human resource management.
Speaker 2In fact, it probably wasn't even tracked. I remember in the early days of the Internet, they'd be like, hey, would you like this log file? And I'd say, OK, yeah, keep everything. And they'd say, OK, well, you got eight thousand dollar storage bill a month. And I'd be like, OK, delete it. Exactly. And they're like, well, we could roll it up and keep some of it. I'm like, well, who's ever using the open rate on our emails? Who cares about that? You know, we send the emails. That's it. But then that changed. And was storage and compute and cloud computing a major accelerant to this big data movement? And then eventually machine learning. Those were kind of the waypoints in this movement.
Speaker 1Yeah. Before we started in 2013, basically people were running their own data centers. And when they're running their own data centers, you have to buy machines and the machines came with storage and compute combined. So if you ran out of storage, you have to buy more machines. And the IT departments of every organization were busy with this. What the cloud enabled was to just completely decentralize this and separate compute and storage so that you could just imagine you have infinite storage. You just dump everything there. OK. And if it's gone cold, so the data is old, it knows how to archive it so that it comes almost free to store it. And then you could get the illusion of as much compute as you wanted. And then, of course, you could start running machine learning on it. And then eventually GPUs showed up. So it kind of transformed everything. We bet everything, the whole farm, on just being a cloud company. We bet everything on AI, the use cases of AI, and we bet on open and open source. And at the time, actually, all three of these bets were controversial. I had a standing bet with one of the most important analyst firms out there where they said it's a strategic blunder. You're going to go under because you're betting on the cloud. And it turned out actually the only reason we succeeded was because of these bets, because they were contrarian. Because other people did not agree with them. So we got a head start for many, many years. And then eventually, when we happened to be right, then we had, you know, we were so far along and we got too big that there's too much momentum to stop it.
Speaker 2It's so funny you mentioned this Netflix algorithm. Netflix started, obviously, in the Bay Area as DVDs that were mailed to you in an envelope. And then you had this queue and it was critically important for people who don't know the Netflix prize. This was in the 2000s. It was in the 2008, 2009 time period. They put a million dollar prize out to any computer science team that can make the algorithm, I think, 10% better at predicting what movie you would give four or five stars to. So they just took all their data, they released it, and then they got probably $50 million worth of work from computer scientists around the world. It was the first time in XPRIZE. And I think XPRIZE. And this were around the same time. And they were all inspired by those old prizes to like who could circumnavigate the globe, who could solo fly over Amelia Earhart, who could who could solo fly over the Atlantic or the Pacific. It was like a very interesting moment at time. And that hit AMP, right? And AMP, the group was algorithms, machine learning. I don't know what the P stands for.
Speaker 1It was for people, actually, because we thought that, you know, the AI. is not good enough yet. So it needs a little bit of human nudging. So it's actually just kind of, you know, human feedback was already there. We were like, we do need the human feedback because this last mile is just too hard for AI to figure out itself. So we had people there as well.
Speaker 2Putting the people in there makes sense because Berkeley is a little hippy dippy. Freeburg's alma mater, obviously. They were like, we need some people in the middle of this. You know, we had Ben Horowitz on, sat in the other week. And he was like, the best explanation I've ever heard for why we are going to be able to control the rollout of super intelligence and this jump between artificial intelligence or artificial general intelligence and super intelligence that gets recursive and runs off with it and P dooms us all and we're all going to die sentiment by some people came from you. So please explain your thesis here. And then I'd like you to go even one step further and talk specifically about maybe the culture in the Bay Area, perhaps even the East Bay and this group of people.
Speaker 1I mean, happy to get into it. You know, I think there's sort of two different things happening at the same time. On the one hand, I actually think we already have artificial general intelligence. I've been saying it since last year, and I based that on the kind of consensus we had in that amp lab where it was, I think, the largest AI lab at the time in 2009. 2010. And that definition of AI that most of us had, AGI, we already achieved that last year and i wasn't actually sure about this so i actually went and checked with some of the people that were in the lab like hey is it just me am i hallucinating and they said no no that definition for sure but we've moved the goalpost you know so uh so i actually think we have all the intelligence we need and actually if you stop the intel if you happen to stop all progress right now i think we have all the intelligence we need to completely transform the planet okay but that's all
Speaker 2intelligence work by humans right i mean if it solves this latest cohort of math problems and we're just starting to build harnesses and build out the infrastructure i guess your statement would be correct and i kind of am in a line with you we've hit artificial general intelligence it just hasn't been distributed equally
Speaker 1distributed yet yeah so the labs and a lot of researchers they're kind of obsessed with super intelligence okay so when i say super intelligence uh i mean it the way it was defined by botstrom in 2014 in his book called super intelligence and there he's talking about something completely different he's talking about something that can do a millennia worth of thought so like a thousand years of human thought it can do it like in a minute okay all of mankind in a minute it thinks in 11 dimensions it's so much faster just a single agent is just you know many orders of magnitude imagine thousand times faster than any of us okay and that's just a different thing like that's it can write a novel phd thesis instantaneously and so on so i think the labs are on the quest of that and in that okay by the way
Speaker 2and so just to summarize that there the ability to write an entire unique phd thesis instantly which do we even have the amount of compute in your mind to do something like that today if everybody got together and said hey today we're going to give all our compute or for the next hour just as a thought experiment to this incredible mega prompt do you think would actually happen is it a computer issue or is it
Speaker 1a model issue well actually the the but we have to do is we have to paralyze them and have many of them and we have to have them to you know collaborate with each other uh so it's actually not the way kind of that he he wrote it down he had also intelligence in the book of how it can be you know communities of intelligence but i'm just saying they the the labs and many of the other companies are focused on this that's why they're so obsessed with the math the clay problems the seven you know millennial problems and so on uh but i argue that the biggest challenge that's facing us is how do we actually just make this ai inside organizations that's what databricks is focused on we just want to help all these organizations and when i go talk to organizations out there enterprises i see that they're just using ai in very rudimentary ways they're just using it as a glorified google search they're using chatbots all day and instead of typing in google search and getting 10 blue links and then looking at the 10 blue links the ai just does that for them the second thing that they're doing is they're doing coding okay and with the coding actually they're not seeing enough speed ups they're seeing maybe like 20 30 percent i ask them this question every day i'm in meetings every day with customers and i ask them these questions that's kind of what reality looks like i always ask them how many of you have you know thousands of agents running around automating all work and doing core work for you and so on it's just not happened
Speaker 2yet now they might be running a half dozen agents a dozen agents like somebody who's at the forefront might actually be coordinating against that many but the human still needs to be on the loop just like amp like there's a person who still needs to be in there because it will return slop it will return weird it can go drift off course it's not possible to have a thousand agents doing stuff realistically am i correct yeah well
Speaker 1somewhat further along i mean i'm actually using this all throughout databricks and i'm happy to show you but you know the the the reality is that they're not and the question you have to ask the question why is that like if it can solve navy or stokes very difficult math problems how come enterprises and organizations out there are so far behind why i think yeah well there is well there's many reasons but the main reason is or maybe one way to look at it is that there's a lot of people that are doing this and they're not doing it well there's a lot of people that are doing this and they're not doing it they're not doing it if we could gather all the information that you have inside a large organization with 200,000 employees record all the meetings which by the way that itself one of the big bank ceos told me the other day no way we're never recording the meetings like not in a million years but assume you have liability liable i mean by the way which i already i was already kind of shocked because if you can't do that how can the ai ever help you like the ai is simply not in the meeting if it's not in the meeting it's useless it doesn't matter if you have terry tau or you have the greatest mathematician in the world if they're in the cold and they're not in the meeting it's useless they don't know what decisions or what you're doing in your meeting with hundreds of thousands of employees there's no way it can help you so if you capture all that context and you could feed it to the ai they would nail the problems they could help us right now already um so that's really the the context problem now you might say okay well let's figure that out and do that then we were doing that at databricks but that's not enough even if you do that you have a second problem which is you know you could unleash something like open claw and just have it access all your data everywhere well then you get into this problem that you're going to get hacked cyber is a real thing and all the big organizations that have successful businesses they're really terrified right now of getting attacked right this is like a big concern it certainly
Speaker 2creates a lot of surface area when you embrace this i mean if a ceo or in a small company you want to connect everything it's a certain amount of surface area but for a 200 000 person organization 10 000 my lord the surface area has now gone a thousand x yes exactly so
Speaker 1that's like the second problem you bump into which is like okay we can access all the data well wait we don't want to maybe access all of it to the agents because you know now we have an insider threat and we might get hacked so what do we do about that well let's say we solved that too so now we've solved the cyber problem which better we have actually uh ways to do that at databricks happy to show you uh then you still have another problem which is okay now the token cost is going up exponentially so if the token cost is going up exponentially is your revenue at your company going up exponentially and i don't know any company who started using ai and as of their use of ai the revenue started accelerating exponentially matching their cost curve so then eventually first of all ceos are like go get the ai get more ai get it in there you know we need to move faster yeah exactly but then at some point the cfo shows up and says hey our opex is going up significantly here and even the cogs what are we doing about it and we have an earnings call soon and then they freak out so then how do we fix that so you need to also do this efficiently which we also have ways to do that at databricks um now you've done all of these three then there's a strategic last question that still remains which is like okay fine we figured it out we access all the data we've done all of these three then there's a strategic last question that still remains we secured it and you know we figured out the cost situation how do we avoid getting locked in and how do we own our own ip because um you know it's otherwise you know there's no business to be made in the future because
Speaker 2data sovereignty is what you're kind of referring to intelligent sovereignty and if you use anthropic open ai or any of these platforms there is this risk that it will leak that it will get incorporated into their next version and as we've seen a company called a half dozen products that just now they're competing against microsoft one of their in i think microsoft is like one of their giant partners and they just came out with uh and basically an excel killer they're killing photoshop and they're just going after everybody
Speaker 1wholesale yeah so these four for free yeah yeah so by the way these companies are amazing they're doing amazing work and without them we wouldn't even have this intelligence probably uh so you know i'm thankful this is a company that's doing amazing work and without them we wouldn't even have this intelligence probably this is awesome i'm just pointing out why when you look around you all these you know successful organizations all the global 2000 all the fortune 500 nasdaq and so on how come it's just glorified chatbot and a little bit of coding uh this is the reason so you need to really tackle all four of those uh yeah and only then can you start getting uh something
Speaker 2out of it and one of those issues as well i believe is the autonomy in an organization to come up with a groundbreaking idea you know you may have captured a large portion of your market you may be getting incremental benefits so if i tell you like hey we can put another 500 horsepower into your for tourists it's like okay my ford explorer now has 500 extra horsepower it's going to rip the chassis off it's it's not this thing is not capable of going 500 miles an hour so then it's well who decides we're launching three more product lines well at entropic somebody is deciding why don't we just go after every single business in the world and they started doing dashboards uh that's a little close to your business is it not little dashboards over here
Speaker 1yeah so dashboards you know we don't we don't we don't have a product where we charge for dashboards dashboards are usually backed by data and you typically want that data to be governed so you you know you don't want just numbers and then you know those numbers are incorrect this is a big problem in organizations so that gets back to the second thing i said the governance security aspect okay so like yeah maybe we can get access to all the data and get all the contacts but we got to make it secure so yeah it's great so you can actually connect all of these agents from outside to databricks but then we are the governed data repository and then when you access that data and you know the governance we charge consumption rates on that so actually that's one of the reasons you see acceleration of our business because all these companies whenever they have agents and they're using it then i think we're just in the beginning as i said i still think it's mostly google search encoding but the agents when they start using that they just you know send
Speaker 2us more queries behavior change when people actually embrace agentic technology and they start working with agents let's say every hour every other hour of the day like you and i do
Speaker 1i think that the organization the org charts are built this way you know it's like a tree structure the ceo at the top and all decision making goes kind of up and down in these org charts and you have all these old processes uh and getting agreement across this org chart is very very hard you know it's like a tree structure it's like a tree structure it's like a tree structure inside of an enterprise or inside of any organization that's sizable, it's easy up until Donber's number, which is 150. You can know everyone. So small startups, it's a different story. But above 150, this becomes a coordination problem. But if you look at how actually any organization actually sells a product or a service to its customers, it's actually not this kind of hierarchical thing. It actually cuts across that org chart. Like here at Databricks, we have customers. They give us feedback. The feedback goes to product management organization. Product management hands it to engineering organization. Engineering builds products. That goes to marketing. That markets it. And then marketing gives it to sales. That sells it to customers again. So you kind of want to go across. So you want to cut out that latency so that you can be really, really fast. But that requires that you have an AI that actually understands your whole business, your IP. It needs to understand your org chart. It needs to know who's who, who's responsible.
Speaker 2What percentage of companies actually understand what you just described? Would you say 10% of organizations understand this?
Speaker 15%? At scale, I don't know how many. I don't know. At scale, I know that Databricks is doing it. I'm doing it. I haven't seen anyone else, to be honest, do it at this kind of, you know, I think probably Block and Square with Jack Dorsey, since he's pushing, he's calling it the company world model. He calls the same thing I kind of described. I think, you know, a different way of viewing it. He's calling it the company world model. So I think that they're probably pretty far ahead. In general, I think digital natives are probably further ahead. Digital natives being companies. That have only existed 10, 15 years. You know, if you've been around 20, 30, 40, 50 years, which means you're an amazing company, maybe you're a fortune 500, the best companies in United States. You have maybe a hundred thousand employees. You know, they're loyal. You know, they love your products and you know what, you're the best in the world on that. You're just, you have so much software, you know, you have so much regulation and you have so many processes around security and so on that it's very hard for you to adopt this. But that is the very problem Databricks wants to solve for them. So we want it, we want them to adopt the technologies so that the AI, so that they can do what I do at Databricks basically.
Speaker 2Yeah. Let's see it. You said you would be willing to show us like how you run Databricks with agents and with AI. And I've been looking forward to this all day because I literally am taking the entire, while you queue it up, my entire life on podcasting and on early stage investing. And I'm looking at each of the components of those things as certain. Services as agents and defining the goal, working backwards to what are the signs of success that you're heading towards the goal? Not that you've achieved the goal. Achieving the goal is kind of binary. Yes or no, you made the sell, you didn't. But what are the precursor signals? And then trying to have each of these bots hand off things back and forth. It's, you have to be a real systems thinker. It's like kind of playing five chess games at once while riding a horse. I mean, you're, it's really like intellectually, I find it super stimulating. I can tell you do too. So, Hey, you can share the screen.
Speaker 1So let's jump in. So, okay. So the key thing here is that, so we have a product called Genie, but the key thing that Genie has, cause you know, there's so many chatbots out there and there's so many assistants and there's so many cowork products out there and agents in the background. The key thing is that it has an ontology. The ontology is a graph that it constantly it's constructing for your company. That graph understands all of the people who reports to who. What are they working on? What are the emails that are coming in? What meetings have happened? And what's the transcripts of all those meetings and all the decisions that are being made there? What's on Slack, what's being communicated. So it has access to all of these things. And then also all of the docs that you're creating all the Excel sheets, all the PowerPoint presentations. So it has access to all of those. That's what we call Genie here. So you can see here. So I wake up, it, you know, I haven't, it has an inbox in the morning. When I come in, it basically has done a bunch of background work for me. It's running 24/7. As you can see, it has access to my, uh, you know, calendar. In fact, I could just ask it like, you know, uh,
Speaker 2Okay. I tell Jason on all in today, here it goes.
Speaker 1Yeah. So see what this is going to say. Uh, but, um, let's look at some of the stuff that I've done. So like actually the other day, one of the interesting things is I was looking at our HR department came back and they were saying, Hey, we've done a survey on all of Databricks, you know, so we had over 10,000 people, employees that. Wanted to survey. And, you know, I looked at the results and she was going through it and, you know, I said, Hey, make me a dashboard of the results. Uh, you know, nothing magical here yet. So it's told me like, look, here's how you're doing. You're doing great on some things.
Speaker 2Executive leadership, 88%. You're doing okay. Yeah. Work life blend 60% that you're working people pretty hard over there, but maybe everybody says that, right?
Speaker 1Yeah. No, but we want to know about this is, this has been, so we want, we wanted to improve these two. Okay. And this action one really bothers me because, um, you know, action, by the way, it means that when we, we ask them for feedback, they give us feedback, improve this and that, and they feel like you're, you're not listening to us. So two out of three feel
Speaker 2you are one out of three feels you're not just to you're doing okay. It's not terrible, by the way,
Speaker 1this could be like 30%. 60% is not great. So, but, and especially not great if you actually take into account that every year it's low and every year I promise the company we're going to fix this and every year it doesn't happen. So, you know, we're going through the results and, you know, we're discussing it with these staff. While they're discussing it, they're speculating. Um, you know, my CFO says, um, you know, Hey, I think it's because of work-life blend, work-life blend is bad, you know, and that's causing action to go down. So it's causing work-life blend is causing low action. And I was like, that doesn't make any sense. Uh, but I just entered here immediately as we were discussing and it immediately pulled up, okay, this is the correlation between these two and this is how it's looking. But then I was like, okay, wow, maybe he's right. But then it says, no, actually it's not causal. Okay. It's not causing it. And here's the exact proof points. We probably have to anonymize these folks, but here's a manager that has work-life blend 44. That's pretty bad. Action is 77%. Uh, so, you know, they're working hard and he's
Speaker 2recognizing and taking action on their concerns. So that's kind of, that's your guy right there. He's pushing people hard and he's listening to them. I like that.
Speaker 1So it's okay. So now I'm learning stuff about the organization. This is amazing. It would have taken me like a week to get the data team to do this analysis for me, but that's just me getting information. That's useful. So like we're cutting this, you know, latencies up and down the work chart a little bit, but then I'm like, okay, why is it low? And look through all of the comments. And it gives me some reasons. I didn't like the reason. So then I actually clarified the question and I said, Hey, can you slice it by department for me? And you know, and then actually figured it out. It said, look, product management has low action because they don't like decision making is too, too, too many, too many stakeholders, uh, engineering. They don't like the tools and systems. There's too much, you know, sales operations, it's career growth and so on and so forth. So that makes sense
Speaker 2because salespeople always think they should get a promotion because they're salespeople and they should get their commissions raised and product people think you should just listen to them instead of, you know, six people trying to design a piece of art. This all starts to track now with the data and the AI.
Speaker 1I love it. Yeah. So how did it do this? It has access to his ontology. So it's not going and searching live only. It actually found that there's already, he already had in its graph, these five people, you know, already five people had already contributed to this and there's 19 pieces of information here, uh, that you can see that it's actually leveraging that it already had. It didn't search for them live. These are in the ontology. So we can look at, for instance, this particular one, if you click on it, it has a definition, it's a metric and there's a graph and this graph, you can see it here. Uh, you know, so there's a favorability and it comes from, you know, this is the author. You can click on the author and because it understands the org charts, you understand who this person is. This person actually has 29 snippets. You know, you can click on the author and because it understands the org charts, you understand who this person is. You know, they have lots of different assets, you know, that they've, uh, you know, that it's learned from them. So this constantly is learning the intelligence or ontology of your organization. So that's a, that's a, you know, typical thing. Here's another one that, uh, you know, is maybe, uh, interesting. Um, I wanted to know, actually there's new results here in the last 59 minutes. I haven't actually looked at all of those, but I wanted to understand the org chart and what's going on with the org chart and, you know, span of control. And I asked it and already out of the box, it was actually, I was shocked by this. It immediately told me like, Hey, there is an org inside Databricks, which is very deep. You know, this person reports to this person reports to this person, the person, this person, but they have three people reporting to them, five people reporting to them, four people reporting them, five people reporting to them. So that's not a flat organization, you know? So then you can look at, okay, what can we do about that? Again, it understands since it understands all the people, uh, it can immediately out of the box, understand that, um, engineering, you know, cost cost, is it, you know, something that we look at at all, right. And, you know, so that's, that's another thing we wanted to see if we could understand.
Speaker 2And what's great about this too, is instead of you as the leader saying, Hey, listen, if we're going to have an opinion, let's go with mine. You've now created a forum where people can be data first test theories, and you don't have to be dogmatic about who came up with the theory. You just have to be intellectually curious and open to possibilities. Uh, and eventually if this is where we are today, can you imagine? Like in a year or two, it's going to just be giving you these instead of through a culture report, it's going to be like a temperature gauge in your company. It's just like, you're going to be looking at the nest in your company and it's like, oh, wow, it looks a little hot today. What's going on? It's like, oh yeah, we made people come in this weekend. Yeah. Maybe we should buy some pizzas or give people a comp day during Thanksgiving.
Speaker 1Yeah. Yeah. This is my favorite. So, so this one actually is, it came out of genie. So this is our 2000 engineers and this is how many pull requests or how much code you're going to get. So this is our 2000 engineers and this is how many pull requests or how much code they've actually shipped into production. Okay. And you look at the middle here, it's maybe per week roughly, but it has this L shape, you know, and you look at this L shape, it's like, oh, this is a lot of work. l-shape uh you know the top people are actually committing way more yes you know so what's going on yeah so what the hell is going on so we looked into this and it turns out they're running automated factories the rest are not so we push them and actually by the way interesting thing i asked that hey can you like add cost to this add the cost to this line and one of the interesting things we found is that it's not even the case that cost necessarily is higher for those people so look here at the top uh the top folks some of them are actually not spending as much as some of the other folks so which by the way goes back to there is a way to spend tokens in a smart way uh you can use open source you can route simpler questions to simpler models and we actually have a product for this uh and you can cut your costs down significantly um so this is the kind of stuff by the way it's it's also recording the meetings is really important so um you know one of the things i heard that there was a customer meeting and i wanted to understand how that product uh how that meeting went and you know uh and i just asked it you know hey was there a meeting with this customer recently how did it go and it literally started by saying the meeting went poorly oh and it listed and it's listing exactly what who said what what are the features they need uh so you know um these are just some of the things that you can do you know with the ai and you know in the background it'll when i wake up it has all these deep dives for me so instead of waking up calling people doing meetings it's doing the work for me it's pinging them it's you know sending uh you know whatever material is needed for the company so it just compresses the time it's doing the work for me it's doing the time it takes to do things well you know
Speaker 2it's a lot of being in management and being a founder is detective work i call it like being colombo if you remember that tv show i think it might show our age but you're kind of like oh you know it's one more thing there can you tell me about the meeting the client meeting how'd that go and who talked most on the phone and what percentage of time did they talk and what did the customer actually think and then you as the founder are getting spun you're getting spun and it's not that an employee necessarily wants to spin you a team member is not looking to spin you necessarily that can happen but that's not the majority most people are good people but they have they're humans right they're the people in the air and here you take the people out of that discussion you say here's first principles transparently what happened now we have ground truth and we can start the discussion without you being the ceo having to be mr detective and you know it is so i mean i don't know about how it was for you two years ago to spend two hours being a detective you get three meetings into this and it's like if people had just told you what happened or if you had that report on culture the problem would have been solved you wouldn't have had three weeks of hand-wringing yeah yeah
Speaker 1i mean or the customer feedback right that i can immediately find out all the customers i you know i have some some of them where i'm like just having it look at features that are requested from all of our customers automatically from all the transcripts of all the customer meetings that we have and it's just gathering those and it's just gathering those and it's just gathering those and it's just gathering those and they're immediately going to engineering and then we have factories and engineering that can build features automatically uh so it's a game changer you know it's just that game of whispers that we played when we were kids right it's like when you have people have to meet they have to tell so one of our main goals is uh reduce number of human blockers so we measure that throughout the organization and we just see
Speaker 2explain a blocker in this context please well that's something i can learn from yeah
Speaker 1i mean the ai is very fast right it's just iterating in the background and it's doing its work and you know it's preparing material and so on but at some point it's like okay well now we need to go to legal and make sure that we are allowed to do this thing or now we need to go to um you know finance to get approval for this budget or now we need to make sure that marketing is okay with uh with this or actually just within the same organization we need to talk to a sibling organization get a meeting with them and see what they think and we have to get an agreement and if we can't reach an agreement we'll schedule another meeting where we'll look at this again and then if that meeting doesn't maybe we escalate it to the managers and then so on so these are the blockers that slow down so it's a blocker
Speaker 2finder which is great if you're the ceo because they're here's the next thing one you have to play detective the other one is the blame game who's why isn't this done is what we always ask when you're in a leader whatever your group another group hey we were supposed to do this by today we're supposed to hit this target who's responsible and then you get three spider-mans pointing at each other if you know the meme yeah here it's like well this actual piece of data has not been resolved and your team has run nothing i had a similar thing i came up with a bot and i was going to share it with the whole company and thank god i didn't but i came up with a bot just called major decisions and i just said go into slack join every room as me just join every channel as me which some people are like wait a second how did you even know we had a room join every room read every notion page uh even if i was invited to it get every email uh and find for me for the month and every uh couple of days give me a report on the most important decision who participated in them and what was the resolution what's not resolved that i said give me the people who participate the most in these kind of discussions and then i realized oh wow these are the people who are actually making decisions in the organization these are the people who are quickest and my next report is going to be is who's quickest to contribute to those discussions now in a venture firm that's going to be a discussion like should we invest or not should we take our pro rata or not should we sell uh our shares if we got a secondary offer in on the podcast is like should we buy or sell our shares if we got a secondary offer in on the book this guest or not should we uh you know use this clip or that one and then it picked up employment discussions and it was like should we fire this person or not or you know do we do pips and like we don't do pips and like i just think pips are ridiculous like the performance isn't here that's like the person didn't show up let's
Speaker 1torture them for two quarters
Speaker 2yeah like you you mispractice and you showed up late for the game that's table stakes you can't be on the team if you can't do those things so now we're going to torture you try to make you come into work on time or do a workout you don't want to do so maybe you could speak to what this is going to do for you as a founder and a ceo in terms of figuring out who are your all-stars on the team who are the ones who are actually contributing because now we're getting to essentially business money ball like that movie money ball it's fantastic brad pitt it's such a good movie it's such a good book we're actually getting into like management money ball aren't we yeah
Speaker 1absolutely but one of my favorite spots is just look at different orgs because we have certain culture principles truth-seeking first principle and so on and i say hey which of these leaders follow these principles look in their whole orgs look at all the meetings and it can derive from those and by the way you could even ask hey this leader what is the actual culture they're implementing it'll come up with different you know they'll tell you now actually the culture there is this not the company culture uh which is deviating
Speaker 2from the core principles that's that drift is cancerous for you as a to change and could be an early warning sign we need to be
Speaker 1consistent whatever it is right maybe they're right and the rest of the organization and including me are wrong but then we need to at least it tells me that there's this divergence and we can sort of you know so those kind of things were like nearly impossible to do like like many of this other stuff you could have done but a question like that like get the culture of this subgroup in a 12 you know 12 000 person organization you know figure out how the japan organization at databricks i don't even speak japanese you know hundreds of people over there working like what's the culture like how to make decisions how is it different from the emia or you know the latin latin you know region that we have so those are kind of game changers uh for us it's really i share all this information with the employees they have access to all of this themselves and then i tell them this is what we want like that l curve we wanted to shift it to the right so everybody let's change that let's let's move towards this uh you know the north star that we have together uh and that helps us just be more competitive and then once we get more competitive we can build more products and we take market share and we can accelerate our growth further you know i'd like to not be 70 80 growth year over year i'd like to be 100 you know way over 100 so then we can put you know that's that's how i see it so this is a in this era right now this is a competitive advantage whoever can do this best you know at some point maybe it's all everyone is using it's fully diffused and everybody's doing this but right now this is big alpha if you can do this really really well better than any other
Speaker 2organization speaking of big alpha a lot of people have been very patiently been on your cap table and we have a big debate between founders and capital right now stay private longer what's the point of going public if we have secondary markets that are quite efficient stripe has been private closer to 20 years i think than 10 and they seem to have given the indication they're going to stay private longer spacex took a little while uber took a little while i felt that one but luckily masio she's on with splashy cashy when are you going public and how do you make that decision i know you got a lot of pressure on you to do it but you also have a lot of pressure relief valves in the the vibrant secondary market yeah
Speaker 1look my goal is to make databricks as successful as possible in the next decade or two that's how i'm thinking about it what's the right strategy to get there i mentioned this alpha of what we're doing here the products we're building but we're getting into new markets we got into security market got into the marketing market uh i'm focused on those then is the path to get there is the best way and the fastest way for us to be public or to be private
Speaker 2and i actually through that what's what's on each side of the ledger for you yeah i actually think
Speaker 1that you know you get access to capital you get much much better marketing uh you know it's like everybody knows about you like every quarter you have this big marketing event you can tell your story again and again and again those are huge advantages we have to organize our own tenders every year every year we do finance has to go through this big process you know i think we raised 20 million dollars we have to organize our own tenders every year every year we do finance has to $20, $30 billion, and then we have to arrange that. It's a heavy lift. There is a marketplace that just does that.
Speaker 2It's also probably a pretty emotional and distracting, I would think, right? Like people have to decide. Then you might lose people that, hey, they take $10 million off the table. Now they're like, eh, maybe I don't want to come to practice anymore.
Speaker 1I think the liquidity is a good thing because people know that they're going to get it every year and it kind of comes and it's predictable. And that's the one time they have. And actually, in the past, it's turned out since we started doing these a few years ago, it turns out that the money that we set aside for tenders, they never use up all of it. So most employees are bullish. Yeah. So most employees are like, no, this is good for you.
Speaker 2You get signal from that, right? Who's selling, who's not. It's a little bit of signal. It could be personal reasons. But acquiring companies is a great reason to have equity.
Speaker 1Yeah. We can, I mean, we haven't had a problem, you know, doing billion dollar acquisitions because they know because of the liquidity that we will provide. And then I think most people do the calculation is like, okay, if it's, you know, the business is $7 billion plus and it's growing 70, 80%. Um, they do the math. It's like, okay, I could, what most, most people say I want liquidity, but then when you tell them like, okay, well, how much do you want to sell? We're growing 70, 80 plus, you know, can you do compounded growth of, you know, 1.7 to the power, you know, to the power of three or four. And then they start scratching their heads like, okay, maybe I'll sell like 5%, you know? Yeah. Sitting on your
Speaker 2hands is a very important skill to learn as an investor and an employee with this, this kind of stock. I mean, listen, you do have to protect downside when you, when you get, uh, in that space, who do you see as your competition right now? Everybody says something like snowflake, but I'm watching a market where, as I said earlier, Anthropic is launching feature after feature. Cloudflare is looking at features next to them. Replit is looking at features next to them. Everybody seems to be looking in their peripheral vision. And because you can build software fast and because you have a lot of developers, if those developers keep going up that ramp that you showed, you only had 10% of people at that peak, which means as that 90% of the developer resources are going to go through the roof. So you have a decision to make. Do we just have less developers at the low end of the tail and be more profitable? Or do we invest all that developer time as they move up the ramp? So, so how do you think about both of those issues? What do you do with massive resources that this efficiency is creating? And then who is the competitor? And do you need to just keep making peripheral product moves?
Speaker 1Yeah, we will. Make peripheral product moves. But maybe I should just start by saying, okay, so those were the upsides of going public. The downsides are, you know, is the reason why, you know, to answer your specific question, like, you know, why do we compete? Who do we compete with? Why do we build these products? I don't want the market to force me that, okay, now it's down. And now it's, you know, sentiment is terrible this week. And now your stock is down. And when the stock is down, the employees literally are getting a pay cut. And I say, well, you know, you promised me I was going to make a million dollars. Actually, you said this is going to be 2 million in a couple of years, but I'm looking at it right now. And it's actually worth 200K. So you lied to me. I'm leaving. Okay. And then, you know, three months later, it's like, oh, no, no, no, actually the revenue is accelerating. We were totally wrong. It's up to the right now. It's like, oh, okay. Wow. It's great. So there's like back and forth. A little bit of a rollercoaster. That's what I want to avoid so that we can focus on that long-term, you know, building the company towards the vision that we have in the long run without being distracted. We are going to be public. But right now, I think during this transition period of AI, there's just too much up and down. And I think it's distracting to companies. It's distracting to CEOs. It's demoralizing to CEOs. You know, they call me, you know, they're like depressed because, you know, stock is down. They don't even understand why. And then the stock is up. So that's what we're trying to avoid.
Speaker 2I think it's an issue people are starting to have with their sleep scores. You know, you get a whoop and I love it, but I got my wife a whoop. And then we're like comparing sleep scores. And now it's kind of like, looking at your Uber stock every day, you get a little bit of volume in your relationship. It could be challenging. Otherwise, it's interesting. Hey, we didn't circle around to what Ben Horowitz said. He said that your formulation of super intelligence is some combination of the model, the power, and the time it takes to train and be accurate. Can you walk us through it? Do you have it off the top of your head there? Ready to go?
Speaker 1I have to do it. I mean, it's pretty simple. So I think the main problem is the context problem. We're solving that with Genie and the ontology. The labs are focused on super intelligence. That's what Botström talked about. Botström in his book also said, how do we get there? He actually, in 2014, I believe, defined RSI, recursive self-improvement, which means that the AI starts improving itself. And then it gets better and better and better and better and better. And then we get this takeoff, maybe a fast takeoff. And at that point, we reach the singularity and we're no longer in control. And at that point, let's hope that the AIs are well aligned, because if they're not, we're all going to be made into pets. And this is where the 10% risk, yeah, paperclip maximizers and the 10% risk to humanity and all of that comes from there. But I think a lot of this is actually coming from most of the folks that are building this AI. They've seen that the AI has become really good at coding. And almost in every company, as I said earlier, the coding is being done by coding agents. So then they're seeing, and I think this is actually causing a big existential crisis in Silicon Valley. And it's actually for most of us who grew up coding our whole lives, you know, it's, it's, it's a big deal. Like, you know, it's like, like I spent, you know, 10, 12 hours, seven days a week, you know, from third grade until, you know, my thirties, just coding all the time. Right. And I was really good at it. And I knew all these languages and so on. And then suddenly it's like the AI is doing that.
Speaker 2Well, the AI is how much better than you. That's the other part. It's like, yeah.
Speaker 1So now it's writing itself. And the next version, it wrote itself. Wait a minute. Isn't that the recursive self-improvement? But wait a minute, then this is going to speed up and then we're going to get into this loop. The books, the book called it super intelligence and the pay-per-click maximizer. So it might go off and all these bad things happen. Actually interesting. Botstrom has kind of changed his view. Now, if you have to follow him now, he's actually not anymore. He's saying that, you know, we should do this. There are risks. Of course there are risks with cyber, but you know, I actually think, you know, you need to actually think this through further just because 98% of the code is being written by AI. We don't have RSI. You know, it doesn't matter if it's 98, 98% or a hundred percent. Why does it matter? Actually doesn't matter. So we really need four conditions to be true. The four conditions is that we need the next model that's trained to, you know, be trained in a much shorter amount of time. Okay. And still the big pre-training runs that the labs do, they don't do actually that many of them per year. Okay. And it's still following the, what's called the scaling laws, which means you need exponentially more hardware to get that improvement. So exponential more, you know, 10 X more compute to get a little linear improvement. Okay. So they're scrambling to get all the GPUs and put together the biggest data center that they can possibly put together with the networking and do all the engineering. And then they having humans do all of this because they really don't want to screw up the next run. If the next run gets screwed up, then, you know, the whole thing falls apart. Uh, but you know, we would like that next training run to be faster. That's condition number one. Okay. And actually it can't just be a little bit faster. It's like this, you know, it can't be like, you know, you're having the distance every time you'll never get there. So in some sense it has to be super linear. Secondly, uh, to use less resources, right? Cause you know, you could say, okay, it's getting faster and faster, but that's because we're 10 X in the hardware all the time. Well, you can't do that trick because we run out of hardware and we run out of atoms. So that's like the second requirement. Third requirement is that the accuracy is also improving at least linearly, or, you know, we're getting better accuracy across the board in a generalized manner. It can't just be, it's getting just better at, you know, this one dimension that we did reinforcement learning on, or just math. It needs to be across the board on all of its tasks. Okay. Even if you do all of those three, it's not enough. Let's say you do that five, six, seven times. That's awesome. You get really awesome models. That's not that kind of super intelligence. The fourth condition is that you can actually put this in a loop and all the previous three conditions need to be continuously happening so that we end up in a world where the next model takes, you know, three months, then one month, then, you know, two weeks, one week, and it's getting smarter and smarter and smarter. And it's, you know, costing us a hundred bucks or maybe it costs us a hundred million, then 50 million, 25 million, 10 million, five, two, one. And then it goes down and down. That's what we, that's the kind of scenario we have. The flywheel, that flywheel doesn't exist right now. The models are taking longer to train, right? And there's more data cleaning and there's more manual whack-a-mole happening where, you know, you have vendors like Mercore that has to actually go get the data from very skilled labor. And then you have to do reinforcement learning in these gyms and you have to kind of whack-a-mole in each domain. So it's very domain specific. And then of course, the pre-training requires more resources. And, you know, so, um, it's, it's, that doesn't look like we're getting into this loop, by the way, even if you satisfy all the four conditions I'm guaranteed mean that you'll have, you know, a conscious being that's just, you know, taking over everything and just, you
Speaker 2know, it's the precondition for that happening. And there's no guarantee that it actually, it's a
Speaker 1precondition, but even if you had that, which we don't have, you would have to, to threaten me right now, you would have to have a killer robot come into this room right now and wrestle me down. Okay. Or something. Right. And those don't exist yet. So, uh, you know, right
Speaker 2now you can have a bunch of hackers take a really capable model and you can do a lot of things. And I think that's, I think it's, I think it's, I think it's, I think it's really important. I think it's, I think it's, I think it's really important. And I think it's really important. And I think it's, I think it's really important. And I think it's, I think it's remotely script kitty into them and turn it into defrost mode and make a bunch of butter go bad and milk, which is a tragedy for the French. I understand, but it is not super intelligence. I can assure you. It means, I mean, did you see that hack and how people were losing their minds over it? I'm like, guys, you know how, when you buy a new router now it has the serial number and then a random number. And I think that's, I think it's really important. Yeah. Yeah. If they had done that on the refrigerators, They wouldn't have been able to hack one of them unless they got into the place, took screenshots of 20 different refrigerators, physically compromised some supermarket in the 8th and Drosmint, which would have lost, I don't know, 10 pounds of butter. It wouldn't have been a big deal.
Speaker 1So look, cyber, I do think, is a real thing. Yes. And I think it's a real risk. We actually have a whole product in the cyberspace called Lakewatch built on all this data that has automated agents that actually detect these kind of intrusions that are happening and they monitor your agents that you have. This is a real problem. And I do think bad things will happen here, but then we will adjust. But bad things happen when internet came out as well. Shouldn't we be able
Speaker 2to solve for this because you tell me if I'm right or wrong here. One theory I heard, which resonated with me for a certain type of hacking, not for the script kitty hacking, but real hacking, is you have a lot of compute. You need a lot of compute to execute certain hacks, like the hugging phase one. You know, a thousand agents. The bad guys don't have that much compute. The good guys have a massive amount of more compute. And so if you just did your basic job as a white hat and, you know, defended properly and looked for these exploits, let's say you have a thousand times the resources, you should be able to get to these problems long before the hackers. Does that make good sense to you? I understand the argument. Yeah. It's a good argument, right?
Speaker 1Yeah. But the issue is that you have to weigh that against the following asymmetry. Which is if I'm an attacker, you know, I can launch thousands of thousands of thousands of agents and I, all of them can fail except for one. And if one of them gets through, I'm completely successful. The asymmetry is that if I'm a defender, you know, I was able to like, you know, you know, protect, you know, my, my boundaries all the time, except one got through. This is a suicide bomber, right?
Speaker 2Not to be too graphic, but you only need to get one 9-11 to make your point. Yeah. You can try a thousand times and not get through. Yeah. So I need to be a hundred percent safe.
Speaker 1I need to be a hundred percent safe all the time. They need to just get one thing through. Get lucky ones. It's like very asymmetric, that risk. But however, the way I would look at it, the reason, you know, I think that we're actually going to overcome this and it's going to be totally fine is that these are actually latent bugs that exist in the source code. By the way, we're not creating more of them now because almost all of us are now scanning it. We all have products. I mean, we actually sell a product where we can scan the vulnerabilities of new code. So new code that's being produced is not going to have these vulnerabilities. By the way, that might sound silly, but think about it this way. In the next 12 months, we're going to. We're going to write more code than ever in the history of mankind, right? Actually, in less than the next 12 months. So all that code is going to be secure. So then, and by the way.
Speaker 2Because right now the AI is sophisticated enough to not make human errors and to look for it. And it's more dogged.
Speaker 1Yeah. You just scan those things. So then what remains is all the legacy code, which by the way, we can rewrite all of it, which is what we have to go through. And people are doing that, right? Yeah. We're doing that.
Speaker 280, 90. Yeah.
Speaker 1Yeah. There's other ways to actually also protect, you know, do shadow patching before even actually without touching the existing legacy code. But so, you know, what's happening is that you can think of it as like, you know, it's like a fruit garden and there are like these bugs that, you know, attackers are leveraging. They're picking the low hanging fruits. And before humans could pick a certain low hanging fruit. Now agents can reach higher and pick higher, you know, hanging fruit. But we're going to clean those up and then they can maybe reach a little bit higher. But there's only a finite number of vulnerabilities in the code. Once we've secured that code base, there is no more. So it's not like there's an infinite thing and this is going to go on forever. No, it then stops. Then we're at the steady state. And then, you know, vulnerabilities might actually be fewer than what we had in the past.
Speaker 2I think that eventually then, you know, back to AMP where you went at Berkeley, it's the people in the loop that are the vulnerability. And it's just going to be somebody getting compromised, as we saw the last week or so. The humans do have vulnerabilities. I'll leave it at that. Hey, I want to end on this one thing. You spend a lot of time in Berkeley. Hippie dippie, EA community, accelerist, decelerist. What's the nature? Of this group of the EA faction and how they look at the world and how confounding they are to the rest of American society? Who are these people? What do they believe? Or do they actually have really important insights? And we need to really focus on what they're saying and take them at their word. It's a big controversy. Dario, you know, during the roadshow is saying cancer and Alzheimer's, I believe, will be gone in under 10. Now, that's very beautiful to say. Not because everybody's dead, right? Yeah, except if your mom or dad are dying from cancer and you think, oh, I got a window here that I could save them or my grandmother's got Alzheimer's. Now you're talking about giving people false hope and you maybe shouldn't make claims like that. But I don't doubt Dario believes it. I actually think he does. But unpack these people because you've been around them for two decades, three decades.
Speaker 1Well, you know, I think that most people on both sides, actually on all sides. They're the heroes in their own book, right? We all wake up thinking I'm the good guy and I'm doing the right thing. So I think it comes from a good place. I think they are seriously nervous about this kind of super intelligence taking over. If you read Bostrom's books, actually, if you read those books and you sort of internalize what he wrote before any of this, then you actually get kind of freaked out. Right. And I was certainly freaked out when I read it. So I think that they're starting with those assumptions and then they're looking at, you know, OK, well, then. If it's getting smarter this fast and look at the cyber attacks that are happening, dot, dot, dot. We're going to get this kind of super intelligence that's described in the book. But there's a few dot, dot, dots there that are not clear. But then they would say, well, look, you know, shouldn't we be cautious? Right. So I think we need to take, you know, we should look at safety. Cyber, I think, is a big deal that we should look at. But the reality is that I think we know how to handle this. We will be able to manage through this. I think we'll be the biggest risk. Right. Right now is cyber and I think it's manageable. This RSI, I don't see any evidence of it. If there's data that, you know, others have that I don't have. Yeah, I don't know what I don't know. But based on what I'm talking to, you know, here in San Francisco, where all the labs are and all the people I meet, I don't see those four criterias that I mentioned even being fulfilled. You know, so I don't see any risk to be like worried about that, you know, that happening. And then the second thing I said, even if we would get there, I don't see, you know, even the means in the society like it's just. Digital, like we can just shut down the computers right now. You know, I do think cyber is real. Like if a hospital went down and set up a refrigerator, right, that is concerning.
Speaker 2Kids using it and developing parasocial relationships with character AI or it giving you bad information if you're suicidal or something like that. There are concerns to have about how the technology gets deployed. And we just went through this with you mentioned Dunbar's number before we went through this with, you know, social and the profound impact that had on society. We're going through it now with truth and news and manipulation of news via algorithms. So there are things to be concerned about being turned into paperclips.
Speaker 1Maybe not. But I think that, you know, the health care stuff is absolutely real. Like, you know, at Databricks, we actually we were involved with Regeneron where they found the actual genome responsible chronic liver disease using Databricks. Their researchers there and actually came up with drugs for that. Best case scenario in under a decade.
Speaker 2If you were to take the claim we had Dario make. Any chance that like we cure cancer in the next 10 years? That's a pretty bold statement.
Speaker 1It's very hard for me to say. I don't know exactly what's needed. I mean, I think progress will be faster, but for a lot of these things, it also has to go through human trials, human trials like, you know, we can't just test stuff on humans willy nilly. Right. And just because you tested it on animals, it's not enough either. Right. So I don't know, maybe in 10 years, maybe not in 10 years, maybe takes 30 years and takes 50 years. I don't know. Maybe it takes six months. Maybe it's already solved more or less, but it is helping already. So use case I gave you with chronic liver disease. That's super important. By the way, that was the previous generation of AI that we use the Databricks. We train the new one now with Merck and it's called Teddy Transformer enabled drug discovery, transforming enabled this drug discovery is a model transformer model. But instead of, you know, the next token being English, it's the gene regulatory network, DRN. So it actually can say, OK, this genome is actually causing this disease. This genome is just reacting to it. So don't target it for a drug. So this will accelerate. Drug discovery, it's a game changer, you know, drug discovery aside, like, you know, I give you another example, like crisis text line. It uses Databricks and uses large language models and these agents and they actually detect if kids want to do self-harm and it helps prevent that. Right. You know, Virtue Foundation uses Databricks, you know, Zipline. This was a use case that was reported before used Databricks to deliver, you know, food. They had these drones, the drones that would deliver. You know, everything about those drones and so on, like their their path, the, you know, really big data. Yeah, all of that was running. So we see the company. Yeah, we see these use cases all the time with AI and it is going to make the world a better place. There are risks. We will deal with those as well. People are nervous because, you know, things are changing. And I think they are companies. There's a lot of researchers who maybe, you know, they understand the research very well, but they haven't looked at, you know, how does the economy look? What's going to happen? You know, how are these things going to evolve?
Speaker 2And they've watched a lot of sci fi. Let's be honest. Like they. Are being informed by, you know, a certain narrative and the idea that there's a 10 percent chance that seven billion people die because of AI seems so farcical and out there that when you pick a number, it makes you seem credible to an average Joe who's like, well, these are smart people who make a lot of money. And if they think it, maybe they're right if they're
Speaker 1saying it themselves. Right. That's how people think. But for me, that risk is, you know, close to zero. I think the cyber risk is real and really bad things can happen with that. That's where we're focused on it. you Um, you know, and then, you know, I think actually for, to solve this, this context gap that I mentioned in enterprises, the thing we're focused on to actually get value everywhere, I actually think it's going to take 10 years. You know, I think this is something that's going to take 10, 15 years. The labs think this will happen like in the next six months, 12 months in, you know, two years, but I actually think GDP will move very slowly and it's going to take decade for us to actually get through it. Why? I mean, as I said, I mean, just go back to the anecdote at the beginning, the, you know, the CEO of the bank that said, we can't even record the meetings. If you can't record the meetings, the AI doesn't even know what's happening in the meeting. How could it possibly do the kind of things that we discussed when I showed you genie and the ontology?
Speaker 2It's going to be a paced rollout to be sure. And there are some of us who are running ahead around the corners, uh, doing really cool stuff, which is exactly what we both experienced with the internet. When the internet first came out, the people who were using it to its fullest extent were like, okay, this is going to change everything. And then 99% of people were on dial up and had never seen. The high speed connection. So they, their brains couldn't conceive of YouTube or, you know, unlimited free photography story. It doesn't make any sense. There was a, there was a,
Speaker 1there was a, there was a kid called Robert Morris, Morris, who took down 10% of the internet with the worm. It's called the Morris worm. Yeah. You know, he almost got a jail sentence for it. Uh, and then he went on and he became a famous MIT professor in computer science and he's like a very famous guy. Uh, but you know, that took down a whole 10% of the internet, uh, back in the day, which
Speaker 2is completely possible. Like at some point you start today. Somebody feels like they may have cracked signal to a certain extent. Uh, okay, sure. Let's see what happens. Another great all in interview, Ollie. Thank you so much for your time and we'll see you next time on the all in interview. Oh, thank you so much. Great job, Ollie. You are awesome. I'm going all in.

Podcast Summary

Key Points:

  1. Databricks was founded in 2013 by UC Berkeley AMPLab researchers out of frustration that industry was ignoring their work combining data processing with machine learning.
  2. Ali Ghodsi argues we already have artificial general intelligence and enough intelligence to transform the planet, but enterprises are still using AI in rudimentary ways like glorified search and chatbots.
  3. Enterprises face four barriers to AI adoption
  4. Databricks' Genie product builds an ontology of a company's people, meetings, documents, and decisions, enabling agents to run analysis, surface blockers, and detect cultural drift.
  5. Ghodsi says recursive self-improvement does not exist because four conditions are unmet: faster training runs, lower resource use, generalized accuracy gains, and a continuous improvement loop.
  6. He views the risk of superintelligence as close to zero, while cyber risk is real and significant, though manageable as AI scanning secures new code and legacy vulnerabilities are patched.
  7. Databricks is growing roughly 70-80% year over year at a $7 billion run rate, and Ghodsi intends to take it public eventually but prefers staying private during AI's volatile transition period.
  8. He believes AI-driven GDP gains will take 10-15 years, not months, because organizations still cannot even record meetings to give AI the context it needs.

Summary:

Ali Ghodsi, CEO of Databricks, recounts how the company began in 2013 out of despair that industry was ignoring his UC Berkeley AMPLab team's research on combining data processing with machine learning. Early bets on cloud, AI, and open source were contrarian but proved decisive, and the company now runs at roughly a $7 billion run rate growing 70-80% year over year. Ghodsi argues we already have artificial general intelligence, yet most enterprises use AI only as glorified search or basic coding assistance.

He outlines four barriers to real adoption: capturing organizational context, security and governance, rising token costs, and lock-in risk around IP and data sovereignty. Databricks' Genie product addresses the context problem by building an ontology of a company's people, meetings, documents, and decisions, which he demonstrates through HR survey analysis, org chart insights, engineering productivity metrics, and customer meeting reviews. On superintelligence, Ghodsi is skeptical of recursive self-improvement, citing four unmet conditions: faster training runs, lower resource use, generalized accuracy improvement, and a continuous improvement loop.

He puts existential risk near zero but calls cyber risk real and significant, arguing AI code scanning will eventually secure the codebase. He also explains why Databricks remains private, citing stock volatility's distraction and demoralizing effect, and predicts AI's economic impact will take 10-15 years to materialize.

FAQs

They came from UC Berkeley's AMPLab and felt the industry wasn't adopting their data-plus-AI research. Out of despair that no one else was listening, they started Databricks in 2013 to build a company around the technology.

They bet everything on being a cloud company, on AI use cases, and on open source. All three were contrarian at the time, and being right early gave them a multi-year head start.

According to Ghodsi, organizations face four main obstacles: capturing context (like recording meetings), securing data against cyber threats, controlling rising token costs, and avoiding lock-in so they own their IP. Most companies haven't solved all four yet.

Genie is a Databricks product built on an ontology, a graph it continuously constructs of a company's people, meetings, emails, Slack messages, and documents. Because it understands the organization's structure and context, it can answer questions and do background work that generic chatbots cannot.

No. He argues that even if 98% of code is written by AI, four conditions aren't met: models aren't training much faster, aren't using fewer resources, aren't improving accuracy broadly, and the flywheel loop doesn't exist. Training runs are actually taking longer and requiring more compute.

Going public brings access to capital and much better marketing, but also stock volatility that can demoralize employees and distract leadership. He says Databricks will go public eventually, but wants to avoid the AI transition period's ups and downs for now.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.