Can Australia move up the AI value chain? These founders think so. | Springboards & Celeris

No country wants to be dependent on another country for their electricity or energy needs, right? Because as soon as something happens, you can be cut off and your entire economy can grind to a halt. Same thing will be with artificial intelligence. You want to be able to own as much of that as you can. What does it actually take to train a model? Talent, compute, data, like what does it actually take? Honestly, it's surprisingly simple. To do it well, it's fucking hard. That's train a model is not that hard. I think open source software and the open source movement has probably been one of the greatest things that we've done collectively in humanity. The open source tools now, be it a 4 billion parameter model to trillion parameter models, are so damn good. If there's one thing that I've learnt following the past year of the AI revolution player, it's that simple definitions rarely hold up to interrogation or at least not for very long. Just when I think I've gotten my head around how all of the pieces of puzzle fit together and the incentives play out, something fundamental shifts and the picture gets muddy again. But a few broad frameworks still seem to hold amidst all of this change. And one of them is how we should think about the different layers of AI. So bear with me while we walk it through. Earlier this year, Nvidia CEO Jensen Huang described AI as a five layer cake. At the bottom, you have energy, the stuff that makes all of this possible. I reckon you can probably add water here too. These are the commodities that are needed to power AI and as we all know, one of the emerging constraints on its acceleration. Next you have chips, the tiny but powerful and highly sought after worker bees of AI, and the dominant player in this space in video is sitting pretty with its status as the world's most valuable company. Then you have infrastructure, the physical stuff that makes a factory out of AI, think land, power delivery, cooling, construction, networking and how all of these systems fit together. Next up, you have models, the layer where we've seen the most dollars flow in recent years as frontier model companies dominate our news feeds. This is the layer upon which so much opportunity is being created, the intelligence layer. And finally at the top, you have applications, which in Jensen's language is, quote, where economic value is created. It's essentially the part of the AI landscape, the majority of us is likely to actually experience firsthand. Sometimes our experience will be explicit and intentional, for example, if we choose to buy an AI product directly in our personal or professional capacities. But actually looking into the future, probably the majority of our use of AI applications will be implicit or not even known to us. And it will come in the form of how AI increasingly powers the products we already use in our everyday lives. Think of banks using AI to detect fraud, retailers using it to manage shipments, smartphones using it to automate photo enhancements. For now, there are real markets and very real dollars being created and made around each of these five layers. But smart companies and governments are already thinking of hard about where the enduring value will lie into the future. What layer or layers of this cake will commoditise and what's going to retain pricing power and thus economic opportunity and prosperity. And, relatedly, hard as all of this intersect with the increasing anxiety over national sovereignty and power. Trailing in particular is an attractive place to consider having some dominance in the infrastructure layer, that middle one, the data centers. Now, it's not without its detractors, of course, but there is real policy effort and capital flooding into that space. But there's a very real danger in stocking there. We've made a name for ourselves as being the bottom of the minerals food chain, shipping unrefined materials abroad only to buy back the products they become at a hundred times or a thousand times the price. So, if we want to learn from that experience and not become "permanent renters of overseas intelligence" to quote from Minister Charlton, then there has to be some discussion over weather and how we should be looking to compete in the model and application layers. Now, actually, on the latter, we're doing quite well, leveraging our success in SaaS and Enterprise. But what of the models themselves? Today's episode is all about this question, and specifically in a world where it feels like building a globally competitive frontier model lab like Anthropica OpenAI is off the table, what else can we do, and what hope do we have in succeeding in it? To bring these topics to life, we're joined today by two founders of specialized foundation model companies, PIP, from Springboards, and Jesse from Solaris. They're bullish on Australia's opportunity. And to draw out the nuances and edges of this debate is Blackbird Partner Tom Humphrey. Over to Tom, PIP, and Jesse. PIP, Jesse, awesome to have you here. We've got a lot to get through, so I'm going to get cracking straight into it. Now, I was deciding who to start with first, and PIP, I've decided I'm going to go with you first because you released your model first, and I've got no other better reason to do so. So you released Flint Alpha in April of 2026, Jesse, you released Solaris one in July. So can we get a quick background, PIP, on the Springboard story, and maybe walk us through just how you ended up training your own model? Springboards is primarily focused on creativity through the lens of advertising, so for marketing advertising teams. I believe though, with creativity, that will always be a very human endeavor. And because of that, what we try to optimize for is sparking inspiration, and in ideas in people instead of giving them the answer. Where we got to for training a model is that what we started to realize is, as we were onboarding users onto the platform, which started off as a web app. Effectively, a GPT wrapper, like four years ago, is people kept saying to us, hey, the outputs are good, but we're finding their repetitive, even when we're changing brands or doing different tasks. And so we thought, oh, there's something interesting there, so we would introduce more models who would make the backend prompt injection way more complicated to drive more variation, more inspiration, and other tools. And basically, people would still say, hey, we're still getting the same ideas that he platform. And so this is about two years ago, two and a half years ago, we go, we realize that there's a much bigger problem that play with language models. In that, regardless of who you are, where you live, what model you're using, they're all converging to the same space. And so we started researching why is this happening, and also looking at ways of training a model to give you higher variance or entropy, particularly in open-ended responses, without degrading the base performance of the model. Because, as I said, we believe in creativity, human ideas, and in order to spark different ideas and send people in different places, you can't be sending them into the same space. Can I maybe give an example to bring us to life? For me, what's been quite helpful is understanding that when you're serving with creative industries like marketing, if you go to an AI model, and you say, give me a random car for this advertisement, every time it's just going to produce Honda Civic or Ford Falcon, and then every advertisement in the world is just a Honda Civic and a Ford Falcon. It's just these models are averaging to the main, right? Is that right? We use words like intelligence and reasoning and creative thinking for models which don't have any of that. Okay, and so what all models are is next token prediction machines, they're probably the machines. And so when you ask these models questions, they send you the probabilistic, the most answer. Okay, and so if you ask a model for a random number between 1 and 10, regardless of any model in the world, you'll get a 7. If you then say another, you'll get a 3. If you say another, you'll probably get a 9. If you say give me a car model, any, you'll get a Toyota Corolla followed by a Honda Civic, followed by a Ford Mustang. If you ask for a tagline for new balance, you'll get run your way or run your world. If you ask for a tagline for Nike, you'll get run your way or run your world. And so because these are probability machines, which are trained to give people answers, when you're going to creative fields, you end up giving up in the same space, same answers. And so what that ends up doing is creating everything starts to look the same, because if you feed everyone's same answers, you end up in the same space. And that's because a lot of these models are kind of trained for coding, physics exams, math exams, these are like deterministic things like where there's a right answer and a wrong answer. It's just like, what's two plus two? It's four, whereas like creative purposes, there is no right or wrong. Like there's gray, and you're like looking for, I'm not looking for a right answer, I'm looking for inspiration. Is that right? 100%. Yeah. So if you look at all the standard benchmarks that all the major labs competing on, it's science exams, it's physics exams, it's agentic capabilities and long term horizons and vending benchmarks. And in those worlds, there is a right answer. There is a deterministic output. But if you think about creativity, creativity gets used up. So if you give the same answer to everything, everything looks the same. And so novelty as a, I guess, a condition of creativity is that you need high entropy, need high variation for that thing to be useful. Otherwise you get bored of seeing whatever it is, like over and over again, it's why like slop that's coming into market and saying, oh, that's slop, that's slop, that's slop. It's not really slop. The writing's actually kind of good. What makes it slop is that everyone's fucking using the same three or four phrases, because that's what language models are doing. Yeah. And so the problem cascades, it starts with language models, but it cascades into image models, it cascades into coding models and everything that you do, because these models are, if you go back to what I just mentioned, is that they're next token probability of machines. That's what they designed and optimized to do. So you were building the software application, which was like you said, like the wrapper on the existing models, you realize that hang on a second, we've got a problem with the models themselves, like the models, the problem we're not getting the creative output from that. So you go. I need to go fix that. So what do you do? Like walk us through the first step of getting flint. Research. And so we brought on one of our founders has a PhD in ML. We brought on another guy with a PhD in ML. And we started reading lots of research papers and doing experiments. That was the step. Okay. And then you go, ah, fuck, that doesn't work. And then you just try something because you know, that doesn't work. That doesn't work. That doesn't work. And you finally go to start playing with different techniques. You go, oh, there's something here. And is this like a fundamental problem of the existing models? Like, you know, like when I go to church, you'd be like, there's like a temperature button or a randomness button. And you can kind of go, let like ramp up the randomness like, can I, can I kind of just prompt my way into creativity and say, hey, don't give me an answer upon the civic year. Like, give me something random. Yeah. This is like major misconceptions is that it changes the phraseology of the output, not the actual the creative idea behind the idea. Yeah. And that's very true temperature as well. So everyone will say, temperature is your creative dial or your randomness dial. But there's a big problem with temperature is that like, I'll take open an example of their temperature settings on a scale of zero to two. As you dial up temperature, and you let's say you ask it for a dad joke, tell me a dad joke, you'll get something to do with the scarecrow every single time, even if you change the temperature from zero all the way up to 1.9. And what you'll start to see is that the way that the joke is phrased to the intro, the outro is slightly different. But the punch line, the core underlying idea is still the joke about a scarecrow. And it gets to a point where you put it up to its max temperature rating of two and it collapses into like catastrophic failure. And so the reason for that is the way that temperature largely works is that it increases, it attempts to increase the entropy at every single token all at the same time. And so you don't end up actually changing the probability distribution then until like falls off a cliff. And so what we do is we train, we just call it a training method that we train for what we call critical tokens because you don't always want variation in your answers. Like if I'm asking what's two plus two, I want the answer for every single time. But what we want to do is identify at what point in a sentence is that a critical token where entropy should be higher, where variation should be higher, where the answer shouldn't always be a seven or a toy to camera or run your way. And then you want to increase the options that the model is allowed to like give out and it's reinforced. Yeah. How do you measure, like measure for this? Because like, you know, artificial analysis and these benchmarks out there and like all the models when they're released to put through like physics exams and tests to kind of go, this model is better than that model. And it's pretty, pretty straightforward because you know, you know, like we said, it's like right or wrong. But when it comes to creativity, like how do you measure that? Like and or maybe put more bluntly, like how do you know that your model is more creative than the others? Yeah. And so just an awesome question. It's like creatively subjective. So to create a benchmark for something subjective is impossible because anything is good, it's different to what the next person thinks is good. So instead, what we do is we look at the number of answers to a given question. And so we look at like, let's say we ask the same question 10 times and we see how many of those 10 questions, how many times we get the exact same answer, how often do we get a variation with that? And so what we do is measure variation and it's not us for up. It's I don't believe it's up to us to decide what is good creativity and what's bad. That's the person's job. Yeah. I just want to make sure that we give everyone different answers and different ideas and send them off in different directions and you leave it up to the person decide what's good and what's bad. So we've got our own internal benchmarks where we measure slop coherence and variation on a crossback 12 15 different dimensions. There are other benchmarks out there like novelty bench, a few others, but there's not many that actually measure creativity. Well, okay, and like we're to here, like if you could just like a maybe paint a picture of like where you're at now in terms of like where's the model at now? Where do you want it to go? Is there is there some sort of like compounding advantage to the model like feedback loops or reinforcement learning as you have these users that kind of give these feedback to make the model stronger? Yeah, so the feedback loops are interesting. So that's why I think it's interesting that it's kind of like the reason why you do want to play at a application layer as well as a model layer because your application layer becomes your feedback loop for your model layer. And so you want people playing with your model so you get insight into what's good and what's bad. So in our tool, we have a thing called just pin in or highlighting, which basically is a suggestion that's a good output versus bad output. And we can then take that into reinforcement learning. We don't train on users like individual outputs, but it gives us insight into what is good and what's bad. Yeah, that's kind of the main bulk of it. I mean, that's really interesting. I mean, I feel like we'll come back to this in a little bit, but like this idea of like actually hat getting user feedback and even like these platforms like LLM arena and popping up where it is just about hey, which model gives you the best output and getting that feedback is so valuable to the LLM kind of development. Right. I'm going to switch over to Jesse. I'm still amazed by your journey in life from first job, Abitwa. And now here we are talking about LLM development, but you and Tom Hamer also started as an application company first with Marco. And that was search infrastructure e-commerce before sort of launching Solaris, which is AI Lab. Can you walk us through that journey from Marco and when that aha moment came for Solaris, if you will? Yeah. So a bit of background. Yeah, we started out. In fact, is a sort of open source search engine. We sort of recognized quite early on that search was undergoing a bit of a transformation. It's one of the oldest sort of AI technologies it's been around for a long time. It's absolutely critical to just effectively everything that we do in our lives, right. Being able to find information, retrieve information, it's probably one of the biggest unlocks that humanity's had in terms of being able to sort of progress knowledge work. And so it had been sort of fundamentally just based around keywords and sort of matching keywords. It didn't really understand meaning. And what we were seeing is that there was this sort of entirely a new paradigm for search, which was based around vectors and embeddings. And so representing concepts is a series of numbers much more abstractly, but this allowed you to capture so much nuance in people's language and you could really you could find information and you could do search that just wasn't possible before because of the sort of bias in this keyword system was so much that you could never get the results that you wanted. And we sort of focused on that initially just because it seemed like a really good opportunity, really interesting. What we sort of saw as well with ChatGPT coming along was everything changed very rapidly overnight, right. We had sort of LLMs appear on the scene. And so you know, really started to think at that point, you know, what is, you know, what is the future of search? Look like we're going to be cannibalized by an LLM. And you sort of, you know, start to think about, you know, what are the invariants here and you know, search and doing that fast is actually, you know, say that as an invariant. That's never going to change. And so even with LLMs, they in fact has become far more important because now they ground all their answers in search. And so searches become even, you know, sort of more, you know, important in the LLM era. So sort of, you know, we're developing the search engine with that sort of lens. And then, you know, we sort of found ourselves, you know, in e-commerce, we've done a lot of work around, you know, images and sort of multi-modality again, it being kind of neglected and particularly in sort of e-commerce search. I think it was very relatable, right? Everyone's sort of got these experiences where they're trying to find something and they simply can't, doesn't matter how they try and express it to the search engine, you just can't get your results back. And so we sort of, you know, started to, you know, really focus on multi-modalities are using text, images, a lot of this sort of unstructured data, being able to search over that and really develop, you know, Marco into this, you know, what is, you know, the best, you know, search engine free commerce and sort of powering, you know, tens of billions of dollars of e-commerce, you know, globally now. The key thing about search as well is it's, you know, real time, right? Everyone has, you know, if you're waiting 10 seconds for a search result, you know, you're not going to really, you know, persist very few, you know, users will. It's inherently a real time, you know, task. And so we developed, you know, we've been training our own models as well for search, you know, for the retrieval, getting significantly better results, you know, has to be fast. So it's sort of, you know, tens of milliseconds, 100 milliseconds. And then, you know, as we develop these systems, you know, building recommendations, also, you know, integrating, then, you know, chat-based, you know, experiences, speed was just becoming even more, you know, important. You know, one of the interesting things about LLM's is they really allow you to do so many tasks, you know, with a single model, you can use text, so it's interpretable. That's a key thing. And this was quite different to sort of traditional machine learning models. It might be much more sort of abstract. You have to define features and sort of a lot more work. And so LLM's themselves, you know, provided, you know, this real opportunity to be able to do a lot of other things that were sort of beyond, you know, the reach of traditional LLM, but then they were too slow. And so then, you know, what happens is, once you actually combine, you know, really fast, you know, LLM's, you know, it really unlocks the whole heap of these real-time machine learning systems that just sort of hadn't been possible, you know, the complexity goes down, interpretability goes up, and just a sort of number of applications really sort of fans out. So that was really what, you know, sort of precipitated, you know, the development of, you know, essentially the fastest LLM's because, you know, in our mind, you know, this is really just untouched. We're just at the beginning of what this means. These real-time decision-making systems. And so really seeing an opportunity to develop that. And so that's really how it came about. So it's all about speed. And obviously this was born, as you said, out of e-commerce, where it's like, you're searching for something online. You care about how quickly you get those results. Maybe walk us through, like, Celerus 1, you've launched your first model. Walk us through, like, what have you launched? What does it do? And what are some of the kind of applications you're finding outside of e-commerce that, where speed matters? Yeah. Yeah. So Celerus 1 is the first sort of model that we've launched, you know, the fastest LLM has sort of been benchmarked on, you know, independent benchmarks, artificial analysis. We've since launched another one. Celerus 1, Magnus as well. And this is sort of, you know, the reason we have, you know, sort of multiple models as well as you can sort of think of this Pareto frontier of sort of speed and intelligence. So, you know, a lot of decisions actually don't require a particular, you know, a lot of intelligence. And they just have to be fast. Some require, you know, a bit more intelligence and sort of higher bar. And so, you know, having this ability to kind of, you know, move along that the frontier is sort of, you know, one way we think about, you know, how we're going to serve our customers, you know, in terms of the model releases. Speed matters. In terms of speed here, what we're doing, you know, the way we actually think of about speed is time to first token. So when do you start getting a response, and then as it's delivering a response, how fast do those tokens come out? And depending on your application, it might not only really care about how fast you get the response because maybe you only need a classification. So maybe the actual tokens that come out are very few. And so your round trip time is what's important, or maybe you're doing coding and you actually care about that. So it's sustained throughput. And so there's different ways to think about speed. They sort of depends on the application. But again, yeah, so beyond e-commerce, what we sort of see, search, there's obviously a lot of applications in that. There's a huge opportunity in voice systems and these real time systems we're seeing a lot of uptake in those domains. Voice is really a very strong commercial traction. I think we're still only at the very start of what voice can enable. I'm not sure if you've used it yourself, but certainly being able to dictate, now it's an amazing way. You can sort of dump all of your thoughts and then sort of aggregate them and it's much faster than typing. And it allows for much more fluid kind of conversations. Controlling interfaces with your voice, I mean, it starts to really become quite magical and what you can do there. So that's sort of one example. I mean, you can think about every transaction on the internet, online transactions. Now you can have an LLM. In them, you can have real time interfaces that change, even fun or sort of abstract and exciting things. When you think about, if an LLM can generate code, sort of almost instantaneously, you can have a computer which is an LLM and you don't even have a program. You ask it to open up a text data and it literally will create the text data when you ask it to. And so all of this now can happen just as you sort of ask it. And so there's a whole host of these sort of paradigms that we haven't even entered into yet. - I was thinking of you yesterday, I called up the dentist to book and they had a voice AI kind of robot on the other end of the call and just like, I would give a response and it would take like 10 seconds to respond to me. I'm just like, where is Salaris right now? - Exactly. - That's what response is. - Exactly. And I think what we see as well, we've seen in some of the internal, with the chat applications and sort of using commercial models and it might take 15 seconds and we switched over to Salaris 1 and it was 1.5 seconds and it's just, the usability and the experience is just so different and I think just vastly improved. - Yeah, maybe I don't want you to give away your secret source here and maybe you're obviously technical but like let's try to pretend I'm your mate at the pub who has no idea about tech. Can you try to explain this in some way just in terms of what are you doing in terms of training this model, some of the techniques you're applying to kind of achieve these results? - Yeah, yeah, first I'd say all right, it's your shout, right, let's give it a go. (laughing) But in terms of the, yeah, the techniques, so I think one of the things that were, as you know, anyone listening as well about sort of large language models and machine learning, a lot of the concepts are actually, relatively easy to understand. I think once you actually get into it and sort of high school level maths and physics will get you a long way to understanding sort of how these things work. And you know, the technology around them has been pretty consistent for a long time. And this sort of I think is also exciting because we're still using sort of relatively simple techniques that are obviously very powerful and so there's a lot of scope for improvement. But you know, the way that they work now is you know, they sort of a language model that will generate, you know, as Pip was saying, you know, these sort of next token predictions and so they essentially generate one token, you know, at a time a token here is basically just a subset of a word. So you can almost think of it like, you know, yeah, they're generating one word at a time. They don't change anything that they've, you know, already said and they sort of, you know, just keep doing that. And so it's very inherently sequential. And so each time, you know, you want to generate a new word, it sort of passes through the models is sort of known as a forward pass. And so you incur quite a bit of cost at each time you're trying to generate a single word. But that's not really the only way that you can, you know, generate, you know, language or words, you know, in a computer. You can actually start to think about doing, you know, more parallel methods. And so what we've seen, you know, in the image generation domain, which is, you know, probably one of the earlier places where we saw a lot of, you know, generative AI, is that they can, you know, generate images, you know, kind of in parallel. So rather than sort of, you know, going sequentially, you know, generating a single pixel at a time and sort of like, hey, let's generate, you know, let's get a first pass. It's pretty rough, you know, it's not that great. So you've got entire canvas now as, you know, filled. And then let's just refine it. So let's pass it back through and refine it and refine it and refine it. And so you're sort of generating everything in parallel, but then refining everything in parallel, rather than, you know, being very, very precise and generating, you know, one thing after the other. And so, you know, an analogy in, you know, if you're thinking about, you know, maybe painting is, you know, instead of being incredibly careful in doing one brush stroke at a time, going from left to right, top to bottom, you know, you sort of flow down, you know, a few broad strokes and then a few more and then you sort of, you know, refine it and you can sort of see it as a whole. And so you have this context of the entire canvas as you're refining. And so that, you know, really opens up, you know, other ways of, you know, other, you know, utilities for these technology, but also means that you can do things in parallel, which is faster. - Okay. And, and where are you at? I asked these question of people, like, where are you at right now? You've just done Magnus, but like where, what, you know, where are things out with the model right now? Where do you want to take it over the coming, like six months or so? - Yeah, so I think we're very focused on speed and delivering intelligence, you know, as fast as we can. And so we're looking to, you know, continue to move up that sort of Pareto Frontier. So more intelligence and faster, you know, and this is a combination of things you can get benefits, you know, from best, better hardware. You can get better benefits from sort of better software. And then you get better benefits from, you know, sort of architectural and model changes. So, you know, what we really want to do is, you know, get to, you know, something like, you know, essentially a Frontier model. But, you know, if you get that a thousand tokens a second, you know, again, it really changes what's possible now. You can generate, you know, instead of waiting hours for, you know, your code to generate or your, you know, campaign to generate, you know, it might be tens or, you know, tens of minutes. And the key thing here is essentially just iteration speed, you know, if you think about, you know, maybe you need to take one thousand steps to get something done. It's like if you can do those thousand steps 10 times quicker, you know, you just can do so much more with your time. And so really the key thing is just about, you know, intelligence as fast as you can. - It was quite fun to be involved with you guys as you launched the first model and just exciting just to see the reaction, you know, just artificial analysis, did their benchmarking and then just even on Twitter, just some of the virality. I'm just kind of curious like what you, maybe if you could highlight like how the launch went, what were some of your learnings around? I'm conscious that like, you know, there's a lot of models out there, there's a lot of noise, like, and you need to like really kind of put your stamp out there and how, what did you learn to about distribution and getting out there? - Yeah, I think a lot of the same things apply here as they do in other places, you know, sort of, sales and marketing go a long way. And it's not necessarily about having, you know, the best product and whatnot. There's so many good products for like, there's kind of just, you know, go by the wayside because they're not marketed well or even exposed or, you know, given the opportunity. And so you've got to really think, you've got to be thoughtful about it, not just rely on it being, you know, really good product. You've got to really think about how do I actually, you know, show people and not necessarily tell them. And so I think that's been a big thing is, you know, how do you articulate or explain or demonstrate what you're trying to do in, you know, is such a, in a compelling way that someone can understand very quickly. So, you know, use of visuals is a very strong thing. I think that that's very effective is, you know, what's a compelling, you know, visual artifact that you can use that someone can look at it and be instantly understand, you know, what the benefit is or why it's better. We learn a lot, I think as well, from, you know, Marco originally being an open source search engine. So we did a lot of our, you know, original kind of marketing for that, you know, through, you know, technical posts, you know, Reddit, you know, Twitter, LinkedIn. And so we learned, you know, quite a lot from that. And again, you know, it's the sort of same things that matter as, you know, just sort of high quality content, you know, what can you show, you know, to demonstrate the value and then, you know, sort of go from there. Again, and you just have to be nimble, you have to iterate. Something doesn't work, try it again. Sometimes you have to post it twice, you know, just don't, you know, don't really worry, right? Like, and I think, you know, yeah, don't worry about it. Also, if it has to be perfect, but just kind of keep doing things, keep iterating. Yeah. And again, it's just about, you know, taking as many steps as you can. Could you maybe give us like a really, like an interesting example of like a customer use case, or someone who's come to use the platform when, you know, the other models didn't serve them. Yeah, totally. I'll give you like two examples, like a traditional built for agencies, more and more clients coming to us. And so I'll give you two examples. One is like L'Oreal, who's just helping their marketing team come up with everything from value props to naming, to legal checks, to briefing the agencies better. But I think the more interesting one is right games who have recently started using us, particularly to come up with like world building and character development for nothing to do with advertising, but more to another creative field in the gaming field. And so like the way that we think about creativities, yes, we were built originally for the advertising marketing industry. But that creative field and a model built for variations is so many more applications, be it like telling shitty dad jokes, working out what you should eat for dinner, all the way through to gaming and everything else. And so really as we start thinking about springboards now, as you make it applicable to any creative field, any open-ended ask, why by combining variation at the model are collaboration with people, very, very important. So people can actually work together instead of in silos and inspiration from the real world, not just language models themselves. - In my own personal example, I've started every night when I'm putting my young boys to bed, I go to like Chatchy PT and I just say, hey, I'll like pick a time, like give me a 12 minute story and it's featuring Taz and Hatch, my boys names. And I want it to be set in India and there's elephants and you know, just kind of like get a story and then I don't know like Messi and Pele kind of turn up or whatever and I get it but I've noticed like as I'm doing these different stories, there is like a common guarantee that all the same under all ideas. - Always the same structure and it's just and I think they're even picking up on it. - The quickest trick is that like people don't realize it in a language model, there's a little refresh button. And so next time you do an answer, click that refresh button and just see what happens and you'll see that you get the same type of idea time and time again and that is the experience that everyone gets and what most people don't realize is when they use the language model, today. the getting one or two personal stories about their kids and whatever else. There's probably another guy reading to his kids in another country just changing up the names exactly right. And so this homogenization happens everywhere, be it like the stories we tell our kids, the ideas that you see out into the world, to the shitty dad jokes that you're telling the weekend. And so like this problem is just so much bigger than what people realize. And the last thing I'll mention on that by the way is that the problem only compounds. And the reason why the problem compounds is that language models have become the orchestration layer for everything. And so even when you ask a language model for a random color and ask it for a hex code and then map those to 250 times in row, they're all blues and purples. And if you like notice the proliferation of blues and purples on websites right now is because these models have a bias for those colors. And so my point of view, or our point of view is that if you fix variation at the language model layer, who is then orchestrating agents, orchestrating video models and ideas, is that you actually start to get a more colorful world for all sorts of different stuff. And so this problem variation is we're obsessed for this once specific issue, but there's just so many more applications of it. And no one's cracked it because no one, like the big guys don't care about it because they're too busy competing on coding agents because they believe that if they crack coding agents, then you can do whatever you like. But still that you end up coding all the same issues in the same box. And so yeah, I just think there's so many more applications that we're only really just scratching the surface on. Okay, Jesse, like let's break out of the e-commerce use case. Like can you maybe talk through like an interesting customer example with Solaris, like where that speed will really matter for them? Before I sort of enter into any of the specific examples, one way to also think about why speed is so important. We saw with reasoning models that this sort of had this paradigm of test time compute where you could basically improve the intelligence and the quality of the outcome by spending more time at inference and basically letting the model kind of think and letting the model kind of refine its answer. And that's basically what speed also buys you. So if you think about being able to execute one search or one invocation of an LLM in one second, but if you can refine it a hundred times, then you can essentially you're much more tolerant of mistakes in the system. So that's sort of one of the ways that you can actually scale intelligence as well as through this sort of test time paradigm, which is related to speed. In terms of interesting applications, one is around a lot of entertainment and games. So yeah, one another one in sort of applications for learning. So seeing someone having language learning, for example, and you're interacting with an avatar. And so if you've got a language model on the other end, you can now learn, say, have conversations and learn another language in real time and interact with an LLM, which wasn't simply possible before if it starts to take a lot of time. Same thing moves over into, say, game development or the entertainment where you can actually now have living kind of environments. Those kind of real time adaptation use cases is certainly something that we've seen. And I think that's incredibly interesting. All right. I want to like now we've introduced both your projects, kind of dig into a couple of themes. And the first thing I wanted to dig in on was this idea of application to lab. And I find this quite interesting because if I reflect on a lot of the journeys of the big, well known labs, like, you know, even the Chinese, but also, you know, anthropic and open AI, they started as research labs, they trained models, and then they kind of built out applications like Claude and Chatchy PT from there. You've done it in reverse where you start off as application companies, macos, springboards, the platforms, and then you've kind of like reverse kind of turned into, if you will, AI labs. So maybe PIP to start with this. And you hinted at this when you were talking, when you were mentioning this sort of like the benefit of the user feedback loops, but I was also just specifically like, is there any benefit to having started an application company going into model lab or any, what's the benefit of having the application company there whilst you're building and training the model? Yeah, I think application like gives you like a perfect concept really fast. Yet people play with it straight away. Obviously at the feedback loops, you get an audience space it's ready to go. But to be honest, the biggest thing is that the open source world has gotten so damn good in the last year that it has changed the game a little bit. And so like, if I think about it, like, do you really need another frontier lab in the world when you've got amazing open source models in China? You've got more like thinking machines popping up in the States, which are getting funding now across Europe. And so the idea of starting another frontier lab to me is kind of insane when there is just so many good options out there. Like, it just depends on where the market's at the time. And the market just evolves so quickly from where we're at. And so a lot of companies like to get into the frontier model game, it's not rocket science. It's a heap of fucking money, a lot of compute, and you can get into it. And so that's been the game for the last three or four years. But now that you do have these open source models, like it kind of changes the market. And so you had a whole bunch of companies take advantage of those frontier labs and use their models into applications. And now I think they're going to go, oh, shit, actually, we can train models. Yeah, just using price training techniques to get to really interested outcomes. So I think it's going to start to flip over time. I think it's really interesting. We had some researchers from the base 10 team just on the podcast. And they were talking about how all software companies hit a certain scale, I don't know, called 50 mil ARR or 100 mil, where they're now getting into model training because they're at that point, they've got enough data internally, which they can train on, but also just the enough of a willpower in terms of like, I need to control my own destiny. I need to kind of have some mode here and get the costs of these third party models that I'm using down. So like, all companies are going to kind of becoming hybrid software and model development companies, whereas you guys have kind of taken that journey very early in your life cycles. And I think that's really interesting. But do you find that too? Jesse, just in terms of like having the market or application being able to train your model, push it out to your customers, there's that benefit of that feedback, Lloyd? Yeah, absolutely. There's that benefit. I think I think it's slightly different take to what you were saying. I think actually the history is sort of littered with companies that started out as applications or consumer, look at Amazon selling books, Facebook meta, it was a social network, LinkedIn, Walmart, all of these companies have sort of started out very much just trying to sell something and becoming a viable business. And then over time, you know, have needed to invest in research and their model development and machine learning. And so, you know, especially like you think of Yahoo eBay, you know, these early companies, we're really at the sort of, you know, forefront of machine learning a long, long time ago. And they started out, you know, sort of as application. So I think almost, you know, the research first approach is actually something relatively new compared to, you know, these companies. And yeah, exactly right. Like, you have this sort of virtuous flywheel where you have a viable business and then you reach problems where, you know, to keep growing the business, you need to start solving sort of harder and harder problems, you invest more in research, internal capabilities, that feeds back into the product and you have this, you know, ideally have this virtuous sort of flywheel. And I think it's kind of interesting to see, you know, like you said, these companies, you know, investing, you know, once they hit a certain scale, they're investing in their own model development. You know, I think again, like this is, you know, nothing particularly new, it's just a slightly different form, you know, where people are, you know, focused very much on, you know, one set of machine learning models being LLMs, but, you know, sort of entered into tech, you know, over a decade ago and everyone was training their own models, internally churn models, you know, marketing models, ads models, you know, everyone had their own internal teams, you know, at least in the US, you know, developing the models. And so I think it's, you know, it's, it's similar to, you know, what we've seen before. It's just now very focused on, you know, a particular type of model. Like, you know, walk me through like, you know, like in terms of talent, compute, like, you know, data, like what, what does it actually take? Can you just walk us through like high-level laments? Yeah, so I think you understand what, yeah, what does training a model actually mean? So, yeah, the model itself is, you know, a sort of parameterization. So there's these parameters in there. In this case, it's often just values, you know, that might occur in a sort of mathematical function. You can pretty much just, you know, smash as many sort of functions as you want together, some matrix modifications. And these are just sort of numbers. And so when you're training a model is, you basically want to, you know, make those, you know, numbers, you know, that make up the model, you know, form in such a way that when you have, you know, an input, it goes through these operations that gives you an output that you sort of want. And so, you know, training a model is really about replicating, you know, the data. So you'll have inputs and you'll have outputs. Maybe the input is a, you know, a customer and maybe the output is, yeah, churn probability or something like that. And so what you're trying to do is learn this mapping between these inputs and outputs. And so training is that process of, you know, determining what are the parameters that will map my input, you know, to the output that I want. And the way you do that is, yeah, you need data. That's kind of the key thing. That's really the one of the, you know, the most important thing is data and it's representative data of what you're trying to do with your task. And then you, you know, parameterize a model. It can be very simple. It can be a linear model. It's just a few parameters or it can be, you know, one of the most more complicated ones like a large language model, but they're all fundamentally, you know, very similar in the sense that you have these operations and these sort of parameters. And you really just define, you know, your task through what's sort of known as an objective function. So you sort of, and that basically just scores, you know, how well is the model doing on this task? So you have inputs, it, you know, it gives you an output and then you just score it, you know, how well is it doing? And then from that score, what you do is you say, okay, we want to improve that score. So how do we improve that score? Well, we need to change the parameters so that it gives us, you know, the outputs, which are more aligned to what we want. And you sort of just, you know, do that. And you basically, yeah, using, you know, calculus to, you know, move these parameters into a way that it gives you the outputs that you want. So that's sort of the simplest, you know, way to think about it. As you add more parameters, you basically need more data to sort of, you know, learn. And that's why we've seen such an immense scale with large language models, and that's was the biggest unlock, you know, I think around 2019 was people realized, you know, okay, what if instead of you training on, you know, 50 gigabyte? of data he's trained on five terabytes of data or 50 terabytes of data and so scale just became you know a huge part of that. Yeah and so if I'm not mistaken you take an open source model and then you pick your model as the base depending on what your objective is and then I'm going to bring more data more techniques of breaking that model apart and doing different techniques to it in order to kind of like change that model to be something different and unique and specialise for what I'm trying to do. Yeah yeah that's right and so the way you think of it is yeah the the model will have been trained with a particular objective function and so even with model training you'll see there's different stages so you'll hear about pre-training you know post-training they're sort of the two you know predominant sort of paradigms and in the pre-training regime you sort of take just large amounts of say internet text and you predict the next token this is very sort of straightforward kind of objective and then you know in the post-training you change what your objective is trying to do and so you make it much more aligned you know to your sort of your task and so you know the key thing is really about changing the objective that you're trying to get the model to do and that's why you know when you think about specialisation and I know this is sort of a big theme here is you can you know you can pretty much always guarantee that you'll get better by specialisation because you've got an objective which is very specific to your task and once you start optimizing for that objective you'll get better at that task for that objective and so you know it becomes it always becomes a viable method of improving performance on your domain is to you know basically train the model for that task I want to bring you in this yeah I want to say that what what surprised you about training models honestly it's surprisingly simple so thing that's it be to do it well it's fucking hard but to train the model is not that hard particularly post-training and so like my background is not technical whatsoever we've got a founder who does that and he runs an ML team but about a month ago I wanted to learn how to train a model and so I taught myself how to train a model staying up to like 12 midnight one o'clock at night train the model and like if you've got a really good open source model that you can train on and you take a set of data and you can create prompt completion pairs of input and output you can then what was the data you were using I created a crazy set of data set to see if I can train a model to think it's insane and I did and it's fucking hilarious and it was just more out of curiosity because I'm sitting in these I am on meetings each week I want to know what's an epoch and you know what are they talking about when they're talking about X Y Z so I took a bunch of sources of information which were for my crazy sources boiled them down and rebuilt them up in a synthetic kind of data approach and then balanced all of that training set so I spent way more time just balancing the data making sure that there's examples of short examples and long examples and bullet points and and numbered points and talk about into prompt completion pairs and and then like basically train the model with the couples like basic type of parameters and so the thing that surprised me is that like it is you don't need a heap to train a model you don't need a massive data set you need a really good data set you also don't need a heap of money if you're training small models and you don't even need a heap of GPU if you're training small models again you can I trained a local model full billion parameter model which works really damn well and it's interesting out of curiosity yeah and so if you got that curiosity and you want to learn it's not hard to learn there's so many resources out there that will help you do that and so yeah I've been surprised at what you can do with a little what's the value of data now I mean like you mentioned just sort of like synthetic data I'm trying to like understand you know as a new but like you know you've got synthetic data you've got real data the internet is kind of trained on also on the internet is like a whole lot of AI kind of polluted data now as well like you know you've got private data which is you know what's the value of data and the equation and and you know my personal opinion is that a vision and an idea is more important than the data because these models are getting very good at creating synthetic data very cheaply that you can train so like you know deep seek models are amazing models very cheap and if you complete open rights to train on that data so creating your own synthetic data sets is not that hard now you have to be careful in the way you do it but personally from what we're doing I don't think data is going to actually data will be probably we'll be out because we handcraft a lot of it and there's a certain way that we do that but without the idea of what we want to train for and why we want to train it that way and having that customer relationship and the feedback looks exactly right yeah yeah I think it's a very context dependent the value of the data so something you know data that's valuable to PIP is probably not going to be valuable to meet all because it's depending on the task but the whole point of gain is like talking to the training side of things in your objective the better you can sort of align you know your data and with your objective and what you're trying to do on the end task the better your results will be and so you know data from different domains is going to be pretty useless because it's not aligned with the objectives that you're trying to do and so you really want to you know just optimize the problem that you're trying to solve as directly as possible and one of the best ways to do that is to get data that reflects that you know data you know dictates a lot of the behavior and data really you know dictates the outputs and so you know aligning that with the objectives is really key and so yeah using a mix of data sources again it really just depends on what you're trying to do but really making sure that your data is aligned with your end goal yeah and that can be just very domain dependent yeah it also comes in I think the other key thing to think about is you know the internet scale data you know using a sort of crude oil analogy is actually quite good you have like just raw data that you pull off the internet is very unrefined crude and then you know really much higher quality you know distillates you know data can be hard and you know can get you disproportionately far you know it's it's worth you know I spent a lot of time you know we knew the value of this a long time ago you know at Amazon when I was working in robotics and we would have to you know build classifiers that had to have incredibly high you know precision you know sort of 99.95 percent you know it was worth my time to sit there and annotate you know images myself or eight hours that you know literally thousands and thousands because you know hand annotated images from me who knew exactly what the task was you know for hours you know an hours was a very good investment because it had disproportional benefit to the outcome instead of you know trawling the internet and getting a million just average ones it wouldn't even come close yeah that's interesting it reminds me of like Harrison AI for example like creating a model for radiology going through and getting like iMeds radiology scans and then just annotating that like it's like you got to create the quality of the data yep that's quality in quality out you know yeah and in terms of you know actual model performance there's basically no better lever than improving the quality of a data you know people talk about architectures and optimizers and you can you know you get you grind out one or two percent you know on this but if you change your data you can get 10 20 30 percent it's by far the biggest ROI yeah okay i'm going to change topics i want to get into a new topic now which is model specialization and i think that this is probably one of the more important take questions conversation because many people out there are probably going to be listening to this podcast and at this stage going come on guys you're down here in Australia who are you kidding like you're you're going to take on the big us and Chinese labs in this LLM game and they've got their billions of dollars but the answer to that is specialization and we've obviously been dangling at this but you're specializing in speed you're specializing in creativity and these are sort of like blind spots for these big companies and the opportunity for like Australian labs or labs anywhere in this game is not to be the number 20th in the generic kind of LLM game but to be like number one at something specialized and something specific so i wanted to kind of like dig into that like you guys are both doing something specific you know creativity and you know speed maybe i'd love to kind of hear your kind of take on this idea of specialization and am i right here like you know when i say this and do you do you see like the opportunity here to kind of like carve out like the opportunity in this nation be the best in the game and and that's like meaningful enough massively i think for two reasons one is no one wants one tool for every fucking job in the world and it's like giving someone a hammer and go the here's your tool and you're going to make everything with that hammer does make sense that's why there is specialist tools out there so there's one whole stream of conversation on that the other thing though i think which is a bigger clue for me is that i think one of the major innovations in i guess the language model world over the last kind of 12 months has been an orchestrator or a routing layer which goes user prompt says this therefore send send to this model that tool or whatever else and so rather than creating one behemoth model that is apparently good at everything my personal take is that you would be way better and having a lot of bespoke individual models for all sorts of different tasks because you've got a really good orchestrator and a router at the top of that which knows that you can then post train models for certain tasks and so i think like there will be a world where you just have a massive model as a makeup of us a bunch of like specialist models stitched together yeah so like i'm in chat gbt i'm asking like hey my year olds got a sniff so already how it sends it off to the healthcare model which is trained specifically on healthcare data and it's not sending it to some big chunky expensive model which has to be trained on everything is it's already it's already happening where the user comes in and they go oh shit is this a reasoning request does this need a high intelligence or low intelligence is this whatever else in the sense it to the appropriate model they just took within that one family and within that one model's capability and so that's already happening personally i think that's the way it's going to go is that you just got a whole bunch of specialists post train models that all get stitched together which will get to a better outcome and you're one of those models like hopefully when i when i'm you know i move on for my question about my eight year old with this The sniff and I go actually, got to do a poster art for tomorrow and I got a creative question. It's going to triage to springboards. And then I've got something which requires a really speedy response. Everything that's true. But do you like paint the picture like, like let's put on our national hats like Australia? Like do you see this is like a broader opportunity for Australia? Like you know, Australia model development opportunity. Like is this like the thing? Like we should lean into like what are the specialties or the areas that are meaningful enough where you can build like a kind of like a meaningful enough model in terms of like it's it's in demand and you know, be the best of that. Yeah, absolutely. I mean, I think there's never been a better time, you know, than now to actually enter into this game. Certainly, you know, historically, you know, the capital required and the sort of compute required was, you know, out of reach of, you know, everyone. But every day that goes by, those requirements actually come down now, right? And so we see, you know, GPT2, which was, you know, sort of frontier in about 2019, you know, now you can train that, you know, equivalent thing for $100 or less than $100. Like it is, you know, insane. If we think about what's happened with, you know, GPUs, it's still very general purpose architectures, you know, CPUs, you know, GPUs. And so now we're seeing, you know, actually what happens when you develop chips that are actually designed to do, you know, large language model inference. And we're going to unlock another orders of magnitude in terms of, you know, performance and efficiency. So again, we're going to keep seeing this, you know, come down. So, you know, what required, you know, $100 million will require $10 million, which will require $1,000, $1,000, very, very quickly. And so being able to train, you know, these models to, you know, frontier level and above performance, you know, by small teams with relatively little compute, well curated data, you know, we're seeing that happen now, and that's just going to continue to happen. And so, and you've got to just kind of be in it to win it, right? There's, you know, there's, again, just never been a better time. And you, you know, miss 100% of the shots you don't take. What's driving that cost down? I mean, it's just, you know, even sentive, right? There's so much, you know, economic incentive to drive that cost down. There's such demand for it. And there's just a huge amount of people that are involved in this now, and open source as well as being just a massive, massive contributor to this. I think open source software and the open source movement has probably been one of the greatest things that we've done, you know, collectively, you know, in humanity, you know, it's hard to actually describe how impactful this is. Every single thing, you know, all of the recording equipment, everything here is running on open source software. If you open up your phone and you look at the licenses, I looked at AMI Pixels, literally like 10,000 licenses, all free software and again, this is, you know, it's democratizing. And we'll see it with AI, you know, open source models. It sort of, it allows, you know, it really is the most democratizing kind of technology in that everyone, you know, it really does level the playing field. Where are we going with that? I feel like there's so many opinions about the future of open source, close source, and America's getting in the game and China's almost like going close source now, like, I just, like, where is that trending? I'm not sure where it's trending. I mean, it's, you know, there's, I think, you know, it's hard to say exactly what the minimum motivations for some, you know, different come trees and whatnot are, but, you know, I think if we think about, you know, one aspect is commoditizing arrivals, potentially, you know, I think there's a lot of geopolitical stuff, but there's different opportunities as well in different parts of the stack. And so, you know, if you've sort of got the frontier intelligence, you know, if you democratize that, then, you know, you look at the economics of it and it starts to come down to, well, OK, how, you know, how efficiently can you serve, you know, these models, you know, say everyone's got access to the same intelligence. And so, you can start to think about, you know, OK, what's the end game and the end game really is, OK, how efficiently can you generate tokens at a particular intelligence? How efficiently can you do it? So how much, you know, how many jewels of energy do you need to do that? And over what sort of time scale? And so, you know, becomes now an effective way to say export really cheap electricity. There's different, you know, different incentives here in different ways to kind of, you know, the economics play out. So, you know, I think open source will, you know, always persist. There's a lot of incentive for it. And I think now, you know, we have a critical mass that, you know, even if big players, you know, retreat a little bit or whatnot, you know, I think people are incredibly determined. We're seeing other ways to train models, you know, people are developing methods where you can use huge amounts of disaggregated compute, you know, from around the world. And so, I think, you know, the cat is out of the bag and open source will persist. The rate at which open source models have closed the gap is so impressive. If you go back about a year ago, the leading open source models were lagging the frontier labs by about probably a month or so, but they were probably half as intelligent based on a, on a benchmark. You look at it today with like, particularly like Kimi models and a few of the others. The gap is strong from about a 30 day lag time to probably a week if that. And the intelligence gap has gone from like probably 50, 60, 70 percent discrepancy down to probably single digits. And so these open source models are releasing frontier lab closed models within a week of the other ones coming out. And so it's just getting so damn competitive between closed and open source models. So in my mind, I don't see any reason for trying to introduce another frontier lab frontier model building from the ground up again, because I think that game is well and truly underway. If you want to compete in that, it's not rocket science, it's heap of fucking money and some compute, probably $300 to $500 million and some compute. And so you could either take your investment if you were trying to think about how should I make my bet. And I'd take my bet and put it in one big company who's going to go into a really competitive frontier model market between open and closed sources, whoever heads start with all of the learnings and the feedback lips and the rest of it, or take that same investment and give it to 30 different companies who can do really interesting post training for all sorts of different things. Like, that's kind of the game we're going in because the open source tools now be a 4 billion parameter model to trillion parameter models are so damn good. And so like getting good data and post training for certain objectives or whatever you want to call it is like a far more interesting game in my point of view, because you've got, you can choose on like exactly where you want to go into that instead of going into a really, really competitive environment now. Three years ago, that wasn't the case. It wasn't competitive. There were two companies competing in this space. And you look, the open, the labs now is 10, really great labs be it closed or open source models, spread across the world, coupling Europe, three or four in China, four in the states. Can you actually describe what open source is, is that right, like just in terms of open source and then there's open weights. Open weights is release the numbers, and you let people train on those numbers or take those numbers and train them for a little bit. Open source is you give them full visibility on exactly the data that it's been trained on and the recipes that have gone to that, into that data set, you're probably better equipped to answer this. What are the main, the main models coming out open weights or open source? Open weights. There's not many open source models. There's maybe one open source model for some ahead. There's a few now and that's really, I think, so we're seeing, yeah, obviously building on top of the open weights. And one of the reason why I think the sort of, you know, the cost to compete is coming down a lot is because we are actually seeing these recipes. So there's a whole, there's a multitude of factors that go into these efficiency gains, you know, it's hardware, efficiencies, just better data, you know, people understanding what the right recipes are. And so this is all driving it down. And if we saw, there was a release earlier in a week from Jomi, you know, the frontier model, you know, it's essentially like, you know, maybe cloud five level, less than in terms of, you know, all of the benchmarks. And you know, their post training was, you know, $2.6 million on a matter of thousands of hours or tens of thousands of hours, like something that's, you know, much, much more accessible. Like we've never seen before. Once you actually know the recipe, then it becomes relatively easy to do a lot of this stuff. Of course, finding that recipe is being the hard part. But that's also why we've seen it, you know, initially it was so challenging. And we only had a very few, you know, number of people competing. But as this knowledge, you know, propagates and the methods get better, it becomes much easier to do this. Yeah. I mean, there was this big moment. I think it was two months ago when Open Router is sort of like, you know, got the seminal data on just like, you know, where demand for models is. And I think the Chinese open source models have overtaken the US closed source models. And I was like, a huge moment when I was like, oh my gosh, like, you know, this is kind of happening. The open source movement is taking over. That's probably skewed though. Is it because the type of person that goes open router is interested in lots of models. And most companies, they want US companies and they're going direct to the source, they're going to open AI and anthropic. And so Open Router is not collecting any direct to source tokens. It's literally people who are interested and naturally curious and want to try lots of different things. And so I think that data is totally skewed. But obviously that's an important subset and a certain, certain data point, but it's not like the be all in the end or can we talk a little bit about national security? And I think like two things, and I'll come to the second one in a moment. But the first is just like, I think when people hear the word China, they start going, ah, you know, and just like using Chinese models and always a bit nervous about China. So, you know, is there any, I mean, you guys are building on Chinese models yourselves? Like, is there anything about a security risk related here? Or is it just like, you know, it's like, I just bought something from China. It's like, you know, it's not a security risk, it's just a product, you know, like explain that to us. Yeah. I mean, it's, you know, that's the beauty of open source or open weights is that, you know, these models are released with, you know, very permissive licenses. So you're welcome to do what you want with them that don't require internet access once you've downloaded the weights. And so, you know, from a sort of, at least that sort of, you know, immediate kind of surface area from a security perspective, they're, you know, inherently very safe, right? Like you can, you know, use them in air gap environments. And that's in fact one of the greatest benefits is that you don't need to have them connected to the internet. They're not sending data to anywhere. It's the data, you know, you retain full control over that secondary, you know, things that people can be fearful of, which is, you know, sort of how concepts or ideas represented by the language model itself, you can export cultural ideas, you can export biases through the, you know, the way that the language models will answer questions. But again, that's more of a secondary, you know, downstream effect. You know, people do worry about, you know, there's, you know, certainly, you know, rumors about things, you know, if the model, you know, the text, if you're a person, you know, you're not going to be able to do that. particular type of person, it will give you different answers, you know, based on kind of that. But again, the beauty is you could just post train and sort of do what you want with it as well. So I think risk is pretty low. Yeah. I think the other kind of, the flip side of that risk is with the US, where from a national security perspective, we had like Fable and Mythos, like, you know, GPT-6, like, you know, access was like kind of turned off for Australia. And it was just like, oh my gosh, like that's a different type of risk, whereas it's like, not about like, is there any biases in this? It's like, actually, do we, can we get access cut off? I think that's a far greater risk, right? And you think about the intelligence, it's going to be such a fabric of, you know, our society. It's going to, you know, similar to electricity, right? Like you would, no country wants to be dependent on another country for their electricity or energy needs, right? You want to have, you know, capabilities that, you know, you don't become reliant because as soon as something happens, you can be cut off and your entire economy can grind to a halt. Same thing will be with, you know, artificial intelligence and intelligence in general. And so you want to be able to own as much of that as you can in the long term. Obviously, it's difficult to do necessarily in the short term. It will take time to build up the infrastructure. But having the sovereign capability is critical to, you know, from a national security perspective, absolutely. So I get kind of, to be honest, like, pretty excited about what you guys are doing. Like, I'm obviously an investor and I'm here to make money and I'm like, quite excited by the commercial prospects. But from a more nationalistic point of view, like I do look at the kind of five layers of the stack as Jensen Quang would put it. And you know, down the bottom, you got energy and like, you know what, we got lots of energy in Australia. And then you got like data centers and you know what, we've gotten a lot of data center activity. In fact, it's probably all of, you know, you pick up the paper. It's just like ever, all everyone talks about. And then there's chips and it's like, ah, well, it's a bit hard to enter the chip game, but then there's models and applications. We've got lots of application companies, but it's just like the model layer. It's just like, what are we doing here? And to your point about like, it is a matter of national security. Like, you know, we do have to have this as a capability. And my view is just like, look, I don't really care if it's about, we've got a model developer on speed or a model developer on creativity. It's just about, do we have people building models here and building an ecosystem? Because that's the start of an ecosystem that then flourishes and we just need that as an ecosystem. And so does that resonate with you? Do you agree? I'm not as concerned on security risk. Why not? Just because I think it is getting so competitive. There's options out there right now. So it only becomes a security risk if every single operator closes their doors to everyone or makes decisions. And so you've got like sure, maybe what is 10 or 12 providers right now, but every single one of those 10 would have to close their doors to you. But don't you think, okay, like, we literally saw, you know, with the US companies that they flicked off with the switch. It was just like Trump woke up, said, shut off access to the rest of the world. And they flicked off the switch. Like, that was that's the US going. And in China, all of them, let's be clear, are kind of like, there's a flick off the switch as well, which is one person there. Yeah, you can flick that switch off. I don't know. Like, yeah, I think one of the issues, though, as well, the PIPs point, the providers aren't independent. So if you assume independence, then yes, you can sort of maybe one gets taken out, the other ones will exist. But they're all running on, you know, the same data centers, the same cloud infrastructure. And so you've got this correlated failure mode. So, you know, if, you know, one of the cloud giants decides to turn it off, it'll take out a number of, you know, providers. It's not going to be just like, they're not all independent. It's all correlated. The bigger risk is Taiwan. And Trump's getting all produced in a very small country, offshore from another country that wants to take over them. Like, that's a far greater risk. And having access to, it's a risk we're probably not going to get into. Back to the script. Now, I want to talk about talent. You know, I mentioned podcasts with the base 10 guys last week. What I found really interesting on that podcast was you have Bay Area kind of research company with base 10. They're over there in San Francisco, but rather than recruiting locally. And in their words, it's just like, why would we recruit locally? Because it's like such a competitive market trying to compete with anthropic and open A on salaries and just the craziness. They're actually recruiting in Australia. Like, they're coming out here, going to the university's university Sydney, they're going to the quaint shops, the, you know, consulting firms, like finding like these like statistics kind of gurus and just recruiting them, giving them E3s and getting them over there. I think they said they had 25 Aussie researchers in their research team over there, which just blew my mind. So their point was talent here is really good. Not a ton of commercial projects, like, but you just kind of like point and shoot them at the right frontier projects. And within a couple of months, they're just like there because they've got the skill. It's just like just getting up to speed on what, what, what the deal is. I mean, like you just said, you trained a model yourself and you don't even have a technical background. So all of that to say, what is the talent situation here? Like, do we have good talent here? Is it world class? Where do we sit? How are you guys finding the talent? I think our talent is exceptionally good. And it's probably not just within this field. I think it's in lots of fields. And I think our education system has been an export for a long time. We've got lots of people come into Australia to study for that exact reason. We also have the benefit of a research grant, which is fucking amazing, R&D, which is a major competitive advantage. So I think we've got all of the ingredients to have a really good talent pool. We also have just interesting people from interesting backgrounds, like one of the developers, one of the ML guys in our team started with us when he was 16. He trained his first model in his 11-year-old kid. And he's now like traveling in the world on the tops of mountain training models in his spare time. And he's like 18 now. He's been working with us for three years. And he's just like a naturally curious, mega smart kid, but he's come from an interesting background. And so I just think it's a great place to like, we've always had a good education system. So why wouldn't we have the talent here? The harder thing is competing on salaries. You just can't compete with that. He's finding like a lot of researchers, like just leaving for that. It's both salaries and opportunities and interesting projects. Yeah. It's the other half. I don't think they all chase money. A lot of the researchers that we have don't really give a shit about the money. They just want to work on interesting projects. Yeah, and you think about it. Like natural, like people who are curious and want to study, want to go, oh shit, this is a big problem that I want to wrap my head around. And there's plenty of those. So I don't know what you'll find in that. Yeah, I mean, I think I echo the same sentiment. I mean, again, Australia has got incredibly high standard living. We have very good education standards. And so our yield in terms of top talent relative to our population is quite high. But of course, we're still bounded by the number of people that are in Australia in terms of the talent pool. And so it makes sense that base 10 would find them because it's essentially almost like an underpriced asset. At the same time, I think we could be doing a lot. The government could be doing a lot more in terms of actually supporting R&D. I think to your point, the R&D tax incentive, that's great. But we're still well below. We see the averages in terms of R&D, expanding shared direct investment. So the R&D tax incentive, we need two or three more of those programs, probably every year in terms of investment. We had a lot of the sort of, I think one key thing to look at is the Bay Area. It started out with this immense concentration of these research institutions and national lab systems through the US. There's just a huge amount of the universities. And so it's always drawn that talent from that came out. The semiconductor industry and it's sort of had the guy in this virtuous cycle. And so one of the best things that we can always do is invest in that fundamental research capabilities and then work on how do you commercialize that and make it easier for businesses? It's a little more commercial projects here. Yeah, the thing about the US as well is you've got such a large population. You've got the government as well who spends a lot on things like defense and energy. And so they've immediately got a customer for a lot of technology. I'm nervous to get into the ballpark of copyright, but it is a hot topic. They talk about training models in Australia and there's big challenges with copyright here and that was in the press recently just with anthropic kind of like having a big kind of talk about that with government. And then also issues of IP taxation of kind of models when they're taken off shore by foreign model trainers. But I'd love to like just understand just maybe even beyond copyright and IP like what are the kind of biggest challenges that you see just in terms of like your objectives of like I'm training models, you know, got talent on your computer. I need copyright. I need you know legal kind of frameworks like what what are the challenges here that and hurdles and how do you kind of frame those? Copyrights a big one depending on what you're doing. Like for us it's a big one. Like we're locked out of training models on a whole bunch of different data because we have really strong copyright laws that you can't train anything in the public domain until that person's dead like seven years after they're dead, which is a very long time. You can't train here or can you train it offshore? You can train any like you go to Japan you can train one whatever you like. Could you train one in Japan? Or we picked up the business and moved to the business there. We could. It can't even be not. Yeah, you're not allowed to just like remotely orchestrate and put one person over there and make the decisions for me like physically need to be there. Yeah. If you could just switch on a different server that'd be easy to get around it. But yeah, you're not really allowed to do that. That is the major thing for stopping building a frontier model in this country. Is it you would not be able to compete with any of the other markets because they're trained on the whole entire internet and our copyright laws would just completely squash that capability day one. So that's a major one if you want to get into the frontier game. The other one is I just think money and support. You don't have the ecosystem either I think at a government level or even at the VC layer to have the gigantic levels of investment that are going into the same market over the states like it is ridiculous to see some like the seed round and money that's done around in those markets. And that's like super hard to compete with. How expensive is it to train a model? Depends on how long you're Yeah, the type of model. Okay, you did a full build parameter model. or you then go to 30, then you go to 100, like how expensive are we talking 100 million? - Yeah, I mean, it kind of depends, but you can traditionally, it's been sort of trillions of tokens in terms of what you do for pre-training and then you have to post-training from scratch. - You're training from scratch, but again, there's been a lot of advancements in optimizers and actual training methodology. And so previously you might have required 10 trillions of data now you require one trillion, just been seeing some more recent work where it was down to several hundred billion. I'm not entirely sure if that's, you know, but less to believe that, but we're set as seeing this trend where you require significantly less data to train things from scratch. And then, you know, like I said, Jamie showed sort of, believe it or, you know, no reason not to, that you can do it, post-training for several million dollars. So certainly, well and truly from scratch, you know, under 20 million dollars is definitely the ballpark of these days. - What we've found on our side is the expensive thing hasn't been the compute or licensing data or anything like that. It has been the experimentation of hiring people to go and play and make mistakes. That's way more expensive in our experience for like the size of models that we've been training. - And it's expensive like their salaries or the expensive like them just using compute to kind of play around. - The compute is a fraction of it. Like we had one go blue five grand in a day, but like you can do that really quickly with compute and token cost if you're using the wrong models and the wrong compute. Yeah, experimentation I think is by far the biggest thing. And so like, yeah, that example there, I don't know how many researchers they had to work out, how do you train a model of that size at those types of levels? They're probably trying a heap of things, yeah. And so that for me has been is the biggest block. - And that's why I think, yeah, that's the pair of open sources that reduces that cycle, there's a number of iterations that you need to do. And so if someone gives you the recipe, then you can implement it with relatively little risk or experimentation. But someone had to find that recipe and probably spend 100 times more compute to get to that point. But again, this sort of knowledge sharing open source, you know, democratizes and you know, once you actually know, you know, how to do it, it can be relatively inexpensive. And that's, you know, every day that goes by this knowledge, you know, sort of keeps getting shared. And yeah. - You can train a good model on a local machine on a local MacBook Pro, the two round for a MacBook on the compute it's got, something like it, something like the world's best model. It'd be small, but I can. - Pips crazy model. - Yeah, there you go. (laughing) - So the data copyright is a challenge. Where we obviously kind of respect copyright, but you know, it is a challenge if you want to compete. And then capital, you know, these things can be expensive if you want to kind of scale. What about like compute, other infrastructure? - Yeah, I think that I think compute is a massive bottleneck at the moment. And you know, it will get worse. - Is that globally or just unique to Australia? - Ah, globally, you know, which means that we'll probably stuff even more in Australia. You know, unless we've got, you know, ability to sort of jump a cure or anything like that. So yeah, certainly compute. And then, you know, data again, you know, I think we're moving to this, you know, world where we don't need this, you know, internet scale data like this raw crude kind of oil in, you know, those sort of terms. And it's much more about, you know, getting really, you know, much longer horizon sort of task specific data, you know, reinforcement learning on these sort of agents. And that data itself is, you know, I think that's going to be much less of an issue for copyright because it's so much more specific to these kind of tasks. And so, you know, the copyright question may even just not be that relevant, you know, in the future because that kind of data is going to become much less, you know, significant to the actual development. And the other thing that we haven't yet seen, which I expect we'll see is that, you know, this, yeah, everyone's sort of doing it from scratch. And if you even look on, you know, hugging face and you see all these models, there's, there's just so much compute that's gone into training these models. And very little of it is reused. I mean, people will post training on top of a model. But, you know, I think there's going to be a whole host of techniques that are able to reuse this pre-existing kind of, you know, trained models and, you know, bootstrap other models. And so almost like, you know, I mean, evolution has done it with things like genes, right? Like there will be a way for us to inherit a whole bunch of information without having to do it from scratch and it will sort of just be kind of passed along. And so there's a mental opportunity in being able to reuse a lot of this as well. When you say, "Computer's becoming a problem." Like, how does that manifest for you? Is it like just cost going up or it's just like, actually harder to get? - You can't even, you can't even, you know, even if you want to use a GPU, you can't actually find one that's available. - Yeah, right. - You can pre-book them if you've got enough money. So money solves all problems again. - Yeah, and that's the thing. But the, it's quite patchy, right? And you're doing experiments. Okay, you're going to do, you know, maybe I need a week and I do a training. But to get, you know, reliability, you know, you've got to enter into 12 months for three year, you know, commitments, you know, that might be $10 per GPU per hour. That really does exclude, you know, startups. - Yeah. If you've got the money, then you can just pre-book them and you're good to go. But then you've got to make sure you're using them or using them efficiently. But when you're trying to do it efficiently, then you don't want to have all that money, like, dedicated. So you're like, just trying to fire them up as you need. You don't want them running overnight and rest of it. So when you're living that open market world, that's where the GPU, like, becomes a problem. Because everyone's trying to do that 'cause most people try to do it efficiently. - Yeah. You know, it makes a total sense. There was some serious credentialed people in the last week who kind of came out talking about 10% chance of humanity getting wiped out by AI. I mean, do you have any perspectives on this or like, do you think 10% is underbaking it and it's higher or oh my God. Like, what do you mean put, like, be serious? Like, what's the dangers of AI here, like? - Well, you know, I think they're probably overstating it. You know, I would certainly question their agendas. You know, we've seen, you know, yeah, prominent leaders who will say there's 10% chance of AI wiping out humanity in the next day they're at Dreamforce, you know, doing B2B SaaS. And so, you got to wonder, is it really that, you know, pressing of an issue or not? You know, I think that there's a lot of, you know, a lot of people don't want open source and don't want this technology to be democratized. And so I think we have to really think hard about, you know, what are the agendas here? I personally think we need to find a way that, you know, again, that everyone can, you know, I really don't want to see the solid way, you know, there's two sort of futures here where one is that power and, you know, the technology just aggregates and there's one where it is, you know, completely, you know, disaggregated. And I think that that's the way that we all benefit and we should be really working hard for that. - Yeah, yeah, I think what you're saying there is like, there's a lot of incentive with the big close source guys to not have the Chinese open source models out there, you know, just kind of like cannibalizing there. So there's a bit of a peon engine about like actually, like this is really dangerous and we've got to make sure that like we have control of it. - They've been seeing this for about four or five years now. Every time they've got a big jump, like oh, this model's too dangerous to release in the world and then they release it two weeks later. - Yeah. - And so yeah, I think it makes their models seem way better than what they actually are. And the humans are very good at self-preserving. - Yeah. - Over many different disasters. And I don't think we're gonna let this thing get out of control that it's gonna destroy us. So it's just got a fear mongering for dots and different reasons for different gains. - Yeah, I mean, there's certainly things to be, you know, we're often there's certainly, you know, I think, you know, and certainly the, you know, the cyber cape birdies, that's definitely something that is real risk now. But 10% chance of destroying humanity in five years, I think it's probably overblown. - All right, final question for you both. I'm gonna start with you, Pip. What's something that you've changed your mind about recently? - Everything all the time. I actually changed my mind during this podcast when you're talking about speed. I'm gonna do a big ups of speed. But like I've been battling with this thing in my mind of like the role of a genetic use case is in our platform. And I think we need to get there. But I also think you can just make, like the best agent in my mind is a person. And if you can deliver really good intelligence, really fast to someone at the same time, then why would you hand over the decision making process to a model which is not good at making decisions? And so I think there's this whole thing of in my head going right now of like, well, fuck. Like why don't you just make people agents by giving them lots of interesting information really fast to make those decisions better? But I don't know, I changed my mind all the time. So it will change in another fucking hour. That's the look outside and decide what to do to get. - Have you added an aha on creativity perhaps? - Yeah, I mean, no, no, it's like, I think one of the great things about the technologies is sort of probably not answering a question. But you know, very simple mental model is, you know, there's sort of people who are good at doing things in sort of dexterity and sort of implementations, people with ideas and creativity. And what we sort of see, I think, you know, with AI is that, you know, really democratizes this sort of doing stuff. And so I've always had a lot of ideas, but I've never necessarily been able to do them. But now I can get a lot of these out with AI. And so there's certainly that that's been, have you got a good example of that like? - Oh, I mean, just like, you know, like, make launch videos, PR videos, a lot of the creative stuff, a lot of design, able to like sort of describe the idea. I was never able to actually implement myself graph, you know, but now I can actually, you know, get an agent to do that because it's much more capable. What I've still got the idea and so that, you know, that has been quite amazing. - Tips of change in my mind. I think I've really liked tune amounts now. So it was a bit skeptical of tune amounts. - Yeah, yeah, tune amounts are now. - I've never changed my mind. They've always been awesome. (laughing) Well, Pip, Jesse, thanks so much for your time. If people wanted to go try Flint, that's at springboards.ai. I can kind of go test it out. Or if speed is your jam, you can go to Salaris. That's C-E-L-E-R-I-S. - And the point of this episode wasn't just to check out these two cool companies and these Aussie founders, even though you should. It's also that there's a lot of momentum building in Australia around the model, development layer, and we're really excited by that. So thanks so much. - Easy, thank you. - Thank you. - We should have, get around us. - Thank you all so much for joining us for another episode of Wild Hearts. If you want to learn more from other ambitious people building, designing, and creating the world that we all want to live in, then please hit the subscribe and follow button. It would mean the world to us, the founders, the operators, and the investors who join us on Wild Hearts. This podcast is a labor of love from Blackbird and our production team. The show is produced by Joel Connolly of Blackbird, our marketing genius is Laura Coford, and our editor is Andy Jones. Thank you all so much for listening. And we'll see you next time. I'll see you all next week.