Advantage Programming - Part II

Hey folks, just another reminder that if you've already binged all of the episodes in the public feed and you're looking for more, you should check out the sub-stack. There are currently 17 premium archive episodes available only on sub-stack. I'll put a link in the show notes so you can find them easily. And with that out of the way, hope you enjoy this episode. [music] Yeah, so I use it to pretty much write all my code now. I'd say I haven't really even opened the coding editor for probably months, at least like three months now. So I trust it to do all of like executing my ideas. And I trust it to like extend upon certain ideas that I have debug old code, explain to me what it's done, explain to me what people at work with are doing in their code as well. So yeah, use it all the time just like it's main thing on my computer. You're listening to Risk of Ruin. I'm John Reader. This is Advanced Programming, part two. [music] In case you missed the previous episode, I am trying to get my bearings as to where we're at with the various AI tools. And how should think about those tools affecting Advantage Play? As I've been making these shows, the thing I keep coming back to is Volatility, which is a very common topic on this podcast. We have had guests who were vol traders in the traditional financial market sense. But then pretty much all of the guests have made a choice to increase volatility in their lives and to subject themselves to chaos. They do this in the macro sense, sometimes leaving low variance jobs to pursue a life in gambling. And they do it in the macro sense. They wake up in the morning, they make bets, and those bets win or lose. So every day is a routine of oscillation. As it relates to AI, I don't think it's a particularly warm take to say that now is a time of extreme volatility. The world wrote software one way for decades, and then that all changed in a few months. The guest for this show is emblematic of this theme of volatility in the macro sense and in the macro sense and also part of the software shift. Kanzi is part of a sports betting syndicate, but he was a developer before he was a gambler. I started coding when I was in middle school. Went to this little summer camp at a University of Michigan and had a code there. My mom's a software engineer, so it was exposed to it at the young age. I didn't think it was anything crazy. I would just see it on our computer all the time, then in high school, started building like iOS apps. And that was like, once I'd say there's much formal education around that, I was just like googling everything, just copy pasting stuff from like over the flow until it worked basically. And then decided in a college that I'd study computer science. That was obviously really useful in terms of just having a much more formal understanding of things. But that the class has necessarily taught me how to code, but introduced me to all the ideas that were obviously useful in being a software engineer after. And then after college, I just worked at a medium sized tech startup. There's a software engineer for a little bit. I'd say that's probably where I got the most reps coding in different environments, with the team on the product that people actually used. But I guess I got bored pretty quick, so I left that and then self-taught data science. Which is still coding and that data science stuff is honestly much more relevant to kind of this board-spreading stuff, I do know. But yeah, I guess I came into data science more with a software background. So it was never like scary coding. There was more of the math and machine learning, that kind of stuff that I feel like I had to learn a bit more. Like I took classes in college for that, but actually applying it for the first time. I learned a lot through that. In case you are not familiar with the gambling Twitter glossary of terms, the types of sports bets that people make often get grouped into two broad buckets. They're top down, or sort of arbitrage-ish derivative type bets. Like if the price for this bet is -1/10 over here and it's +1/10 over there, then maybe the +1/10 is good. And you could imagine all sorts of variants of that same idea using alternate lines, etc. But basically just taking the information the market gives you and betting based on that. Probably it is fair to say that a huge amount of sharp sports betting volume falls in this bucket now. The market makers on prediction markets are often getting their fair values from other places. And then there is another type of bet which seeds a market with an opinion about how games will resolve and the basic relationship that exists between what's notable before the game starts and how the future will play out. You know, this player is out and there were three points to the game margin. We would call that type of bet bottom up. So of course, individual bettors could at any time choose to be bottom up or top down, depending on what they're doing. But I thought it would be worth laying that groundwork because Kansi came to betting from a place of trying to predict games, not trying to exploit off-market lines. So I didn't even come to sports betting through like thinking about advantage play or anything like that. I worked for a best-fultane in college as an intern. So that's where I was exposed to like, I guess like, money-ball ideas and just finding value in markets where in that case the market was just like NBA for agency or something. But I once like, post in my ratings on, for like players on Twitter and then I guess people on gambling Twitter are always just refreshing Twitter. So some guy saw it and dammed me and kind of told me about gambling. I guess I knew about gambling but I had never really thought that one could make money in it or it would be a job. I listened to bett the process even in college around 2017. So I knew of its existence but I never really thought that I would be involved. So I had a nice normal job and stuff. But then as I talked to someone personally that actually was gambling for a living, like it captivated me a lot. And then I realized, oh, you can just build a model for everything, right? And they're like, naively, that's why I thought, like, oh, let's build a model to like predict NBA games. So that's kind of how I got into it is more, more from like sports analytics. The top-down method that I described earlier is just sports betting's flavor of efficient market hypothesis. Sort of the 800 pound gorilla that everyone has to contend with if you're going to make money as a speculator. Both honestly, like the first time I tried to like bet any model opinions like it won. So I was obviously way overconfident about it. And it was for basically no money. But it's impossible, I think, to just like, $10,000 for your first bet, right? That's just not, I don't know, it was like $25 or something, $50, whatever. But it won. So I thought, oh, this is just easy even though I know that in the back test, obviously the edges are really small. But then as I got more comparisons to the market, like I realized that, oh, I'm just like off always on this team or something, right? And then that's where I got the feedback and then maybe so losing and then even just on the back test, it's just like, oh, the market, our MSC is always better than my models. Like once I realized that, it was kind of like, okay, let's think more about this like EMH and then like, I think that really like helped me level up like how to think about winning at sports is like, okay, the market's actually really good. You're not going to just really win like 10% ROI or something. I asked Cansey if he could remember any specific breakthrough where he went from not quite understanding to being able to win. And he said basically no, his ability to win is just the accumulation of lots of progress. It's literally all been like incremental. I think one great example that would just be say you're building some model where you or any model you need to wait like a recency, right? So especially pre AI and like writing code kind of took a long time. Like I knew that you had to wait recency. So I would just pick a random like 0.95 as like the exponential like decayed, you know, constant right? And then use that for a while. And at some point if I have time to revisit it, it'd be like, okay, well, we can actually tune this parameter, right? And then look at we tune it maybe like 0.97 is better. It's kind of like that. It's like super boring, but you just just have worked on that kind of stuff like over time and everything gets better. Obviously tuning parameters, not the only thing, but like that's just one small example. It's super boring to talk about, but it's impactful and you just do that every day. Cansey is a good guest for this episode, not just because he's using AI every day. He says that the emergence of the AI tools actually pushed him toward gambling. I'm pretty online, so I'd seen people tweet about it, like, especially since I'd been on tech Twitter for a while, it's a long time. Like follow people there. I guess I'd seen even like chat, but was before 3.5. But anyway, they were just like, it just seemed like a fun like silly text generation tool. Like you could get it to write a little story or something and like I mess around with that. I didn't know what a transformer was. I didn't really understand like how it worked under the hood. Besides just the very naive, like, oh, tries to predict the next deal again. So I didn't really like have much imagination that it could replace like coding or just like knowledge work. Because honestly, a lot of that is like the harness that's built around it now. I think that power is a lot of that, right? Like it's not literally, even chat LGBT is not literally just a model, right? [MUSIC] was like the system prompt and there's like a few things going on behind it that make it useful as like a check opinion, first of all, right? It's not just, you know, you give someone else the model, they could build the worst version of the LGBTQ. And then the progression for me in 2024, I was like using cursor, which basically, before using cursor, I would start like, with like little scripts or something, I would just describe the script and it could generate like pseudocode or like code that somewhat worked maybe, but it would still like hallucinate variable names and whatever. Like it was still, it was generating a lot of code that was like somewhat useful. And then I think at that point, a lot of people are like, oh, I spend just as much time fixing the code as I do like actually using it. But like, by the time I saw it generate code, that was like somewhat useful. And I'd already been like, okay, this can definitely improve, right? Even just, even if the models didn't get any better, you could totally automate the step of like running the code, it's being that it broke, pasting it back in, and then like watch it fix it again. Like I was doing that manually. I knew at the time like, there's no reason that part has to be done manually. It's just, it's just how the product JTBT works right now, but you can even feel any intelligence, you could already get a lot more intelligence out of it, right? And at that time, I was like, still working, I was working at a started different company where I was writing a lot of code. Just a lot of like front end product stuff, like building web apps. And once I started using cursor that like it integrated, so I didn't have to copy paste back and forth with the JTBT, like it, I was integrating the code base. And I could see like the auto complete. Like those were pretty impressive auto completes. I thought it would just like generate a whole function pretty quickly. Yeah, like once I saw that kind of, that's like, okay, well software engineering is definitely going to change. I can already code by the fact that I've been doing a lot of like manually selecting what context to look at was also like silly, because like, can't the AI just decide what context is useful, right? So there's like seemingly a small like manual steps that humans were doing to operate the AI that like could have been run in the loop, right? And I guess like obviously everyone realized that and people started talking about agentic frameworks or whatever, right? That's kind of what we went next, but I was like pretty stunned by all that at the time, like when I was working as a software engineer, I actually just like left that job 'cause I was just like software engineering's cooked, like my skills are no longer valuable. And people were kind of confused because they're like, yeah, what do you mean? Like obviously you're still like writing code contributing. I was just like, it makes, so like, it makes very little sense for me to like continue trying to improve like what I was good at at the time. And I would say like throughout my career until then, I was just, I was like useful as a generalist, like I knew how to build iOS apps, so I had like a decent understanding that I could build like, you know, back in front end, like could basically take any product, consumer product thing and go from like zero to one, like felt pretty confident about how that stuff worked, but like lacked deep understanding in, you know, how iOS is working under the hood or like infrastructure stuff for web apps or like databases. But then it kind of became obvious. Like I guess my value was like generating lots of code. But then when I saw this machine, like be able to generate code across basically any software engineering domain I'd pointed at, it was like kind of alarming to me that like, okay, well, the thing that I'm valued for right now is just not going to be valuable very soon. And then also I did want to just like the non-sports as a job, because like I was working on the sound of well that and had confidence that we'd be able to win, like building models and just felt like speculation was kind of never going away. Like it was kind of sad to move into a zero sum game in some ways, but like at the same time, feel like the skill set to like win zero sum games was like still going to be valuable, right? That's not something that's like generate more code equals you're better at zero sum games, right? Like it wasn't quite that simple. -Kanzi's group also includes the former points bet trader known as Spraws Better, who does the heavy lifting around betting logistics. -Yeah, like everyone's kind of coding. And then I guess there's more of like the like getting large bets down part guy work with Spraws. He was on that process too, but yeah, basically getting down is like a huge part of it as well. And I think that takes dedicated time. It's good to have someone like think spend most of his time thinking about that. And then the other guy that we hired last year, he came from coding background too, but yes, more like data science, more like Matthew then me. So I think understands like model lining that set of tools much better than I think intuitively can reach for the right tools and that really like magnifies with AI like I think he can just he has the better ideas to put into the AI and it just executes it right. Like that's been really helpful. So we both code but just kind of divided up based on like what we're thinking about. And honestly, like the way that we're working together has like evolved a lot already just since we started in like November with him. Because then it was just like code editor was open at least half the time and then AI was like bidding the code and I would be like watching it edit the code and I would still like manually run a bunch of stuff. Now it's like we spend time writing the prompt and just kind of let it go and like return the results back to us in the chat so that we know things are working. Like while we're waiting for the AI to like complete a task it's just like really do we can't just sit there. So it's like kind of made us work on like multiple things in parallel. So I guess it's allowed us to like get a lot of tasks to like 80% completion really quickly and then maybe like the last 20% it's like okay, we actually have to lock you and not just this one thread like let's let's really watch what it's doing and think about it's right. It was really understand what code was written. The AI tools are really incredible and somewhat annoying and it's also kind of hard to imagine working without them now. Among the various trade-offs that comes with using them is that you didn't go through the process of learning while you built. So to counteract that maybe you create some guard rails to understand what's in the code and maybe you have to do some investigations after you know that something has gone wrong. - We had some expense mistakes but a lot of it is just breaking things into like smaller tasks I guess is like each prompt for me still represents like a pretty small task. I just I guess just goes back to what I was saying. Like I kind of think of the AI is like a coworker like I would never really trust coworker to just like I wouldn't say go build a winning college football model, make no mistakes. Right. It would be like I was thinking about what I need to build a college football model and then be like okay scrape all the data from and see the way dot on. Think about what schema would be good. Here's how I might use the data, et cetera. And so like the task would be small enough where the whole thing is still driven by like my human judgment on like how this system should be built, right? And like the AI is just kind of executing like small parts at a time. And I know a lot of people talk about like getting the AI's like talk to each other or have some like agent swarm or whatever like maybe some skill us skill issue on my part, but like I haven't really gotten that type of stuff to work. I still kind of I'm not like supervising the AI like writing the code, but like I'm supervising like pretty frequently that each small thing is done in a way that like I think makes sense, right? And yeah, like the AI does lie sometimes and like we'll add like oh decides when I say like oh performance matters like it needs to be fast. I've learned like maybe need to be a bit more specifics. And those started just adding like cashes and stuff to random places. We had this like terrible issue like four months ago where after I prompted to do that, like it was just caching some coefficients or something that like shouldn't have been cached, right? And I've kind of gone back and forth. I'm like how to prevent that kind of problem in the future. And I think like that's just where humans intervening, writing some tests or humans telling the AI like hear things they need to test for like that stuff all matters like maybe not try to build things as fast as possible and actually go back and think about what was built. You can even use an AI to go back and like check things over, right? I'm still doing that step like mostly annually. I know there are people that are like first pass AI generates all the code and then like another AI will come and like adversarially like try to find bugs and then another one is incentivized to try to like correct that agents like false positive bug breaking all that. I haven't quite gone into that. But yeah, and then I guess like the AI's tried to write a bunch of tests and I've never really found the test like useful in terms of preventing these kinds of like money losing bugs. I feel like the tests are more for the AI to like guide itself in writing code that actually runs, right? The test always passed by the time it delivers the code to you. So in some cases, in some ways they're like trivial, right? The logical implication of the fact that somebody gamblers are able to code kind of almost whatever they want to much faster than before is that markets should be getting more efficient, right? There are people like Kansi who have excess time while they're waiting for code X to return something. And so they can turn their attention to Russian table tennis or you know, that's maybe an extreme example, but the idea holds the barrier to exploring some new market goes down. And then the top down better is picking off slow moving lines or ARBS. They should be able to move faster with custom setups. Of course, as I say that, my Twitter feed is currently full of people bragging about how inefficient the world's cup markets are. Anyway, nevertheless, I think long-term you would expect markets to get more efficient. So I ask Cansey if he had actually been able to perceive markets, getting more efficient. I would say it's like a feeling, yeah, having really like had definitive proofs that exists. For all the logical reasons you mentioned around more competitors that were not intervening at all who now can get into it or like existing people can just execute faster like we are. For those reasons, like, yeah, I imagine markets are getting more efficient quicker. But at the same time, it's like there's a ton more retail flow as well. Like just because someone can use chat GPT doesn't mean they're going to be able to build like a good model, right? Like if the market is getting better and they have something way worse, like they're no different than like a losing better, right? In some ways, if they're not acting early or if they're betting into some place where like price won't move or something, right? Like it's just kind of hard to say like across the board, what's happened? For example, we were betting NBA props. Like I feel like that has just gotten more efficient. Maybe there was some, this is like all anecdotal. Like I haven't really gone back. Like there hasn't been like a great way to track it. I guess I could look at like some aerometric over time, but like especially in spots where like a high usage player like goes out, it seems like the market just didn't really know how to redistribute that player stats in the past. But now like it seems to get it more right more often, stuff like that. That isn't necessarily even like an AI thing, right? Because it's just like an astute trader or something or you know, who we just not mess it up the next time. Kalshi offers parles or combos via their RFQ API. So this is something that is pretty new. A year ago, P2P parles weren't a thing. So the result of this is that today, you have traditional market-making firms like SIG, you know, firms where sports are a tiny part of their business, competing with groups like Kansi's group, who are focused on sports, and they're all offering parles and this all bypasses draft kings and fandule, which is pretty striking to think about. That would have been like an expensive task in the past. And our first like more automated interactions with prediction markets, like we're literally done as like one of the background tasks I was talking about earlier, where it's like, oh, we're thinking about like improving our player ratings for NBA, but like I'm watching the AI like code. Like I just opened up a new chat and said, hey, look at like the RFQ API. You know, like how do we build our future quota? Here's like all the stuff we have at our disposal. And so like our first interactions with that literally came in like background time, right? For now, the player prop menus on Kalshi are pretty limited, but they do seem to be growing. Yeah, the menu is really sparse. And like, I don't know, I follow the prediction market stuff like a little bit. Not a lot, just like what people are saying. But my feeling is like they're kind of trying to avoid just building a product that has parity with standalone, for example, like they're trying to try to, they're trying to chart their own path. I'm like what a modeling product should look like for better or worse. Like there's a very bad UX spots, but at the same time, like maybe some things are better than like a fandal app. So the limited menu there is just, it is a bit baffling because a lot of people could market make that. Maybe not that, I don't know. Maybe it is not that easy to market make, but there's not, it's just not featured in the app. In the same way that is unfindable. The base question for this episode series, if you require fidelity in your coding and you're not just building trivial data dashboards, do the AI tools help? That was answered pretty quickly. So I think the next question is really related to let's call it configuration edge. Okay, if the AI tools offer an edge for those that use them, because they can just go so much faster, then it stands to reason that people that push the envelope could also extend that edge. Just so this idea is concrete and not nebulous waving of arms. Here's an example. In the open source Hermes agent, they have a tool now called mixture of agents, where you can send the prompt to multiple LLMs. So think like open AI, an Anthropic, and Grock, whatever. And then get back an aggregated synthesized answer. And you could see some obvious use cases for this like, could three top tier models beat one top tier model or could three cheap models beat one expensive model. It just has all sorts of implications for the future. But when I talked to Kansy about this, he said he doesn't really think it's worth his time to chase some custom configuration. If there's some hot new thing that also works, it won't take long for the good stuff to show up in the codex app. Yeah, I guess like we never really thought about like models driving like ROI. Like it's really like our ideas that drive like winning basically. And the models are just like a tool to execute those ideas. I feel like honestly switching has more been like I just like talked to other people who are also using AI for various tasks like, you know, coding or I mean mostly coding, but like you know, building web apps or also doing like more data sciencey things or some people kind of get crazy with like the harness or like open claw was like kind of hot for a minute right like have friends of like all over the spectrum on like AI usage and just into it like kind of what the consensus is on like what tool should migrate to. And a lot of mituses just literally been driven by like how good the UX is. Like just really liking the codex like desktop app right now. Like I mentioned, I don't open the code editor. It's because like the codex app is really good. I figured out how to like even get the browser to like run in there. So like, you know, if I'm making like changes the web app and all happens in codex is just like one app for everything. And it felt magical and I switched to that from cloud code, just cloud code, the desktop app like at the time kind of sucked. And I just felt like things were better because the developer experience and not necessarily like I can really tell the difference between like codex model versus cloud code model. Because yeah, it was probably like executing my ideas like similarly well or like I wouldn't stop prompting it to execute until like I felt like it executed the idea well. Actually, I should say that candy said that yes, he expects that the processes that become useful will just show up in the codex app. But he also said that he's not chasing the custom configurations because he doesn't want to get involved in the never ending follow. I think partially I do it for just like mental sanity. One of the other groups who were talking about earlier is just like the AI promoters. And there's like so much of that content on Twitter, Rooms rolling all the time. So like I do feel the FOMO of like, oh, it does seem like people are doing all this stuff like while they're sleeping and like agents are doing all this crazy stuff. And you know, they're competing against each other to generate the best answers. So like I was feeling that FOMO but then I read something about like all the small noises you have in like using like to say the codex app. Like the team working on the codex app at OpenAI is probably like experiencing those same things as they're building codex, right? So like I just kind of have ended up trusting that these like frontier lab companies will also build a good product for coding. And like yeah, I might be a little bit behind in figuring out how to get like all this like parallel agent and sort of like these long running agents that like, you know, can go towards a goal or something. But like if that is really so important for software engineering, like the engineers at OpenAI or Anthropic are going to build that into the next version of codex or cloud code. Right. There's going to be some button where you can enable multiple agents to review each other's code or you know all that kind of stuff is if it's obviously good, it will just be there in like three months, right? And I guess like that three months were on behind. I just like have accepted that as a cost of just not freaking out every day about OpenAI tool, right? So I get to go to work every day thinking about like the data science problems or like creating problems rather than like what is my setup problem? And I guess like I just don't really love to tinker with that kind of stuff. All of the following things are true. One, these AIs can still be really annoying to deal with and they hallucinate and make mistakes. Two, they have limited imagination and tend to serve up pretty conventional thinking. And three, the frontier labs have still been able to turn the models on novel problems where the solution did not exist in the training data. And the AIs have solved some of these problems. I think this issue of can AI solve problems where they weren't fed the answer ahead of time is pretty relevant for advantage play. Basically how much can the machine do on its own and then how much can an experienced programmer do with the machine? Yeah, I feel like some of my efficacy with it is like because of actually coded by hand, I guess it's nice to feel that way. Like I just don't know what it would be like if I never actually coded by hand. Like in some ways it would make me more fearless. I think you were mentioning earlier about like they're the AP types that I just like to you know get really into would be the type person get really into like figuring out how to get these like insane agents setups and you could probably squeeze something out of that. That's like an edge right for for a bit. So yeah, if I hadn't coded before like the younger people I guess just are probably more fearless as they are with like any new technology. And like the right tale of them are going to actually discover I'm like some insanely effective uses of it, but like the average or like below average might just like literally generate swap, right? - One thing that Kansi said to me that I really am surprised I don't see more of is this thing about fatigue because I notice it too. My wife can tell when I've been battling with clot for a few hours because I look completely toasted. - And I guess like another interesting thing I've noticed is like, like I almost don't want a bunch of stuff working like while I'm sleeping 'cause right now I feel like a bottleneck is like keeping track of all the context in my head, especially since I'm like now able and kind of forced to work on like three things at once. Like I have no three different chats going, right? I'm like three different work trees and the codex out. One might be like it's all she's thinking the other like like, like, like, "I can't wm be like just completely different tasks." And like I just feel like I get tired working like way faster now 'cause like, my brain really has to be on to hold like all that context in my head of like here are the three chats that I'm like concurrently working on, right? And that's like free tiring, which feels really different obviously than like over a year ago when I was still like in the code editor. There was a point where I was like, "Oh, AI is like making my like coding endurance go up 'cause I would just be able like type the prompt in and kind of just like wait as it completed a little thing and then I would like look at it, run it, we're kind of the next thing." And like during that period of time coding felt very, like it felt way less cognitively demanding for a little bit and I was like probably producing a little more work than I was like without AI, right? And I was much more easily able to work for like four hours in an hour or something. Yeah, and now that's kind of gone back the other way where it's like, okay, four hours of like intense like working on three things that wants is like really tough. - Kansi says that this new feeling of mental fatigue is slightly different from what he was used to dealing with pre-AI. - Yeah, before AI, like definitely just building that like house of cards in my head before even stepping into the problem was like is really cognitively demanding and especially when I was like just working at our time for a couple of years, like if I had two hours to work and I'd take a lot of energy to like even just getting the right head space to contribute like a small bit of where I thought might be an improvement or something like that. I think the whole thing we're talking about in these episodes is an acceleration. I will say that I am probably biased against believing in the acceleration in the first place. If I see FOMO merchants on Twitter talking about how they're replicating the work of 20 software engineers, my first reaction is skepticism, but that's why I wanted to talk to Kansi and the guest for the previous installment, Cartwright. They both said that they're getting much more work done than they were before. If things are accelerating, then it might be useful to think about what other stuff is out there that seemed far off, but might not be anymore. I asked Kansi if they plan to use computer vision and he said, probably not. - Yeah, we've thought a little bit about computer vision, but I think the best models are ones that are simple and win. It's better to win with public data in a unique way of interpreting it versus you get proprietary data and have a sloppy way of interpreting it and you just win based on the data, what it's like, that goes away one day or something, I think you're gonna go closer. - In the past, I have framed Advanced Play as being about violating conventional wisdom. This is particularly relevant for this discussion of AI because LLMs are the greatest conventional wisdom machines that have ever been invented. They are very good at producing thought that represents the center of the distribution. So an Advanced Player using the LLMs has to actually fight the tendency that the models have to rely on what they quote unquote know. And then the other thing is that for these edge-related domains, it is axiomatic that the most valuable ideas have never been published at all. - Just like there's a lot of bad ideas published online about how to model sports. Maybe they're not bad in and of itself. They're just not market beating or even market replicating ideas. For example, I don't even know what happened now but it gave a very basic prompt about how to model software. But if it says anything about like, Poson distributions, that goals are Poson distributed. That's just kind of the wrong way to think about the problem now. And you're not gonna get anything market beating from that. Right? But all the literature started from there 20 years ago or whatever. It's just like, oh, the two teams have goals that are Poson distributed. Oh, shoot, we need to correct for ties or some extreme game states or something. But you have to push the AI to kind of go beyond that to be like model it in some different way. And maybe if you have the experience of building one of those models and it didn't work, you might have some ideas like, oh, maybe you just directly predict the how the goals are something as an empirical distribution. Yeah, those are the human ideas that I still have faith in humans to be necessary in that sense. And then I don't really think about the AGI type of outcomes that are like, okay, AI just takes over all knowledge work or whatever. That's kind of, if we get there, we get there like, you know, I'm not involved in that at all. The running joke about AI is to give some ridiculous goal like Claude, build a machine learning model that can produce a three-sharp ratio, training US stocks, make no mistakes. Right, preposterous goal plus don't screw up is supposed to make the preposterous goal somehow achievable, except it is true that both prompting the AI to be careful the first time tends to work and also getting it to find its own mistakes tends to work. So the job of a coder now is to prompt the AI to not screw up and then to prompt the AI to fix any problems that might exist despite the first instruction. If you're gambling for a living, then one way you might be glued in to bugs in your code is through losing. Yeah, I mean results definitely like affect things, not even really in a logical way, but like, I don't know, losing all these thoughts and feels like, you know, if the model's a little better, you're like a little less likely to lose, so if there's like losing, then just ask like, codex to go back and view everything, honestly, as like kind of a background task for the day, or you know, we can, yeah, we try to actually like look at back tests and stuff and try to figure out if there's discrepancy between how the thing is working in like production versus the back test. Like obviously we try to back test everything and we're never really deploying anything where the back test would be losing right like. Yeah. So if there's any bad results, oftentimes it's discrepancy between how the back test worked and how it's running in broad. So can ask codex to like look over that stuff. The amazing and terrible thing about using the new AI tools is that anyone with access to 200 bucks a month can create an amount of code that in the past would have cost tens of thousands of dollars. But for now, those tools are pretty reliant on humans for coming up with the ideas and a single human with a few good ideas is probably equal to a lot of machine brute force search. Although if I'm being honest, when I say that, I can feel myself looking for ways to assign importance to the future role of humans. And if I kept more of an honest scorecard, it would probably land a lot closer to LLMs plus a goal, plus some loops, can probably replace a lot of human effort. I mean, I feel super needed because I still don't really trust the model to come with good ideas. But that's just my way of predicting the outcomes of these sporting events. I'm coming to it from a-- what's the thing about the sport? It sports data and predict it, right? And so I still think my idea is there valuable. But yeah, what if you just took all the trade logs on call street and constructed some model based on market activity, right? Is the human-- yeah, maybe there's less human judgment there. I don't know. So I still feel, for my workflow, I need to be there and I've never had good success with the AI coming up with ideas to modeling ideas to pursue from scratch. But then I had this interaction with some guy we worked with a little bit on some data stuff. The other day, and it was really bizarre where we were checking over some data as this crepancy between our two systems. And we were just sending back and forth codex outputs. I would check over the data. Codex would respond with some stuff. And I would just paste that into our chat. And he would take that message and just paste it into his codex. And then his codex would run. And then he would take those all of the things that passed it back. And I would paste it in mine. And then after a little bit, we're like, why are humans necessary in this specific interaction? We literally just did agent-to-agent communication. But I guess we were forced to read the messages. Then codex sent back. And at some point, we decided to terminate it. Because we figured out what we needed to figure out. So there's still some human judgment going on there. But it felt really bizarre where we weren't really thinking. We were just reading the outputs and making our agents try to solve the problem. And that sense, I was questioning why we needed to exist there. Risk of ruin is written and produced by me. Special thanks to Cansey for doing this interview. I'll put a link to his Twitter profile in the show notes so you can follow him online. To get in touch with the show, you can email me [email protected]. You can also follow me on Twitter @havekelly. (upbeat music) (upbeat music)