AI and Tax Research: A (third!) Chat with BlueJ Tax CEO Ben Alarie

This is Tax Chats. [MUSIC PLAYING] Hello, I'm Scott Dyering. And I am Jeff Hoops. And we're here to chat about taxes. [MUSIC PLAYING] Hello again and welcome to another edition of Tax Chats. I'm Scott Dyering, Professor of Accounting at Duke University. And I'm joined, as always, by my friend and colleague and the Tax Museum Curator at the University of North Carolina at Chapel Hill, Jeffrey L. Hoops. Hello, Jeff. Hello, Scott. How's it going over in Chapel Hill today? Pretty OK. Pretty OK. All right, I hear the Tax Museum is expanding some. Yep, you hear a lot of things, I guess. I think you're still hard at work as dean, doing all the important things you do there. Oh, it's very important, let me tell you. Fighting battles, winning wars. Well, one of the interesting things I'm doing over here is thinking about AI. And we wanted to talk about that today on Tax Chats. So what are we going to do? Yes, we are. We have a repeat guest, Bandion to introduce yourself. - Wait, wait, wait, wait, is this our first three-peat? (angry groan) I'm not sure. - Alright, well, one of the few, many repeats, yeah. So anyway, sorry for the interruption. - Ben, do you wanna introduce yourself a little bit in your mind as COR? - Sure, my name is Ben Allery, and I'm co-founder and CEO of BlueJay, which is a tax technology company here based in Toronto. I'm also the Osler Chair in Business Law at the University of Toronto Law School. And excited to be back here, guys, talking about tax with you two. - Yeah, and I will say when I emailed you at first, I thought that when we talked before, BlueJay as an AI platform accessible to everybody, I think was not super, super, super old. And I was kind of expecting when I emailed you this time for you not to be associated with any kind of law school, just like completely abandon that and only being a CEO. So is that, how do you manage both of those at the same time? - Well, you're not so far off the mark since I have gone unpaid leave from the University Effective January 1st. And the university is very generous with the leave policies to accommodate different things. This is an unusual kind of leave. I feel for the law school, but the dean has been super supportive and the Central University Administration has been super supportive, just recognizing that this is a certain moment in time for this kind of technology and allowing me to create some space in my life to dedicate myself entirely to BlueJay. - So maybe episode four, when I emailed, it'll be a different email address. - Well, putting on my dean hat, I can imagine that the dean would want you to come and speak to the students about the ways AI is affecting like the legal world, right? If it's the law school, isn't that like a huge topic of conversation for them? - And it has been, Scott, you're exactly right. And I've been teaching law and technology classes. I've done a seminar for the past, close to the past decade at the law school. I didn't offer it this year for the first time in several years, 'cause I was busy actually doing this stuff rather than teaching it and talking about it. And my research continues. So I published a book called The Legal Singularity, How AI Can Make Law Rattically Better, Back in 2023, the successor volume to that is gonna be called Super Justice and it's gonna be coming out with Oxford University Press early next year. So earlier today, I just spent a couple hours going through and finalizing the draft manuscript before submission to OUP. So my academic work does continue. I'm still very much interested in talking about the academic ideas and all that stuff. I just, I can't spend the hours necessary in the classroom to keep all this stuff moving as quickly as I'd like it to. - Okay, so as a reminder for listeners, so this is, I don't know, like a year and a half ago we had you on and this all started with, I think it was an ad or something somewhere where I saw you had this product, Ask BlueJate, which is basically a chat GPT-like tool where you can ask questions about tax law. And I reached out to you, said, "Yellow's chat." Before we chat, gave me access to the tool. I tried the tool and probably asked it, 10 questions, probably got, I don't know, seven of them wrong. And I emailed you if you remember and they said, "Look, maybe you don't want to chat 'cause until, if you don't want to make your tool look bad." And you said, "No, let's chat anyways." We scheduled it for like a month and a half 'cause that's the soonest time we could find. And maybe a couple of weeks after that email, he emailed and said, "Basically try again." And I tried again and it got quite a few more answers, correct, like within a few weeks it had improved, which to me is kind of like scary, you're talking about this legal singularity, like it's basically a bonus. And I've been kind of paying it, so we had that episode we recorded. Another episode after that, I've been using your product since then, every once in a while, and it seems to just get better and better and more and more accurate. And most recent interaction I had with it, there was a tax director from a large publicly traded company for having this discussion about something that he does pretty frequently. I know very little about, he said something that just didn't set well with me, and I just put it in ask Blue Jay and said, "No, and he was wrong." And I was right, and he very quickly confessed that he was wrong, 'cause he's like, "Oh, okay, I see this code section now, I guess, as wrong." I just like startled it. I don't know anything about it. This is what he literally does for living yet there's this AI tool that completely, I could easily use to get to the right answer. So that's where I stand. I've got these tools. So tell us a little bit about what is kind of like the updates, what's happened in the last while in the AI and taxes world. - It's super interesting. Maybe I can throw some additional kind of detail on what you're observing kind of anecdotally, there Jeff, with your experience. So at the start of 2024, one of the things we really pay attention to in the platform is how frequently users click the thumbs down button when they get an answer. That to us is a very good indication that there's some disappointment on the part of the user, the answer isn't correct. It didn't meet their expectations for whatever reason. My best guess is for every thumbs down response we get, probably 19 or 20 others that somebody didn't click that button, but they had like a not completely satisfactory answer. - I wouldn't ever click that button 'cause I don't ever want to get any AI tool mad at me 'cause I realize that one day they'll rule the world and when they become incarnated, I don't want them to come after me. So I don't ever click the thumbs down. - And Jeff always says, please, and thank you. - I always say, please, and thank you too. - Which I have learned actually consumes pretty significant I retort this so. - Open a comment on that a few months ago. The please isn't thank you. It's like $100 million. - So we started really paying super close attention to this. Probably around the same time you started getting into the product and using it. At the start of 2024, what we were finding is, it was about 0.8% of the time. People were clicking that thumbs down button when they got a response. Which you might say, well, Ben, that probably means 99.2% of the time, they weren't responding negatively. And like, okay, yeah, but if it's 20 times that number is the true, not fully happy, then that's 84%. And that was at the start of 2024. By the end of last year, so by the end of 2024, that number had dropped to 0.3%, which is a pretty significant improvement. And we're talking over like many hundreds of thousands of tax queries. So it's empirically quite valid, the reduction by 2/3 in the negative feedback in the platform. And every time somebody clicks that thumbs down, they write in, they have an opportunity to write in. This is what I, like a text box pops up and says, can you give us more information? And that goes to the product team to figure out, okay, what's actually going on here. How did we disappoint this user this time? I challenged our product team at the end of last year to say 0.3%. It may sound good. Like we may sit around and pat ourselves on the back, but I think we should push harder because there are probably 20 negative kind of interactions for every one that we observe through the thumbs down button. Let's shoot for 0.1%. If we can get this down to 0.1% or better, then I'll feel like, okay, this is really great. One in 1,000 interactions then is leading to a user being disappointed enough to click that thumbs down button. By the end of March of 2025, the end of the first quarter of the year, it was down to 0.2%. And most recently, it's hovering just above 0.1%. So I think it was like 0.108% at the end of June. And so very rapidly, it is getting better. So Jeff, like the crowd agrees with you and we're seeing a lot more usage over time. So we've done millions of questions this year with a spike in usage in April during tax season. And then more recently in the wake of the pandemic, the big beautiful bill, like we actually set some new high watermarks for usage last week, as people are trying to figure out what does all of this stuff mean. And so it's super valuable to have this feedback from the user community, and they tend to be very sophisticated tax people asking these questions. And so we get very often, very detailed and very on target critiques of the answers, which we then try to generalize, understand. It's kind of a heat map of where in the product we need to focus our attention to improve the performance. It could be, we need additional content, or we need a different kind of answering strategy, or we need to set users expectations. So for a while, we didn't have all of the state and local tax information in the product. Now we have treatment of all 50 states. And so if you're asking about a state that we didn't used to have in, that might lead to a thumbs down. Now we have all the states. That's a work in progress. But it's exciting. So we're picking up empirically on this improvement in the product that you're witnessing anecdotally. So how much is the improvement actually changing your algorithm or changing something in the backend? And how much of it is just taking all this user feedback and dumping it into the system, and the AI just kind of sorts it all out and figures that out on its own? Like if you did nothing other than just feed all these feedback, how much would it improve? I think the vast majority of it is accounted for by improving the content. More content and better content. More content, better content. Part of it is pruning and curating out. I think since we last talked, all of tax notes is on there, that's a lot of content. But I have a question because you said pruning out. It's not just like putting more in. You've got to know the articles that Jeff Hoops wrote in tax notes because you wrote the patterns. Yeah, we've got to get rid of the noise and isolate the signal exactly. But that's really interesting. So can you figure out what is the signal versus what is the noise based on the types of queries where people are saying actually that one gave me the wrong answer and then you can look and say, oh, we got a noise in our signal because this tax notes article maybe was incorrect or something like that. Sometimes, I mean, the law is changing all the time. Scott, so the best example is the law has changed and yet there is a case making reference to an older version of a code provision or something. And then the system is finding that case and finding what the court said about that provision and then applying it anachronistically to an analysis of the updated code provision. So that's the cleanness example. We need to go through and introduce additional information basically into the system. To say, this is good for a certain period of time but not beyond the time when the law changed. Does a human actually look at that and decide that? Or is it just yet another AI tool that looks at it and says, oh, actually, we can see that this was updated. Stop using this. It's AI assisted, but our tax research team is doing all of this work. So we have tax lawyers, tax accountants who are doing this work. And it's like, they're smart. They're using AI to accelerate that work. But also there's a lot of judgment that goes into it. And you need to understand the ontology of all of the information to make really good decisions there. How many total employees do you have if that's not a secret? Yeah, we're close to 100 now. Last time we talked a few years and I feel it was way fewer than 100. Yeah, we're growing quickly. So is this an advantage that you have over just some generic thing like chat GPT? Like if I go to chat GPT, I could go to chat GPT and I could say, hey, I'm trying to figure out if this certain thing that I'm doing is legal and I could give it the scenario. And it will probably give me an answer. But what you're describing is a very almost labor intense process of curating the training material, which I would imagine, OpenAI cannot do because OpenAI is sort of like trying to capture the whole world. And you're just trying to capture the tax world with that most important part of the world. Most important. I think that's right. I think this prize isn't big enough for I think OpenAI to spend a lot of cycles trying to chase. I mean, their ambitions are global and much bigger than solving tax research. Whereas we have the luxury of-- I still think of solving tax research as an enormously powerful and important thing. And I think the other thing to note is, we're actually collaborating quite closely with OpenAI. Because like, think about it. If you're trying to improve a model, you want different situations where there is a better answer, but it's not obvious. And you can't bluff your way to a better answer. Taxes a fantastic area to really put new types of models to the test. And so we've been getting through our relationship with OpenAI early access to some of the new models. And working-- our engineering team is working closely with their engineering team, giving them feedback on different things that they are tweaking on their side, giving them feedback and working closely with them. So for example, when OpenAI announced the GPT 4.1, that new release in mid-April, I think it was April 14. They did talk about the work that we had been doing with them at BlueJ and how we had seen a material improvement in the model's ability to answer kind of the tougher, more complex tax research questions that we had been struggling with with the previous 4.0 model. And it was like a 53% improvement using 4.1. And so we're able to benchmark for them how the different iterated models are working. And we had the benefit of all of these answer strategies and the curated data and this domain expertise that their engineering team doesn't have. But we can give them a lot of signal about, does this tweak actually help or hurt the performance of the system for our particular use case? How do you-- when you say 53%, or whatever percent you said, what is that? Is that the number of thumbs up versus thumbs down? Or how do you measure improvement? So we have a database of questions, like technical tax questions. It's like-- You're going to test it. Exactly. There could be, at any given time, there will be 2 or 3 100 of these difficult questions. Almost like give it the CPA exam. How do you do? I'll be right. Exactly. Harder. And we've gone through. We've got these questions. And then we've got the canonically correct answer that our tax lawyers have gone through and laboriously assembled over time. And the success criteria that must be met in order for us to say, yes, this was a good answer. And we can automatically kind of run new model iterations through this using scripts and auto evaluate how well the models are doing. And then we go and have tax lawyers actually go and verify. If it looks like it's a candidate, that's an improvement. We go through and verify whether the criteria are actually met. So it's an initial kind of first cut that's quick and dirty algorithmically. And then we go through and have expert humans go and get nerdy on it and make sure it is actually doing what we think it's doing. OK, so I have two questions that are kind of related. One, it sounds like you're targeting your target audience. And I believe we've talked about this before. It's not like Joe Blow doing his taxes in his living room, but it's like a professional, like a firm that's a tax firm or something like that. So question number one is, what prevents-- why is what you're doing-- why does it have a sustainable competitive position? Why don't other people just replicate exactly what you're doing? That's question number one. Question number two is, how far away is it that something like this will just exist for Jeff and I to sit in our house when we go to ask our complicated tax questions and we just type it in and we get the super accurate answers that you're starting to generate at Ask Blue J. Well, number one, I think we've been talking about a lot of this stuff already, right? So one reason why this is difficult to mimic. Lots of people have been trying to build this stuff. So just think back to the immediate wake of the announcement of chat GBT, right? At the end of 2022, early 2023, I think a lot of the big accounting firms, the big law firms thought, oh, we can do something. We can leverage this technology. We can build our own tooling in-house. And a lot of those projects produced very quick early results that seem very promising. So somebody who has some level of capability, that coding could create a rapid around GBT 3.5, GBT 4, GBT, whatever, and produce a system that could do an adequate job of answering questions about the same as chat GBT. The challenge is creating something that is even better. And so you need a lot of content. You need carefully curated content. And then you need to maintain it. So it's not just a project that you do in a weekend or in a month, and then you're done. NewTak stuff is coming out every day. So you need to be updating the content and then intelligently ingesting that content and then identifying the knock on consequences of that new content. And like, what does this knock out from the existing database? So this is why you need an editorial team to maintain all of the content. And so there's this ongoing thing of like, how do we make sure we have the current information in the system and we're doing all of the housekeeping and the hygiene work necessary to keep the data set current. And that's not something that even big companies like to do. Big firms do not like to invest in people just to do this kind of data work. It requires a lot of expertise. And really, it kind of falls to somebody to be like, we're going to invest, we're going to build a team of a dozen people or more whose job it is to do this. And they're going to become very good at it. And so that's one source of this advantage. The other is this product improvement flywheel that we're talking about. So on a given day, we could have tens of thousands of tax research queries coming through the system. If one in 1,000 is leading to a thumbs down and a snippet of text describing what is not optimal about that answer, that's a lot of fodder for our product team to try to identify the hot spots in the product that we should prioritize for different R&D experiments to improve the performance of the system on those kinds of questions. That coupled with the fact that tax people really value their time. They do not want to use the crappy system or a general system when a better system exists. I don't know. Maybe I'm over-generalizing, but I myself will, like I really value using a better model. And I will, if I know something better exists, I will not spend my time chatting with something that is not, you know, the state of the, state of the art. And so, BlueJ really benefits from our, you know, our existing really high quality answers. And the fact that people perceive it as delivering the highest quality answers, leading to more usage, leading to more feedback, leading us to be able to improve the system better. Because the, like, what's the counterfactual? Okay, so BlueJ, say, is at 98% accuracy today, 0.1% times 20, call it 98%. If somebody else produces a system that's at 85% or 90%, it's going to be really hard to convince a user to deal with that kind of noisiness in their tax research. They'd just, like, well, BlueJ exists. I just want to use BlueJ. So, who is your main competitor? I mean, is there anybody out there trying to do this? Or are they still alone? There are some other startups that are trying to do this. They're in a difficult position. We are so far the only firm that has licensed all of the tax notes content. That's a lot of content. And we have a very, like, I love the tax notes folks. We have a great relationship with them. We've built all the data pipes so that as they publish new stuff, it gets pushed directly to us. And then we can ingest it immediately. It's a really great relationship. So we have a data advantage, vis-a-vis others who would be trying to come into the space. The challenge for the others is their main competitors are really chat GPT and perplexity and the other systems that are increasingly doing web search and leverage all that content on the web. BlueJ is now doing this too, but with a curated set of high fidelity websites that we know have really authoritative tax information. And so in addition to all the stuff that we've curated, we have a secret list of amazing websites, mostly government websites where we know that that content is kept up to date it's current. And sometimes it's not updated and so we'll actually suppress that for a while. So that doesn't muddy the answers. So an example is in the wake of the big beautiful bill, like we can't just be relying on, you know, at a date IRS guidance on the IRS's website right now. It's just like it's a lot of those FAQs are now at a date and they're going to be misleading users so we can turn that off in real time and just really tune the system so that we're flying as accurately as possible at any given time. So that's like that's the kind of the first question Scott. The second is like everybody can sign up for BlueJ like it's available. So like I don't know if you're just being super friendly by asking that second question, but you can just go to our website and sign up for BlueJ. There's a self-serve capability there. You can sign up, put in your credit card details and get access to BlueJ. How much? How much? Like if I'm an individual user, how much is it? It's $1,500 roughly $1,500 a year for access to BlueJ. So that's completely, that's 1000% reasonable for a CPA. So that a lot of returns, but if my sister is going to fill out one single return, she's not going to pay $15,000. So I think Scott's question is like when is the version that gives you, you could buy like per query, $5 a query or something like that. And you're really only to ask at do three queries to fill out your tax return. Yeah, I don't think it's that far away. The issue is like, let's, because I think only something like 3% of the potential beneficiaries as tax professionals in the US currently are subscribed to BlueJ. So let's first get like 90 plus percent of them on BlueJ. And then we'll talk about like getting it into other people's hands because I think that raises all kinds of go-to-market complications. But I think that's, I think that's the way things are going. I also, like I'm skeptical, like I think a lot of individuals have a very low willingness to pay for super technical tax information. They're used to googling things, used to getting information for free. There's this weird paradox where somebody who is more capable probably has a higher willingness to pay for really great technical information because they know what to do with it. And they are skeptical enough and well informed enough to be pretty sophisticated consumers of that information. So you mentioned that some big accounting law firm type people at the very beginning tried to create their own tools. It didn't work so great, just because they don't have the will to do it. That's not what they specialize in. So what are they doing now? I mean, are there a lot of these bigger type organizations that are using products like yours or your product or are they just not using AI tools at all? Yeah, we're seeing a lot of these projects circle around to the point of view that, oh, we should be leveraging the work of other firms to build specific things on top of. And so increasingly, we're seeing that kind of movement amongst the large enterprise kind of professional services firms to subscribe to BlueJ, role BlueJ out and start building on top of BlueJ, leveraging BlueJ alongside their own internal resources and then building on top of that joint informational backbone, whatever proprietary stuff they have. Plus, what they're getting from BlueJ in order to generate even better answers and then build low-code, no-code widgets on top of that informational backbone to accelerate all kinds of different workflows across their various service lines. Those companies keep paying you, which to me implies that they believe that there are productivity gains. Do they talk to you about the productivity gains that they're achieving because of your product? Or is anybody actually explicitly measured it? I mean, as empiricists, you might understand the complexities with actually measuring this kind of stuff. It's interesting that we've done some of this work with some of the bigger customers and it seems like kind of the central tendency, a lot of these estimates of time savings per user. It's on the order of between two hours and three hours per week per user in terms of time savings from tax research. That's actually kind of big if it's like say 40-hour a week. That's like 5 to 10% time savings. And even an intern at UI is going to be built a couple hundred dollars an hour. That doesn't take a 5 to 10 hour. It's big for itself. It's actually kind of big. Yeah, it doesn't sound that big to me, but I think you're right. You accumulate that across potentially thousands of tax people and the 52, 50 weeks that people are working in the year. And it's like, it's a very substantial time savings and money savings, right? I mean, huge for sure. On the other side for those of us who want there to be more accountants as opposed to fewer accountants because we educate accountants, if you're only saving two hours a week out of a 40-hour week, if that's like the limit to how many accountants you're going to end up firing because we're all much more efficient, that actually also makes me a little happier. Our massive accounting program isn't going away any time soon just because BlueJay does everything. There's the other thing. Per hour, people are more productive, right? So it actually cuts the other way, too, because if you think about the value per unit of time worked by an accountant, it's actually going to go up, right? And so this actually, this should lead to in equilibrium actually having more accounts rather than less accountants in practice. You know, it's complicated, of course. But there are things that point to an ambiguous result here. Yeah, I totally can see that. Kind of pushing this a little bit further. If I take your product and I say, you know, I have this scenario and I want to know what the right kind of tax Compliance answer is it will probably give it to me and now we know it'll give it to me fairly accurately What about tax planning like let's suppose that I'm going to open up operations in Europe And I'm trying to figure out and maybe Europe doesn't work because maybe you don't go multinational So maybe let's just say it's in the states or something I'm gonna open up operations in a different state trying to figure out where's the best place to Organize things like like just an X-anti tax efficient plan as opposed to an X-post Compliance role is this capable of planning X-anti or is it really only an X-post compliance kind of a tool? I would argue those are those are really the same kinds of things It's just a it's a it's a matter of You know sophistication and how much work needs to be done to Canvas all the possibilities if you're thinking about a really sophisticated forward looking Tax plan like you need to analyze the same kinds of things to take a position on something that's backward looking versus something that's Forward looking I think what I'd say Scott is as the systems become more and more sophisticated The tax planning Capabilities become more and more impressive. So if you were to ask me 18 months ago when our our Disagree rate was 0.8 percent. I'd say you can do it. It's like it's gonna be quite hit or miss Now increasingly it's becoming better and internally Based on what we're building inside what our data scientists are working on they're working on some some deep reasoning Stuff which actually has a bunch of recursive properties And so you may you may have to wait a number of minutes for your answer But the deep reasoning stuff that we are working on internally is absolutely capable of what you're describing because it's It's kind of looping back double checking its work double checking its work Exploring new angles. It's doing this like divergent convergent divergent convergent thing on the question that you're asking and Like tax planning is the sort of thing that that really excels at if you're like just like I want a quick like one shot Like you ask your question you like that's really good for the compliance sorts of things But this like divergent convergent divergent convergent double checking along the way is really I think that's where we're gonna see Really nice emergent tax planning capabilities and like I think this year like later this year It's gonna come partly I just imagine all these different scenarios that you know so You would say okay, I am this company and I want to buy that company and we're gonna engage the transaction to purchase one other There's a whole bunch of different ways to structure this Hey ask Pooje what's the most tax efficient way is at why and you feed it all the parameters and it just like goes through and tells you Like everything that's a lot of manual labor that manual Mental labor that would have been assigned to a bunch of lawyers and consultants in a previous world that might eventually just be done by asking your product So it's very very cool in my opinion, but I'm interesting to think about I think it's I think that's a holy grail. I think that's you know once I Mean I still think you want a human double checking all of this stuff and validating the steps and going through it and And thinking about you know the non-tax dimensions of this all the business dimensions Yeah, they're usually it's usually a Very complex freeform kind of decision space in which people are doing these kinds of planning exercises But this is a hugely valuable input into that that broader Kind of positioning strategically and thinking about the business and tax implications of different ways of proceeding for sure Then this is Very very interesting and it's it's fun to think about As we kind of all this navigate This new world of artificial intelligence and sometimes it's a little freaky and sometimes it's like astounding how amazing it is and It's very fun to think about so thank you for joining us again and talking to us about AI and taxes Thanks my pleasure and if you don't mind me plugging it People should go to bluej.com BLUEJ.com just the letter J and and check it out learn more if they're if they're interested and hey if you know $1,500 sounds like a good deal for you just sign up on the website and get going with blue J and Anna as Jeff mentioned Jeff And I did have access to it for a while and we used it and it was very it was very amazing very impressive so Thank you. Thank you for that and and check it out. It really is a it's a cool cool product My name is Scott Dyer and I'm professor of accounting at Duke University. I've joined as always by Jeff hoops at the University of North Carolina at Chapel Hill Our guest today has been Ben Allery who is professor at Osler chair and business law at the University of Toronto on Leave unpaid leave so that he can manage all of the amazing things that are happening at blue J Thank you so much for joining us. We'll chat with you next time. Goodbye [Music]