NeLI POD Season 2: Ep. 13 with Martha Louks

Welcome to Nelly Pod, the official podcast of the National E-Discovery Leadership Institute. Hi, I'm Daniel Gold, your co-host and navigator through the evolving landscape of E-Discovery. In my name is Brandon Mac, I'm the co-host of our podcast, Our Guests, from judges to E-Discovery professionals, share insights that cater to all experience levels. Whether you're looking for a broad overview or in-depth technical analysis, this podcast is your gateway to understanding and implementing effective E-Discovery practices. It's important to note, the opinions expressed on this podcast are those of the individual guests and the hosts, and do not necessarily represent their views of their respective employers. We're here because of our passion for E-Discovery and our commitment to providing free, accessible educational content to legal professionals everywhere. Welcome to the Nelly Pod, the official podcast of the National E-Discovery Leadership Institute. I'm Daniel Gold, joined by my co-host Brandon Mac. Our guest today has built a career at the intersection of litigation and technology. Martha Laux serves as managing director of Discovery Technology Services at McDermott, overseeing the firm's Discovery Technology Initiatives. Her expertise spans preservation strategy, predictive coding, and the practical implementation of modern AI tools. Martha joined McDermott over 15 years ago as a technology project manager, ascending to her current leadership role. And as a recognized relativity master, she holds five different relativity certifications. Her experience includes collaborating with the DOJ's Antitrust Division during the AB-Inbeth Groupo Madeleo merger. She designed predictive coding protocols to satisfy regular regulator demands by isolating critical documents, proving that her technical workflows withstand actual rigorous external scrutiny and art just theory. And I have to tell you that if you don't know who Martha is, you have definitely seen her unlinked in for sure. She's a frequent speaker at Legal Week, Relativity Fast, A-Seds, Ilta, you name it. Her contributions emphasize, and this is what we love the most and why we're so excited about talking with her, practical application over industry hype. That's a big difference. And having developed AI-cisted review and investigation strategies long before the rise of generative artificial intelligence, she has a very unique seasoned perspective on the current technological landscape. So today we're going to explore the evolution of Discovery alongside modern data. Sifting from paper and email to mobile and conversational evidence, we'll discuss where AI earns trust, where human judgment remains vital, and how the next generation should navigate these careers shifts. So Martha, thank you very much for joining us today on the Nellie pod. Thank you so much for having me. I'm excited to be here. Well, Martha, I have to start off with this. Look, you have worked through paper. You have worked through email. You have worked through mobile data. You have worked through TAR and now generative AI. Which of those transitions actually changed how lawyers think about a case? And which ones just changed the tools simply on a lawyer's desk? Ooh, that's an interesting question. Can I say all of them have done both in different ways? Yeah, so walk us through that. So going from paper to databases, the big innovation there was we can all work in the same place and share work product in one spot. That was the selling point. In fact, then for me to attorneys was, if we work on paper, then everybody is siloed in the specific thing that they're looking at. If we put it into a database, we can share work together and collaborate. And so there's more efficiency to be found that way. So I think that kind of brought efficiency and standard practice into a little bit more of a more visible way than it was. There were of course standard practices with paper, but not in the way that it was more regimented when we had to do it in a database because everybody had to be clicking the exact same buttons. That's what we had was one responsive button. At least you hope there's only one responsive button. But Tara, it's interesting when we started doing machine learning. I guess even just the advent of ESI was sort of interesting. The fact that you could deduplicate documents was to me at the time extremely exciting. I had reviewed a lot of dupes. And so it was exciting to me that the computer could do anything. And I remember talking to people about the rigor of hash values. There was a lot of, I don't know if it was skepticism exactly, but lawyers were doing their due diligence as to the technology. How robust is the d-duplication? How do we know for sure that a duplicate is in fact a duplicate? Well, it is. Now we don't even think about it. We just do it. But back then, that practice was asked about and had to be defended. And the same thing happened with technology assisted review when we started doing machine learning with it, whether it was simple passive learning or continuous active learning. How do we validate it? We know the results are good. There were a lot of external studies that were done that were cited too. And it took a long time for attorneys to become comfortable with using the technology. So that one took quite a while for I think there was court acceptance of the use of the tool, government acceptance of the use of the tool. And now that that's all, that bar has already been cleared. I think there's faster adoption with the Gen AI tools. There's a little bit of familiarity with the process. It's a new tool. But the same rigor is used in invalidating the results from these new applications. So this one, there's a little bit more explanation into the details of process. But there's less skepticism about the potential benefits of applying a new technology to workflow this time around. There's a lot in that answer. A lot of little things you've seen change in the discovery industry over the years you've been involved. And I think there's something there in that your career seems to show that what we typically think of as legal expertise has increasingly over the year extended beyond what are traditional attorney roles. Do you think that firms are really recognizing that as quickly as they need to as the landscape kind of shifts with technology? It is interesting. I think so. I would say so. I see a lot of different job titles out there. So there is certainly appetite in the market for technology professionals or hybrid legal technologist roles. Folks who have, whether their attorneys or not, they at least have experience working on legal matters. And there's a lot of growth at seams in knowledge management and innovation positions, at least at law firms. And in house, it seems like there's a growth of AI centered roles recently. Now I'm interested to see how sticky these these roles are and you know what value these organizations are finding from the different positions that they're developing. So it's there's a faster adoption. It feels like there's this almost like a race right now in a way that there didn't seem to be in the mid 2010s mid 2000s comparatively speaking. There's a little bit more big tech hype around Gen AI that I think has flowed into the legal technology space that didn't necessarily happen the same way with machine learning. You can see me here me right? Yeah. Perfect. Yes. Yes. Okay. Martha, let me ask you this. What we have seen, we talked a little bit about this right before we started recording. Most discovery platforms over the last, let's call it 20 years, they've really been built around the document. And the document has always been like that's the basic unit review right? And then we start adding all the structured data and so we've got slack threads right and we've got text messages. And the funny thing is is that like we're still calling those documents right right there they're not there conversations that happen to also have context that lives outside of the entire message itself. Here are our platforms and our review protocols still catching up to that. We're just going to keep rock in the document object. It seems to me I've been talking about this for years now that it's data and not documents. I think I started on this during COVID where I was like it's the document. Is it even real? You know, I mean a little bit from paper days, sure, but even email isn't really a document. It's a database entry and it's segments of communication that could be parsed into smaller data pieces and then strung together. And yeah, it's a real challenge in our systems that we are kind of locked into the system of documents. It's also interesting that we still produce largely in TIFF image. We make fake paper, you know, and I mean it's, it is what it is. I guess I don't know if you have to redact, I totally understand that and you do want maybe a page number, particularly when you're citing to something. But largely speaking, you know, it increases the hosting size. It's a lot of files. It's a lot of stuff to manage and it's like shoe horning, modern. data into the format of yesterday year. It is kind of a curious thing. I mean, there are some reasons for it. I do, I see them, but I do question its continued utility to us. What I see as the big challenge is that when we're focused on these so-called documents, it's really just a predetermined slice of data, we are lacking the ability to track activity and analyze activity across sources and put together a timeline. So if I, for example, let's say I've got slack chats, or now I guess people could be grabbing, it's kind of interesting, you're chatting with AI bots all day, clawed or chat GPT or whatever. We're going to put those into 24-hour slices, too. Does that make sense? I don't know, maybe. But in any event, if I put all 24 hours into one document, my date for that document is the first date and time of the first communication of that day, and everything else gets collapsed. And it's not really something that I could analyze. So if a person is chatting all day, and then it's free in the afternoon, that chat inspires them to go into another platform and do some activity and talk to another person about a topic that's relevant to the case that's going to be harder to piece together. It's just going to take more time. So I'd love to see innovation in platform design around this. It's something I would like to see. I don't know, though, whether it's the most important thing for many attorneys in practice in the way that the way I think about data does not always match up with how an attorney might think about data in the litigation. And a lot of times, I would go maybe down to pass of deep data analysis. But it's simpler sometimes to just ask a custodian. You could just ask them in a deposition. It's another way of doing discovery. Another thing that I think you've been talking about for a while is AI. And I mean, AI is the umbrella term that covers everything from machine learning all the way where we are with Gen AI. And you were discussing AI and AI practices long before everyone discovered chat GPT and clawed and everything else. When you look at this, and you look at the evolution of what we've done with different forms of AI over the years, what misconceptions do you think the market has right now, or that attorneys have right now, that reminds you of the earlier AI hype cycles that we've seen. You mentioned that you think the adoption's going to be quicker this time, or is quicker this time. There's a little bit more trust. But what things are we getting wrong this time, or are proceeding wrong? The hype is frustrating, to say at least to me. Yeah, there's a lot of misconceptions about AI being able to do a lot more than it can actually do in practice. You have to check everything. With the discovery, it's kind of OK. If it doesn't get it right 100%, when we're doing a big doc review, we've built in margin of error. So if it gets it wrong, 3%, 5% of the time, OK, I'm not expecting a perfect result. But if you're using it to draft a brief, and then you're submitting that to the court, that's a whole different ball of wax. I mean, that is much more concerning. And there's a higher standard that needs to be applied to that work product. So I think there's a difference between using AI on a high volume task versus precision work. And you have to verify all of the output. So the question to me, especially for precision work, is just the time and cost of verifying the output, actually make the cost savings negative, or are you still getting at least some return on your investment in the use of the tool? I think there's a lot of hype. The marketing vibe is very simplistic. That AI can do in minutes what used to take hours. Well, maybe. I mean, I don't know how many minutes are we talking about? How many hours was it? How much are you going to have to verify? There's just the devil is always in the details. And a lot of the hype, it smooths over all details. And a lot of the tools, they have huge promises, but they're very thin on how they present, at least in marketing materials. They're thin on talking about the concrete ways that they work. I would love to be able to just go on a website and be able to read and understand, like, oh, OK, it has x, y, and z function. It does these things. And without having to necessarily talk to somebody and go through a sales cycle to understand the basics of a product. So yeah, there's a lot of-- and there's a lot of fear mongering, too, this time around. There was before, but it's worse this time. A lot of like, oh, it's going to replace the lawyers. It's going to do this. And I feel like it's another sales tactic in a way. It's interesting you say that because I remember vividly back then when we were doing tar. And it was like, hey, guys, the robots are going to take over. And it was-- and at least from a sales perspective, a sales cycle perspective, you are dead on Martha. The hype was, you got to get on board with tar. Otherwise, it's basically going to replace your job. And that's literally what everybody was talking about. But the problem was, is that it never latched on. There was too much talk about richness of the data set. There was too much talk about the algorithm. There was too much talk about a seed set. And from a lawyer's perspective, it's like, I just want to review the data, right? Just give it to me. And it seems as if the hype now is AI has removed that administrative burden a lot from what we used to talk about with tar. Because now it's just get into it. And you have the answer. Just ask the tap, but you'll get into it. So I'm kind of curious, based on that premise alone, if you think that-- and you talked about a margin of error with the Docker viewers-- I'm wondering whether or not you feel like now with the reviewers knowing that there's a margin of error, that there is work that AI can take off of the reviewers play without even thinking about it. Like, it's just, boom, let's do it. Like, we shouldn't even think about it. Let's get it off of their play, where-- and there's not a risk involved. Do you see that where there's the obvious no-brainer? Let's just use AI with the reviewers. Well, so with a Docker review, I'm still going to have to validate it. And using Gen AI is still technology-assisted reviews. I'm using it at scale. I'm still going to have to do an illusion test or some kind of sampling. And at some level, you do want to know what's in your documents that you're producing, at least. So you can chat with this, and you can learn some stuff. I think the funnest application that I see the easiest application that I see is when you have your hot docs, it can extract facts and help you create a timeline. That's so much busy work that people would spend time typing in the facts or copying and pacing from the document. You're going to still need to confirm that the fact is correct. But theoretically, you're looking at your timeline and thinking about the results in light of the actual evidence. So that would come up as part of your QC process. But to me, that's like-- I think that takes a lot of manual work away. So that's one area that's very useful. And I think it's great for early case assessment or an investigation where you don't really know much. Just run a prompt, throw it in, see what comes up, and then use it to learn a little bit about your documents. But these were all techniques that we've been talking about for years and years and years. So it all just depends on the appetite of the team to kind of get their hands dirty and start digging through the data a little bit. You're still going to have to think, you need to use your brain. That's never going to go away. I hope it doesn't. I don't see why it would. It shouldn't. It shouldn't. You mentioned in there, quality control, QC checks, needing to validate. You referred to Genai. Still, it's just another TAR process. When you think about the principles that we learned of validating in the world of TAR, which were a little bit different from what we did when we were talking about manual review. And you might QC, 10%, 20%, whatever you might do at a particular organization. We changed that. Got really statistical with TAR. What lessons or what cornerstone principles do you think from our TAR learning, we need to keep as we start looking at Genai. Are there changes? From the statistical standpoint, no. What's interesting to me, though, we have studies that show that computer assisted review is more effective and higher quality than a large review team. And it's basically because the computer is consistent, super unfailingly consistent in a way that human beings never could be. So if you scale that across a large set of documents, there you go. got big consistency or something. results. I have never had an opposing party interrogate a manual review and say, "produce quality statistics to me. Prove how good your review is. Prove how good your search terms are." We trust people more than technology even when we have evidence that says we should trust the technology more. So that is really interesting to me and I wonder if that's still going to continue to be the case. But there's a trend of, I don't know if you think, even a trend is the way we interrogate the results of computer assisted results far more than we do human results. And yet, the results are the results. I mean, that's really interesting. I mean, we think about it and we have human document reviewers back in the day. Let's, you know, the debacle litigation or whatever it might be, right? Massive class-action litigation and we have human reviewers that get tired. They don't have breaks. They're hungry. They need, you know, go out for a break or whatever. And they're just sitting out the computer reviewing and reviewing and the chances are high. We're talking about margin of error again. Chances are high that mistakes do get made. But you just said something interesting, which is we kind of overlooked that a little bit, right? We don't really look at the human reviewers and say, "Well, did you actually review that one document correctly?" And is that statistical anomaly that you mark something as non-responsive when it really was or as a turning client privilege when it wasn't, etc. But with the technology that doesn't get tired, that, you know, objectively looks at facts in a certain way. I mean, those are things that AI is selling to the saying, "Look, we're going to be more objective, more efficient. We don't get tired. This is the AI way, right?" But it's interesting that you're saying that, "Well, we're questioning the technology more, but why is it that we never questioned the humans more?" I mean, aren't we at a point where it's like, "Well, if we could really do first-pass review driven largely by an AI-agent AI workflow, right?" And then we get to a second-pass review and now we're ready to put the humans in play because we obviously have to validate, "Wouldn't that be a more productive workflow?" Or do you object to that? What was the question again? Yeah, I know. I just went on. I mean, think of it this way, right? If the humans get tired and the humans make mistakes, but we don't question the humans, and the technology doesn't get tired, but the technology can make mistakes as well, but maybe it's less likely. And the AI is more effective at a first-pass review. Why don't we just accept the fact that AI can do a better job at first-pass review and then put the humans, elevate the humans to the second-pass, this quality control check review, and just accept AI as being, "It's going to do first-pass review, save all those costs at the bottom layer." Yeah, I see a couple barriers to it. Why don't we? In practice, it's a good thing that there's case law that discusses technology-assisted review, the acceptance of it by the courts, validation, and having a defenseable process and all of that. However, I think that in the, there's a consistency in that the case law wants the parties to collaborate with each other and in an adversarial litigation, when you start talking about a technology-assisted review protocol, it's like fodder for another fight. And so a lot of times, I'll see parties say, "Nadda, we're just going to do search terms. I don't want to have to have this argument about it." Now, interestingly, there isn't an obligation to disclose your use of the technology, at least in, you know, civil litigation. There's Sedona Principle 6 where the producing parties in the best position to determine the means and method of production, something to that effect. So I say, "Go with it." And then focus your reviewers on the media or substantive analysis. Use the AI to lock off the non-responsive stuff. Don't waste the time reviewing that. Anyways, it's not going to add value to your case substantively. And then focus the review team on the issues related things. You know, we've got classifiers for that. You can use Gen A.I. for that. You could use a chatbot. You can use clusters. You can even use search terms. You can do all the things and find the interesting documents that actually add value to the litigation process. But I think the, there's a sense that you have to have agreement on all of these details. And in an adversarial case, that's kind of a heavy lift. You know, I mean, who wants to burn political capital on that issue when you are going to need to use it for something else in the case. But I think, say less. Do more. Say less. That's interesting. When I was listening to Daniel's question that you just answered, I thought, "Oh, we're getting dangerously close to that, that the negative hype of AI that, oh, it's going to take all your jobs." You know, we've got a whole group of people out there who spend their time doing review, what are on a contract basis or as young, young litigators and firms. And now we're talking about that first-level review disappearing and being done by Gen AI. But then I listened to your answer. And what I heard in your answer was that same group of people just have an opportunity to evolve their job a bit. Now, that as we look at what happens with people who specialize in review over the next five, ten years, it sounds like you're just seeing an evolution of moving to different areas of review or different areas of discovery, just looking at discovery differently with Gen AI as sort of the vanguard of what you're doing. Is that kind of your philosophy on what you think happens to the review industry? I think, yeah. And I think, you know, there may be some cases where there's unnecessary review that's already happening. And maybe that kind of goes away a little bit, but there is still a need for review. You're still going to have to know what is in the stuff that you're producing. You have to know if it's helpful to your case, if it's harmful to your case, you're still going to have to assess documents for privilege. I don't think the standards are disappearing. The privileged doctrine is, you know, 100 years old. So we're still going to care about privilege. Logs are not going away. Privilege logs are still going to be a factor, even if there's a draft description, someone's still going to have to confirm that that's accurate and worthy of including on the log. And then there's other new functions like one of my favorite new things with Genai, new ish is using prompts to structureize documents. So if you've got, let's say, 100 PDFs, let's say their letters, I can use a series of prompts and say, what is the date of the letter? What is the provision in this paragraph? Who is the recipient? And it can pull that information out and put it into a table. That's a type of project that can be very useful for a number of different use cases. And that still needs to be reviewed and confirmed by legalize. So the nature, that's something kind of new. It's almost like bibliographic coding in a way, in a way. So yeah, there's always going to be a need for humans because this stuff isn't perfect. And if you are going to rely on the output for anything truly substantive in case, then you do want to have some assurance that what you're relying on has been confirmed. We would be remiss if we didn't touch back on something you talked about before, Martha regarding costs. And it's interesting that AI runs on this infrastructure and token costs, right? And they don't really show up in the old billing model. And you talked about how there may be potentially more costs involved here. So I'm curious from your perspective, do you think right now or will it eventually that have AI actually lower the cost of litigation? Or are we just we just playing a shell game here, moving from one line item to another line item to stay basically where it was? I mean, like what do you think is going to happen? Or what do you think is happening right now? That's a really interesting question. A little bit of a shell game maybe, right? So I guess the way I would do it is calculate my cost. I look at the total project cost. So there's your technology fees, processing, analytics, you know, whatever it is that you have a line item for. If I'm reviewing less and using more technology, then I have to consider, is there an overall savings for that project? One thing we haven't talked about though, the volume of information is still going up. That is not stopping. And the nature of the information is changing too. I don't know if you guys are seeing it, but there's a lot more video, a lot of recordings, a lot, you know, I mean, all these different chatbots. There's just more sources for more information. So even if we review less as a percentage of the total, that doesn't mean that the total amount of stuff that has to get looked at is actually less. Do you know what I mean? Yeah. So I may have to spend more on technology to review slightly less or the same amount than nature of the work. It's hard to see it disappearing. It's hard to see it disappearing, but yeah, I would, you'd have to really think through your workflow. I like to combine the tools. There's nothing wrong with old fashioned classification either, you know, like technology assisted review. Do that on a day to set. Get ready or non-responsives. Use Gen AI on some, you know, targeted stuff. set perhaps if that's a cost concern and then you know use the tools that make the most sense for the task at hand. Yeah. And then consider budget on every level. But yeah, I don't know. I guess it depends on how the users approach their their budgeting and workflow. Yeah. Yeah. Well, Martha and your imagination, if you could look forward five years, so it's it's 2031 and you're looking for the perfect AI enabled litigation team of the future. When you look at that, what does that team look like? And if someone were just now getting into the industry or young professionals who's looking to be in litigation, you discovery, what advice would you give them on how they prepare for what you think the world will look like in five years. Be flexible. You just got to be super flexible and have strong fundamentals because even with the new technology as we've been talking about this is a theme that's come up in this conversation, the fundamentals remain the same. You know, I'm going to approach a project today with the same rigor that I approached a project in 2012, actually probably even more. But we have to be super flexible every six months, we're encountering some weird data that like it came out in a different format now than it did six months ago. So you just need to be strong on understanding what it is that you're looking at so that you can kind of massage the data into something that makes sense. And you're able to understand and staying up to date on new tools like what are people using to get work done because that's where the information will will live. And so I there's going to have to be folks it's about the I be flexible process minded and adhering to the core principles of good work. And staying curious on things and skeptical. I think a healthy dose of skepticism is really useful. I love the core principles and I love the curiosity pieces. I think those are so great. Martha, it's been so great having you on the L.A. pod podcast. Thanks. This has been a fun conversation. Yeah, thanks so much for joining us. If you've enjoyed our discussion and want to learn more, don't forget to subscribe to our podcast on your favorite streaming platform. We're here to help you stay on top of the latest trends and developments and you discovery. We'd love to hear your thoughts and questions about today's topic. Reach out to us via our social media channels or email us directly. Your feedback makes our program stronger and more tuned to the topics you care about in the world of legal technology. Remember all the resources in insight shared today are aimed at enhancing your practice and understanding of the discovery. Join us next time on L.A. pod as we continue to bring you expert knowledge and lively discussions. Until then, keep innovating, keep discovering and keep pushing the boundaries of legal technology. Thank you for listening and we'll see you next time.