LangChain and LangSmith for LLM Applications

Welcome to the Deep Dive. Today, we're getting into a really big topic in the world of large language models. It's all about how we actually take their amazing power and turn it into reliable apps we can use every day. - Right, it's huge hurdle. - We've looked at a bunch of sources on two specific tools designed to help with exactly that, Langchain and Langsmith. - Yeah, think of it like LLMs are these incredibly powerful engines, right? - Okay. - It's a good car built around them, a chassis, and someone who knows how to drive it on tricky roads. Langchain is kind of like that framework. - A chassis. - Exactly. It gives you these modular parts, these building blocks, abstractions, to actually construct complex applications using those LLMs engines, and it's open source, which is a big deal. - Okay, so Langchain helps you build the thing. - Right. - It's the tool kit. Our sources say it's all about making these complex development jobs, well, simpler. Standardizing the tools. - Precisely. - So what are some of the key pieces in that tool kit? - Well, at the core, you've got components. These are like the Lego bricks, the fundamental parts you use. - Right. - Obviously, there's LLM itself. But then you need things like prompt templates, super important for telling the model what you want, clearly. And then you get this raw text back. So you need output parsers to structure it, make sense of it, and getting information into the LLM is critical too. That's where document loaders come in. They can pull data from all sorts of places. - Like what? - And you know, standard stuff like PDFs, text files. - Yeah. - But also more interesting things like directly from web pages or even extracting transcripts from YouTube videos? - Wait, YouTube, seriously. So you could build an app that watches a video and summarizes it. - You absolutely could. Langchain can handle that whole pipeline, load the transcript, break it up using text splitters so the model can handle it, create meaningful representations with embedding models. - Right, the vectors. - Exactly. - And then store those in vector stores so you can find the relevant bits super quickly later. That whole combination is like the foundation for powerful R.A. systems, retrieval augmented generation. - Got it. Individual tools doing specific jobs. But how do you connect them? How do you make them work together to do something useful? - Good question. That's where chains come in. Think of them like recipes or workflows. They define a sequence, maybe take user input, format it with a template, send it to the LLM, parse the output. That's a simple chain. Langchain has lots of pre-built ones for common tasks. But the real flexibility comes from building your own custom chains. - And how do you define those custom chains? - There's something called the Langchain Expression Language or LCEL. It's a declarative way to compose these chains, makes it really clean. - declarative. - Yeah, you say what you want the chain to do, not necessarily has step-by-step. And LCEL unlocks cool stuff like streaming outputs back to the user or running things asynchronously, which makes apps feel much faster and more responsive. - Okay, so chains are for structured workflows. What if the application needs to be more dynamic? Like decide what to do next based on the situation? - Ah, right, for that you need agents. - Agents. - Yeah, instead of just following a fixed chain, an agent uses the LLM itself as a kind of reasoning engine. - Like it's brain. - Kind of, yeah. It figures out the steps needed and it can interact with the outside world using tools. - Tools like what? - Things like a search engine to look something up or maybe access a database or call out to an external API. It lets the agent gather information or perform actions dynamically, much more flexible. - That's sound incredibly powerful, very versatile. But how do you keep track of the conversation? Like if you're chatting with one of these apps, how does it remember what you said before? - That's a good point. That's handled by memory components. - Memory, okay. - They basically let chains and agents hold on to information from previous turns in the conversation. So you get context, the interaction feels more natural. It builds on itself. - Right, avoids that goalfish memory problem. - Exactly. And one last piece, callbacks. Think of these as like little hooks. Yeah, they let you tap into different points in the application's life cycle. So you can, you know, log what's happening, monitor performance, trigger other actions that specific stages really useful for visibility. - So Lenzhen gives you this whole box of very flexible, connectable parts. The source is really stressed how this modularity speeds up development. How does that play out? - Well, because it's modular, you get a ton of flexibility. Let's say you wanna try a different LLM provider, maybe compare OpenAI to Anthropic for cost or performance. You can just swap that component out relatively easily without rewriting everything. Same for trying different vector stores or other tools. - Makes experimenting much easier. - Definitely. And Lenzhen has built-in integrations for tons of LLM providers, data stores, third party services. It just makes connecting things much smoother. That plus the power of chains and agents means you can build really complex stuff faster. And honestly, it's pretty user friendly with good documentation. So prototyping is quick. - And it supports Python and JavaScript, right? TypeScript too. - Yep, Python and TypeScript primarily, which covers a huge chunk of the developer community. And it seems like it's really matured now, especially since version 0.1, they're committing to stable APIs, which is crucial for production. - Absolutely. Lenzhen isn't just experimental anymore. It's a solid foundation. But building the app is only half the battle. - Right. - We're looking sure it actually works reliably, consistently out in the real world. That's a whole other challenge. - And that brings us to the second tool. Lengsmith. - Exactly. Lengsmith is like the quality, control, and observability layer you put on top. - The cherry on top, some sources called it. It's not just about building. It's about ensuring performance. - Precisely. Lengsmith is this unified platform designed specifically for debugging, testing, evaluating, and monitoring LLM apps. It doesn't matter if you built it with Lengchain or something else. - Oh, so it works, even if you don't use Lengchain. - Yeah, it can, though the integration is obviously tighter if you do. Its main job is to help you get from that cool prototype stage to something you can actually trust in production. And our sources mentioned it tackles those tricky LLM problems head on like they don't always give the same answer or non-determinism. And the workflows can get super complex. - Exactly. How do you fix something if you can't see what it's doing? Lengsmith gives you that deep visibility. That's its mission. Provide the tools you need to build reliable high quality LLM apps. And yeah, it's built by the same team as LLMchain, so they understand the pain points. - So think about the typical workflow. - Right. - You develop the app, maybe using LLMchain. - Right. - Then comes this critical phase before you deploy it. - Yeah. - Where does Lengsmith fit? - Right, in that middle stage. - Productionization. - Productionization. - Okay. - This is where you hammer on the application. You test it, you evaluate it against tons of scenarios to make sure it's actually good enough, reliable enough. Lengsmith is crucial here. - And then you deploy. - Then you deploy. But Lengsmith doesn't stop there. It keeps working, monitoring the app while it's live, making sure it stays healthy. So it really targets that productionization bottleneck that phase can take ages, right? Trying to iron out all the kinks. - It really can. LLMans add unique challenges there. Lengsmith aims to shorten that cycle by giving you specialized tools. It bridges that gap between, wow, cool demo. And okay, this is a dependable product. - Okay, let's get into those specific capabilities. And chapter three is all about Lengsmith's features, starting with debugging and tracing. This sounds essential given the complexity. It absolutely is. The tracing feature in Lengsmith is, well, foundational. It logs pretty much everything about how your LLM app runs. The inputs it got, the final outputs yes, but also all the steps in between. - Internal working. - Yeah, exactly. What parameters were used for each LLM call? How long each step took? How many tokens were used? Even the estimated cost. It's the super granular step-by-step view. - Wow. - And that detail is priceless for figuring out why an LLM gave a weird answer, you know, because they can be unpredictable, or for finding the exact spot where an error happens. - Is it like a flight recorder for your LLM app? How do you actually make sense of all that data? Is it just a huge log file? - No, thankfully. Lengsmith has a web UI that visualizes these traces really clearly. It shows them hierarchically. - Hierarchically? - Yeah, so you can see the structure, like this chain called this LLM, which then called this tool, which then retrieve this data. You can follow the whole flow of calls and data easily. - Makes it easier to spot where things went wrong or took too long? - Exactly. Bottlenecks, weird D-tours and logic. It makes them much more obvious. - And if you have thousands, maybe millions of these runs logged, how do you find the ones you care about? - Good question. It has strong filtering and search. You can filter by things like runs that had errors, runs that took longer than a certain time, runs from a specific user, or even search the content of the inputs and outputs using natural language. - Natural language search, nice. - Yeah, like show me runs where the output mentioned refund. And if there was an error, it gives you the detailed error message, the stack trace, all that stuff to help you fix it. - Okay, so how do you turn this tracing on? If you're using Lengchain, is it complicated? - No, surprisingly easy. You basically just set an environment variable. Lengchain tracing V2 to true? - That's it. - Pretty much. Your Lengchain app then automatically start sending all that trace data to your Lengsmith project. You can also use a decorator at traceable in your Python code to specifically mark functions you want to track closely. - Okay, so debugging is covered. But just because it runs doesn't mean it's good, right? How do you ensure quality? Testing and evaluation must be key. - Absolutely central. Lengsmith has a whole suite of tools for testing and evaluation. It starts with creating datasets. - Datasets. like test cases. Exactly. Collections of input examples. And often you'll include reference outputs like the correct answer you expect for that input. You use these data sets to systematically check how your app performs across different situations. How do you make these data sets? Sounds like work. It can be, but Langsmith tries to make it easier. You can create them programmatically with their SDK. You can import them from runs you've already logged. Maybe flag some good ones and bad ones. Oh, that's handy. Or upload a CSV file or just build them right in the web UI. It handles different types too. Simple key value specific LLM input output formats, even chat conversation data sets. And it versions them. Versioning. Why is that important? So you can reproduce your tests later. You know exactly which version of the data set you ran your evaluation against. Super important for tracking progress. Okay. Data sets created. Now, how do you actually evaluate against them? Langsmith gives you a bunch of ways. There are built-in evaluators that check for common things automatically, correctness, maybe, or conciseness, relevance, coherence, helpfulness, even harmfulness checks. Armfulness like toxicity. Yeah, checks for things like misogyny, criminality, that kind of stuff. But sometimes you need something specific to your application. So you can create custom evaluators. You can either describe what you want to measure in natural language. Like does the response directly answer the user's question? And Langsmith can use an LLM to judge that. Or for more complex logic, you can write your own evaluation code in Python. Using an LLM to judge another LLM. Yeah, that's the LLM as judge technique. It's surprisingly effective, especially for subjective qualities. And you can do it with a reference answer or without just judging the output on its own merits. And humans. Can people get involved? Absolutely. Langsmith has features for human annotation and feedback. You can send runs to human reviewers, have them score responses, leave comments, label things. This is gold for capturing nuance that automated metrics miss. So you get a multifaceted view. Automated checks, custom logic, LLM judges, and real human eyes. What about comparing different versions? Like if I change a prompt? Langsmith handles that through experiments. Every time you run an evaluation against a data set, it logs it as part of an experiment. So you can easily compare results side by side. Ah, so I can say, OK, prompt version to perform to better on relevance, but where is on conciseness than version one? Exactly that. And you use those data sets with the golden outputs for regression testing. Make a change, run the tests, make sure you didn't break anything or make performance worse. OK, building, debugging, testing, evaluating. What about when the app is actually live running in production? That's where monitoring kicks in. Langsmith lets you set up live dashboards. Dashboards, what do they show? What time view of your key metrics? Things like cost, how much are these LLM calls costing you? Latency, how fast are the responses? Token usage? And importantly, those quality metrics from your evaluations or user feedback scores. So you can see a performance suddenly drops. Right. And even better, you can set up real-time alerts. Alerts. Like get a notification if something bad happens. Exactly. Get pinged if, say, your error rate spikes or latency jumps up. Or you start getting a lot of negative feedback scores, helps you react quickly to production issues. What metrics can trigger alerts right now? Currently, things like error count or rate, average latency, average feedback score, they're planning to add more, like run counts and token usage too. And all this logging and production gives you data over time, right? Yeah, that's the production analytics piece. You build up this history of how your app is performing, how users are interacting with it, super valuable for finding trends in areas to improve long-term. So it's proactive, not just reactive. Yeah. Okay, another huge part of working with LLMs is prompt engineering. Does Lang Smith help there? Oh, yeah, definitely. It has dedicated tools for prompt engineering and management. There's the playground. Playground? It's like an interactive sandbox. You can try out different models, tweak your prompts, see the outputs side by side immediately, compare them. You can even test prompts against your data sets right there. Can you handle images and stuff too? Multi-modal? Yeah. It supports multimodal content now, like images and PDFs in the playground. Pretty handy. And then there's the prompt hub. Prompt hub sounds like a library. That's basically what it is. Essential place to save, share, version, and manage your prompts and their configurations. Like Git, but for prompts. Makes collaboration on prompts way easier, I bet. Huge. Everyone's working from the same source of truth. You can track changes, reuse good prompts, really help standardize things. Collaboration seems key. How else does it help teams work together? It uses shared workspaces, so a whole team can see the same runs, data sets, experiments, prompts, all organized together. And sharing specific things is easy. Like, hey, look at this weird trace I found, you can just send a link. You can even share traces directly into Langchain GitHub issues if you think you found a bug in the framework itself. Nice. What about permissions? Big companies need control over who sees what? Yeah. All set ups, especially enterprise, it offers roll-based access control, RBAC. So admins can define who gets access to which workspaces and what they can do there. Good for security and organization. Yeah. Okay, last thing on features flexibility. You mentioned it works without Langchain and deployment. Right. Big plus that it's framework agnostic, even though it shines with Langchain. For deployment, you have options. Look. The easiest is the cloud sauce version. It's hosted by them on Google Cloud. Just sign up and go. But if you have really strict data privacy rules or compliance needs or just want total control, they also offer self-hosted options. You can run Langsmith on your own infrastructure using Kubernetes or Docker Compose. So cloud convenience or on-prem control? Good to have choices. It really does sound like a complete package for managing the whole LLM lifecycle. It really aims to be that control center. Build, test, monitor, refine. Do it all with more confidence, more efficiency. Go K chapter four. Let's talk about the impact. How does Langsmith actually change the way people develop LLM apps? Well, it streamlines and frankly upgrades the whole process right from the start during development and prototyping that instant tracing and debugging is huge. Yeah, seeing inside the black box early. Exactly. You understand how your components are interacting way sooner. You can experiment with prompts, swap out LLMs, see the effect immediately. Visualizing the flow helps you iterate much faster. It's like having a debugger designed specifically for these weird complex LLM flows. Precisely. But, its biggest impact arguably is in that productionization stage, the testing and evaluation part. Right, getting it ready for prime time. Langsmith makes that process much more rigorous and systematic. You've got systematic testing using those data sets we talked about. You get comprehensive evaluation using all those different methods built in custom LLM as judge hitting quality from multiple angles and the human feedback loop. Crucial. Integrating human feedback for that nuanced understanding. And of course regression testing to make sure you don't break things when you make updates. It turns evaluation from an afterthought into a core part of the workflow. And then once it's live, in deployment and monitoring. It keeps providing value. The production monitoring dashboards give you that real time pulse check on cost, latency, quality. The alerting catches problems before they escalate, hopefully. And capturing all those production analytics gives you the data you need for continuous improvement over the long haul. It's not just launch and forget. Our source has also mentioned it really helps different people work together. Developers, product managers, maybe even domain experts. Definitely. Because you have this shared platform with shared prompts, clear evaluation reports, ways to give feedback. It breaks down silos. Everyone can see how the app is performing, contribute to improving the prompts, understand the trade-offs. It helps teams ship better, more reliable apps because everyone's more aligned. Okay, let's talk data. Building good LLM apps often needs good data. Does Lang Smith help with like data augmentation, finding or creating more data? It's more indirect, but yeah, it provides valuable support. It doesn't generate data itself, typically. But by evaluating your model thoroughly, it shows you where your existing data is weak. It helps you identify data gaps. So evaluation highlights the problems. Exactly. The model performs poorly on questions about X. Maybe we need more examples of X in our training or test data. Those insights then guide your data curation, or maybe your synthetic data generation efforts. Perhaps using Langshane itself to help create examples, you evaluate your way to better data requirements. Got it. And model evaluation itself seems like a real pillar of Lang Smith. It's absolutely a cornerstone. The diversity of evaluation criteria is key, built in metrics, custom code, natural language prompts for LLM as judge. The human in the loop part is critical for trust and nuance. It handles evaluating against labeled data, like test sets with known answers, but also unlabeled data where you're just checking for things like coherence or safety. And it can even evaluate agents, right? Not just simple chains. Yeah, that's super important. For complex agents, especially those built with Langgraph, you can evaluate not just the final answer, but the intermediate steps, the agent's reasoning trajectory. Wow, that's deep. It gives you much more insight into how the agent is working, not just what it produced. And they're adding more support for evaluating multimodal content, too. Okay. What about fine tuning, taking a base model and training it more on your specific data? Does Langshane's help with that? It does facilitate it. One common challenge in fine tuning is getting good data. Right. Langshane makes it easier to select high quality examples from your real world usage runs that got good scores or positive feedback. You can then easily export or prepare this data for fine tuning. Oh, like using the logs to create a fine tuning data set. Exactly. There are even specific helpers like a Langsmith Run chat loader for formatting data for open AI's fine-tuning API. It's streamlines using actual usage data to improve your model's performance on your specific tasks. It's really valuable to see concrete examples. The sources had some great industry use cases. Let's touch on a few. Finance. Yeah, in Finance, MUFG bank used Langshane to speed up research for their corporate sales teams. Done in Bradstreet built their chat DMV AI assistant using Langshane with Langsmith as the backbone for monitoring and testing quality. And there's Harmonic, a startup building AI agents for venture capital investing. They use Lenggraph for the complex agent logic and rely heavily on Langsmith for debugging those agents, running evaluations, and iterating on prompts. They specifically called out how crucial it was. Makes sense. What about health care? Reliability there is non-negotiable. Absolutely. Visient, a major health care improvement company, is using Langgraph and Langsmith to build a platform for querying patient care data reliably. Observability and managing prompts carefully are paramount for them. And handles sensitive data, HAA. Yeah. The enterprise plan for Langsmith offers high-p compliance, which is obviously essential for many health care applications. And customer service. That seems like a primary area for LLMs. Big time. Clarna, the e-commerce payment giant, built their customer service AI assistant using Langgraph and Langsmith. They saw much faster resolution times. Langsmith was key for testing and evaluating to ensure the agent was reliable and helpful. And minimal, another company in e-commerce support built a multi-agent system, again, using Langgraph and Langsmith, for testing and benchmarking different approaches, leading to big efficiency wins. These are just toy projects. They're real businesses getting value. OK, chapter five. Let's peek under the hood a bit. The technical architecture of Langsmith, and how it stacks up against other tools out there. Sure. Langsmith's architecture is built for this specific job. You've got the front end, which is the web UI we interact with. Then the back end core handles the API requests, processing logs, coordinating things. A key piece for reliability is the Langsmith Q. Q, why? It ingests all the trace data and feedback asynchronously. That means logging data to Langsmith doesn't slow down your actual application's response time. It just sends it off to the Q. Then there are standard platform services, authentication, infrastructure management. There's the playground service that connects out to the various LLMs you want to experiment with. And the data itself. Where does all that trace information live? It uses a few specialized databases. By default, clickhouse is used for the massive volume of trace data. It's really good at analytics on large data sets. PostgreSQL handles more traditional operational data, metadata, and retis is used for caching and managing the Q. Can you use your own databases if you self-host? Yeah, for production or self-hosted setups, you can configure it to use external instances of these. Oh, and there's also the arbitrary code execution, ACE, back end. ACE sounds serious. It's a secure sandbox for running user code, specifically those custom Python evaluators we talked about. Keeps things isolated and safe. The whole setup is designed to scale. They mention axiom handling really complex agent systems, and self-hosting gives enterprises full control. OK. Now the comparison. This space is crowded. How does Langsmith differentiate itself from, say, traditional MLOPS platforms like MLflow or Weights and Biases? Right. Tools like MLflow, WNB, CometML, our broader MLOPS platforms. They're adapting to LLMs, sure. But Langsmith was built for LLMs from day one. So more specialized. Much more. Deeper tracing into agent steps, dedicated prompt management workflows, LLM specific evaluation metrics, WNB might give you generic experiment tracking. But Langsmith's evaluation framework is tailored for language tasks. What about tools focused specifically on LLM observability? Like Helicone. Helicone is a strong open source option, often focused on ease of integration and cost monitoring. Langsmith, while having a free tier, generally offers a more comprehensive feature set, especially that tight, lengthened, length-graph integration and the very detailed testing and evaluation suite. Arise AI. They have Phoenix for Erdi evaluation, too, right? Yeah. Arise is strong in production monitoring, and Phoenix is great for RE evaluation during development. Langsmith aims for that unified platform across the entire lifecycle, development through production with particular strengths and chain tracing and its broad evaluation capabilities. Langfuse is another popular open source choice, strong on prompt management. OK, what about YLabs or Human Loop? YLabs focuses a lot on granular text quality metrics. Langsmith covers that, but also includes debugging, prompt engineering, and evaluates broader aspects like harmfulness. Human Loop often centers on the prompt engineering and feedback to fine-tuning loop. Langsmith integrates feedback, but within that wider observability and evaluation context. So many tools. What's a bottom line for Langfuse positioning? I think its main value props are one that deep seamless integration with Langchain, which is hugely popular. Two, the end-to-end lifecycle support dev test evil monitor. Three, the really robust and flexible evaluation specifically for LLMs and agents. And four, the flexible deployment options cloud or self-hosted. OK, chapter six, getting practical. What are the real advantages of using something like Langsmith versus just logging things yourself and using spreadsheets? Ha. Well, the advantages are pretty significant. You get a unified platform built for the job, instead of patching tools together. You get specialized LLM tooling that understands prompts, chains, agents. You get much better observability into these complex, often non-deterministic systems. You have integrated testing and observability. Your test run in the same system that monitors production. And it really does boost collaboration. Makes sense. But it's not magic. What are the potential challenges or things users need to think about? For sure. Cost is one. It's a service and high usage can add up, although there is a free tier to start. There's definitely a learning curve. It's a powerful platform with a lot of features to understand. Data privacy and security are huge. You're potentially logging user interactions, maybe sensitive data. So you need to be careful how you configure it and use features like masking. And defining truly effective custom evaluations can be tricky. It requires thought to get the criteria right. So what are the best practices to get started on the right foot? Good account and environment set up is key. Get your API key sorted. Configure the environment variables. Lanchin a Piki, Lanchin Crescent V2, maybe Lanchin project to organize. Think about your organizational structure. Use work spaces for different teams or projects. Use resource tags effectively to categorize runs, like by application or environment prod, staging, dev. Tags are better than work spaces for environments. Generally, yeah. Keep environments within a project workspace, but tag them. Manage your data sets systematically create them carefully. Use reference outputs for regression tests, leverage reporting. Develop a solid evaluation strategy, mix methods, built-in, custom, LLM's judge, human feedback. Think about how to evaluate complex agents, not just the final output, regularly audit your evaluators. Keep refining them. Exactly. Set up monitoring and alerting thoughtfully for production. Use dashboards, configure alerts for critical issues, but tune the thresholds to avoid alert fatigue. Be mindful of data retention and cost. Understand the policies, use the right tiers, maybe set usage limits, and always prioritize security use data masking. Consider self-hosting if needed. It's a structured approach. Any common pitfalls, especially for people trying to self-host it, that sounds complex. Yeah, self-hosting can have specific hiccups. Running out of disk space for click house is common you need to allocate enough. Database migration errors during upgrades can happen. Sometimes you need to force it. Hitting request size limits on the front end might require config changes. And click house permission issues sometimes need tweaking in its config files. And just general usage pitfalls. underestimating the sheer volume of traces an active app can generate. Not putting enough effort into setting up meaningful evaluations. Setting alerts thresholds too sensitively, leading to noise, forgetting about external dependencies your app relies on, checking the service logs and using the chain community resources are key for troubleshooting. And integration wise, it plays well with standard tools. Very well. Native Python and TypeScript SDKs. Open telemetry, hotel support is huge for integrating with other languages and broader observability systems. Seamless fit with the chain ecosystem core, length graph, length serve. And again, it's framework agnostic at its core. It can take data from anywhere. OK, last chapter, chapter seven. Looking ahead, security, ethics, and where this is all going. Let's start with security and privacy again. It's so critical. Absolutely. Langsmith takes it seriously. We mentioned data masking and anonymization. You can hide inputs, outputs via environment variables, use rule-based masking with rejects or PII detection libraries via create anonymizer, or even process data within your at traceable function before it gets logged. So you have control over what actually gets sent to Langsmith? Right. And self-hosting gives you maximum control if you need data to stay entirely within your network. In their cloud offering, data is on GCP. They state clearly they don't train on customer data and you own your data. Yep, GDPR, SOC2 type 2, and high PA for the enterprise plan. Plus secure access controls, RBAC API key management. OK. What about the ethical side? Bias detection, responsible AI development. Can Langsmith help there? It provides tools to support it. You can evaluate against safety and ethics criteria those built in harmfulness checks or custom ones for specific biases you're worried about. And for, for example, uses it to check for hallucinations and bias. The human review features are vital here for catching subtle issues. Evaluation results can guide your data set curation to improve fairness. The transparency from tracing can help understand where bias might be creeping in. Done in Bradstreet mentioned, using it for a "show your work" capability. Can you monitor for ethical problems in production? Yeah, you can monitor feedback scores related to safety or fairness and set up alerts for ethical regressions. But again, it's crucial to remember Lengsmith is a tool. The responsibility for building ethically lies with the developers and the organization. Makes sense. Okay, Crystal Balltime, future trends, what's happening in language modeling, and how does Lengsmith fit in? The fields moving incredibly fast. We're seeing trends towards smaller, more efficient models, more focus on real-time fact checking, using synthetic data for training, deeper enterprise integrations, more domain-specific models. Also, huge growth in multi-modal capabilities, more sophisticated autonomous agents, Gartner predicts they'll be major and continued intense focus on safety, alignment, bias mitigation, and security. How is Lengsmith keeping up? What's new or coming? They've been releasing updates constantly, things like enhanced evaluation, data set splitting, handling repetitions better, specific RG evaluators, better regression testing tools, more collaboration features like workspaces, tags, multi-annotator review, prompt playground enhancements, more enterprise features like RBAC, better off options, performance booths, official hotel integration, the Lengsmith alerts features relatively new, they hit general availability in early 2024. So they're actively developing, how does this position them for those future trends? Pretty well, I think. The playground supports diverse models. They're leaning into R-Agg support with debugging and evaluators. Data set management helps with customization and synthetic data workflows, enterprise features cater to that market. Crucially, the deep integration with Lengraft positions them well for the rise of autonomous agents. Support for multimodal evaluation is coming along, and the core focus on evaluation, human feedback, and monitoring directly addresses the need for enhancing safety, reliability, and mitigating bias. That hybrid approach automated evil plus human oversight seems essential. With Lengshane being so dominant as a framework, Lengsmith is kind of the go-to for productionizing those apps. So wrapping up this deep dive, it's really clear, Lengshane and Lengsmith are, well, a powerful duo. Lengshane to build, Lengsmith to ensure it's reliable. Exactly. Lengshane's the engine in chassis, Lengsmith is the dashboard, the diagnostics, the testing facility, the driver monitoring system. You really need both to build serious, dependable LLM applications. It's about moving past just playing with these models to actually building robust solutions you can trust. Right, bridging that gap from potential to production. And as these kinds of AI applications weave themselves deeper into our work and lives. Being able to understand them, test them, monitor them, it's not just nice to have, it's becoming absolutely essential. Think about how these tools, these principles of observability and evaluation might reshape AI development. What new challenges, what opportunities pop up as this tech gets even more common, it's a fascinating space to watch. Absolutely constant evolution. For everyone listening, if you want to get hands on, definitely check out the Lengshane and Lengsmith documentation and resources online. There's a lot there to explore. Thanks for joining us on this deep dive.