153. LLM Inference with Bedrock

If you're curious about building with LLMs, but you want to skip the hype and learn what it actually takes to get something working in the real world, this episode is for you. We have been building a lot of LLM-powered applications lately, both for ourselves and with customers, and I'm talking about workflow automations, smart data pipelines, query generators, AI-powered dashboards, that kind of stuff. And along the way, we picked up a pretty good collection of battle scars, as I'd say. So what we learn is that the art part is not getting a demo to work. The art part is actually making it reliable, predictable, affordable introduction. So that's what we are here to share with you today. And this trend isn't slowing down. We feel that almost every new project that comes through our door asks some kind of AI component that needs to be baked into it. So it's not really, should we use an LLM here anymore? It's more questions like which models do we pick? How do we call it? How do we make it trustworthy? How do we keep the bill under control? And if you're building on AWS, you know that a lot of these questions lead to Amazon Bedrock. So today we are sharing what we learned about running LLM inference on Bedrock, what works well, what surprises us, and the godjusted nobody wants you about until you find yourself debugging, hopefully not at 11 p.m. on a Friday. So today we'll see what is a quick definition of what an LLM is and what do we mean by inference, the kind of AI power application that we have been building, why AI is a lot more than just Gen AI and what we mean with the world agents. And finally we start to talk a little bit more in detail about Bedrock, what it is for, and the different godjusts that we learned about. My name is Luciano and I'm joined by Owen for another episode of AWS Bites. So maybe we can start this episode by giving a quick recap of what is an LLM and what do we mean by inference. I think that a lot of people might be familiar with some definition of those, but it's probably worth giving our own view on this. Let's start with LLM, large language model. We may know that this is the type of neural network trained on huge amounts of primarily text data. They learn statistical patterns in language and can generate remarkably coherent context to where text and the landscape of available models is growing really, really fast. You might know OpenAI's GPT family and THRAP XTLOD, Google's Gemini, Metaslama, Amazon's Nova. Then there's Mistral, Deepseek, Quen, GLM, and Minimax. The number of these really capable models keeps increasing and that might be great news for builders. So then we talk about inference quite a lot and that's when you use a trained model to generate some output. Training is really expensive. That's the long process of teaching the model. This is what model providers do and it's not cheap. We're talking about millions or billions of dollars in computer data and research. There's a reason all these companies need very deep pockets or very persuasive pitch decks. Inferences, what we do as users or developers generally, we send a prompt which is in push text and the model generates a response. And it may be an imperfect analogy to understand the business of adult and providers. Training is like spending years in medical school racking up enormous student debt. Inference is the doctors seeing patients and hopefully making that investment back one consultation at a time. We're the ones booking the appointments. Now the analogy breaks down a bit in a few interesting ways. And reality training in LLM takes weeks or months, not a decade of studies, but you do need a lot of GPUs. And unlike a doctor who specializes in one field, an LLM is more like getting degrees in medicine, law, engineering, creative writing, and it doesn't other fields all at once. So that's what makes them so versatile and why they're showing up in a lot of different applications. And then there's tokens. You might have heard a lot about tokens. We've covered it before, but LLM's don't think in words. They think in tokens. And a token is, you could say it's roughly about four characters, maybe three quarters of a word in English, something like that. That's just kind of a rule of thumb. And a detail that's easy to overlook is that different models tokenized text differently. So the same sentence might be 20 tokens in one model and 25 and another depending on how each model's tokenizer splits the text. This can matter more than you think because you pay for inference based on input tokens and output tokens. So when you're bearing pricing across models, cheaper per token doesn't necessarily mean cheaper for the same job if one model uses more tokens to represent the same text. But tokens are the fundamental unit of cost, rate limiting, and context windows in the LLM world. So it's good to get comfortable with this concept because it comes up everywhere. So, Luciano, given that all this stuff is everywhere now, what are we building? Yes, I guess first of all, it's worth clarifying because I think at this point, almost everyone assumes things like cursor, copilot, cloud code, codex, whatever, all this, coding AI agents. And I think it's worth just saying that this is the technology that we are talking about. LLM's are effectively what's power powering all these tools. And that's one type of use cases. So these are like coding agents that you used to whenever you're writing some code to get assistance from this intelligence and knowledge that's been baked in into the model. But we see a lot of use cases where we are embedding LLMs directly into applications. Applications we build either for ourselves while we experiment and learn more about this technology. But we see an increase in demand from customers. And we've been already building a few examples. And just to give you an idea, things we have been building are like smart data transformation pipelines. So you can imagine that there is an LLM component that helps the user to describe what they want in natural language. For example, I don't know, merge these two example datasets, normalize them in some kind of way, flag duplicates, for example. And then the system using the LLM is able to convert that natural language requirement into reproducible deterministic code that then can be baked into a pipeline that can be reused over time. So effectively it's almost like giving somebody doesn't necessarily know how to code that pipeline themselves. An easier door to basically be able to describe what they want with a process that somehow gives them a preview of the results, slowly converts to something that is actually doing what they want to achieve and then save that into reproducible code that can be reused later on. So imagine almost like a notebook, but rather than writing code, you use LLMs to get to the final version of the code that you want to use. And this is just one example. Another one that comes up a lot is for data analytics. Being able to generate queries, for example, for Athena is one we have done recently, but you can extend that idea to other databases like Redshift Postgres Elastic Search. And the idea is again, if you don't know the specific language that is required to query the data, you can use an LLM to convert some kind of human language of that query. For example, I want to know, I don't know, the top 10 spenders in these e-commerce, give me the user name. For example, an LLM should be able to do a good job in converting that to a specific query for another base, and then you can execute that query and give the results to the user. So again, it's always about trying to lower the barrier of entry to use specific technology. So you can use more natural language, and if the LLM do all the hard work of converting that language into some that is more specific for, I don't know, converting that in some way, making queries, but there are other use cases. For example, we can automatically generate dashboards. So for example, bases on some of these data pipelines that I mentioned before, another example we built is the system is capable of understanding the type of data using an LLM and picking up some of the metrics that are more relevant and creating dashboards with charts that may possibly the most sense. And again, it's not always perfect. I think there is a lot of the LLM can generally do an average work, and then you still want a human to maybe go in and refine, but we are seeing lots of use cases like these ones. And other ones we have seen online from just looking at other examples are, for example, customer support automation, where you can have a chatbot that helps the user to start asking questions and eventually route them maybe to an actual human agent that can help them to perform certain actions. Also, sometimes it's the LLM itself that can do certain actions on behalf of the user. And other ones are dogument processing. So imagine like an OCR process, but much smarter than that because a classic OCR will just give you the plain text. While when you combine OCR with an LLM, you can actually ask questions to documents and get out a structured data from your documents. So, yeah, I guess before we get carried away with all these examples, I think it's fair to say, to remark a list that these LLMs are not magic and they are not necessarily the right tool for everything. In general, I think LLMs are good to effectively convert this kind of requirement in some kind of a human language. So you can type something and try to describe what you want to achieve. And train the LLM to convert that requirement into specific options that will make sense for the system you are building. So basically, understanding text is one of the main superpowers of LLMs, but they are not good at everything that needs to be deterministic and precise because they are probabilistic by nature. So if you ask the same question twice, probably you will get a slightly different answer the second time. Sometimes these LLMs can hallucinate which is when they confidently state something that is not necessarily true and they cannot necessarily do what it medics very well. So sometimes they will do mistakes there, like the classic example is if you tell I don't know how many vocals that are in the word strawberry. Most of the time you might get a wrong response. This used to be one of the common jokes when LLMs came out. I think they are getting better but the point is that there are lots of things that LLMs are good at and many things that they are bad at so don't try to use them for everything, try to understand what is good about them, what is bad and then pick them only for the right use cases. And I think the key principle is try to use LLMs for everything that is a little bit fuzzy again as a human that is trying to describe something and you want to have death understanding of the description and do something but all the precise parts I think you should still try to to use code and more kind of regular automation to achieve those results in a more predictable way. And yeah I think in general the last point I want to make here is that it's going to become a standard building block as many others that we have been using throughout the years so it's always as any other building block to understand what are the patterns, what are the good things, what are the bad things, what are the common problems and hopefully today we are going to be able to cover some of that. Anything you want to add on this part of it? Yeah it's where it's definitely I think recommended to experiment frequently and not assume if you haven't already experimented frequently that you just just add a feature request and use LLMs in production for the first time and it'll be smooth sailing. I think we've seen statistics that the vast vast majority of these projects are not making it to production right now for a whole host of reasons maybe people don't have the right data, they didn't have the right use case in mind or it's the results just aren't effective enough to meet the use case that was envisaged like we find more success where the use case is very well defined and simple and it's a good idea I think in general to focus on areas where you're spending a lot of time that you might benefit from this level of automation but simple things rather than trying to assume that AI is so intelligent you can throw the most complex problem you have at it which usually ends in failure. We can maybe talk a bit about AI and Gen AI and what do we mean by agents as well? AI has become shorthand for Gen AI in popular conversation but of course AI is much broader and has a much longer history, traditional machine learning classification regression anomaly detection is still AI and still incredibly useful computer vision speech recognition recommendation engines they're all AI but not necessarily Gen AI and AWS has a whole ecosystem of services for more traditional AI angle like SageMaker recognition textract etc we won't cover any of those today but still worth knowing about it doesn't always have to be LLM based. So Gen AI is specifically about generating new content like text images code audio and video and when we say LLM inference in this episode we're talking specifically about Gen AI. Now inference in practice is generally at the simplest level text generation you send it a prompt and then you get an answer which we call a completion but increasingly we're also talking about agents and agentic workflows and these are really I suppose more sophisticated loops where the LLM can try and reason or simulate reasoning plan and take actions it's like orchestration of multiple steps of an LLM really. So what we mean when we talk about agents like an agent is like a smart loop rather than hoping that you get a good completion back the LLM receives a task decides what to do, uses tools, observes the results and iterates and this is one of the main things that makes agentic LLM so powerful the actual tools that you can use. With tools you can expand the LLM's capabilities far beyond just generating text. We know that generating text alone is subject to hallucinations and errors but by combining LLM's with access to deterministic tools it can actually become very powerful. So the LLM itself still just generates text it could describe what tool to call and with what parameters but then your code executes the actual tool and feeds back the results and you can write tools that do virtually anything like check the weather to give a location, look up a customer record in a database, call a third party API, read files, run code or just trigger complex workflows on behalf of a user and this is what turns an LLM from fancy auto complete into something that can actually take actions in the real world. With this power comes a lot of responsibility and safety boundaries are required that prevent the agent from doing things it shouldn't. So serious guardrails are required to stop it from executing destructive actions, leaking, personal information or going off topic. If you're doing it in a AWS world defining roles and very minimal permissions help a lot with this as well as your network boundaries. So when you have an agent as well there's also consideration for memory and context the agent would have to maintain state across steps building up context as it works through a problem. So you can think of it as LLM plus tools plus the loop plus context management plus guardrails put all those things together and you have an agent. In practice there is low in practice there loads of frameworks that help you build these patterns like lang chain strands from AWS the VSL AI SDK and there's plenty more in every language plus AWS has its own bedrock agents feature. We're not going to dive deep on the whole agentic agentic side of things today or on the AWS services specifically built to host and run agents at scale. That probably deserves its own episode but it is important to understand what we generally mean by agentic because it shapes how you think about inference. It's not just one prompt in one response out loops tools are called you've got context building and it all translates to more tokens more latency and more things to think about when you're setting up your infrastructure. With all of this context in mind let's talk about bedrock what it is and why it exists. Yeah exactly so I think I level bedrock we can think about it as the AWS managed service for accessing foundation models by API. So you can imagine it as a unified set of APIs for calling hundreds of models and there are many providers like we mentioned already some of them Amazon and Tropic made a mystery AI deep seek open AI and more and through this unified API you basically get a few interesting things. The first one is that you don't need separate accounts or API keys for each provider because bedrock is kind of your own centered place and it runs within the AWS ecosystem which means that you can also use IAM for authentication. You can use cloud watch for monitoring. You can use VBC endpoints if you want to keep everything as private as possible not have traffic going through the public internet. You can use cloud trail for all the things so all the nice and convenient things you generally use when you build production ready systems on AWS. Of course there is an alternative you are not forced to use bedrock like you could use the APIs of the different providers directly open API as its own API and Tropic as its own API pretty much every provider needs to give you access to the model when they offer the cloud version of the model through an API. So you can just go through them create an account and call their API directly and I think this is not too bad it probably works fine. If you're doing prototyping at small projects it might actually be a little bit simpler than just getting started with bedrock which probably comes with a little bit of extra complexity especially if you're not too familiar with AWS but I think then you need to know what you're missing out because I think you need to understand that if you want to go production ready probably what bedrock is giving you is worth it and it's worth the initial effort of learning bedrock and learning all the tools that you get with bedrock and just to give you a few examples you will get security and compliance because basically this is probably one of the main selling points especially if you're working in industries where it is important to to respect the privacy of the data of users and what bedrock guarantees you that data stays within your AWS account boundary you can pick specific regions where the inference runs so if you have also legal requirements where you need to make sure that your data never leaves a specific region like Europe for example you can do that through bedrock that is encrypted in transit and the rest they effectively there is an agreement between AWS and the model provider that they will never use data that you send to the models to do additional training in the future this is probably one of the biggest selling points for bedrock so effectively you can trust bedrock a little bit more than just having to go through the agreements that you will get with each individual provider which is probably going to be very different times and conditions so if you want to test for example both andropic and open AI models you probably need to go and retrieve the two different agreements and understand if they will work out for you while with bedrock you have a more unified experience once you understand the guarantees you have assist that allows you to try different models. Then we already mentioned governance because you can use IAM and CloudTrail. You have, again, model flexibility. I already mentioned that where if you want to try to see which model works best for you, once you are in bedrock, it's relatively easy to switch between the models that are available there. And yeah, then there are a bunch of interesting bedrock specific features, which I don't think we're gonna be spending a lot of time on it on them today, but you can easily build knowledge bases, sometimes called Rag. We already mentioned agents that are entire subservices within bedrock and frameworks that allow you to make it easy to build and run agents in production. You have the concept of guard rails, so being able to effectively limit some of the capabilities of the LLM, for example, if you want to make sure it doesn't go off on a path that you don't like, maybe I don't know. Classic examples are like limiting the LLM interaction to, for example, not be able to talk about politics or maybe not go outside a scope that maybe is the scope that is specific in the domain where you are implementing the LLM. Or maybe I don't know, you can remove some PII, so there are ways to detect that PII is coming into the conversation with the LLM, so you could obfuscate someone that PII before, it goes back to the user. So you have all these kind of additional features that I think are really important for when you're about to cut the production and you want to make sure you are ready for it. Now, there is one interesting caveat that I find a little bit disappointing sometimes, because although I said that there is support for hundreds of models, not all the mainstream models are out there. For example, a good example is Gemini, which is a very capable model, and it's not currently available in Bedrock. You can imagine this is due to competitive reasons because Gemini being from Google, of course, it's solved through Google Cloud, so I don't think it's very easy for AWS and Google to agree on a way to make that work on AWS as well. I know that it's currently the same for OpenAI with some of, like there are the GPT OS models available, but you don't get GPT, for example, 5.3 would be the kind of the bleeding edge model at the moment. That's not currently available in Bedrock. I suspect that that might change because I'm hearing that there is a big round of investment coming into OpenAI where Amazon is taking part, so maybe that will change soon enough. - They announced as part of that that the GPT models would become available in Bedrock, so that is the plan. Yeah, it only cost $50 billion, that was the price. - Exactly. So yeah, right now, just be aware that if you want to use Gemini, currently, that's not available is probably not gonna be available for a long time. While if you're interested in GPT models, they would probably become available very soon, but not just right now at the moment we are recording this. And again, just want to remark that there is so much to talk about when it comes to Bedrock. Today we're gonna focus just on trying to use the LLM programmatically part. Maybe we'll have future episodes if there is enough interest, and if we get to learn enough to make it work for us to create an entire episode dedicated to the other features. So with all of that introduction, how do we get started using Bedrock? - We talked about Bedrock maybe a little over a year ago, I think, and since then, there's a new access model. So the first thing you need to do is understand this a little bit. You'll find a lot of outdated articles out there, Bedrock used to have a model access page, where you had to manually enable each model in commercial regions that old workflow is gone. Today, access is mostly IAM plus one-time agreements for some models. So you'll want to follow the current documentation rather than anything from 2023. Models are now available by default in commercial regions as long as your IAM identity has the right permissions like Bedrock and Volk model. This brings Bedrock in line with how other AWS services work, which is nice. There are a couple of things that can trip you up, right? So some Bedrock serverless models are served from the AWS marketplace. The first time your account uses one of those, Bedrock automatically tries to create a marketplace subscription and you need IAM permissions for AWS marketplace in case that's something that trips you up. Note that models from Amazon DeepSeek, Mistral, Meta, Quinn, and OpenAI are not sold through the marketplace. So this only applies to certain providers. We'll talk more about this gotcha in a little while. Anthropic models specifically still require a one-time use case submission. You just fill out a little form and you can invoke them. You can complete this through the Bedrock playground in the console or using the API. And if you use AWS organizations, if you complete it at the management account level via API, it extends the approval to all organization accounts. Once that bit is done, you can pick a model and you've got the cloud models as we mentioned, which are very popular accent for decent complex reasoning, coding, non-documents. This is one of the ones we use the most, I think. And then you've got the Amazon's own Nova ones, which are the kind of the budget option, good for price performance, balancing, especially for simpler tasks. And the light and micro Nova ones are very cost effective. You have the Meta Lama ones, open weight models, good general purpose. Maybe starting to show its age, I don't see a lot of use of them. And then Mistral is good for coding and multi-lingual tasks. Grand and GLM are really starting to make an impact. I think in our opinion, there's lots of potential for those to be competitive in price. So that's just some examples. And as you mentioned, Luciano, you don't have all of the competitor models, but we can expect open AI's ones to become available at some point in the future, provided that agreement goes well. A good idea to start with a capable model like Claude Sannert to validate your approach, then see if a cheaper, faster model can handle it. No point in prematurely optimizing. And the bedrock web console offers a good UI that allows you to send messages to multiple LLMs at the same time so you can compare responses. Probably worth also mentioning that, as you can imagine, everybody's experimenting with bedrock and with LLMs. And as a result of that, it might be more difficult than you expect to get the quotas you might need. If you really start to run this production and need the scale, so prepare to have to make a business case and plead for quotas that are beyond prototype POC scale. Once you have your model, you can call the API. So we're talking about the invoke model or the Converse API for the more standard chat interface. We generally recommend using the Converse API. It's more of a unified interface across models with a consistent format, a bit like OpenAI's chat completions. And it's not all about text as well. You can use images and documents in the Converse API in the same message format. So multi-modal use cases work out of the box. And it supports streaming with Converse Stream for real time token by token output, which if you're doing chat is probably a must have. And you have the AWS SDK for doing this in your language of choice, Python, Bowdo3, JavaScript, TypeScript, Java, et cetera. These SDKs are generally split into two parts, like one for the control plane and one for the runtime. So if you look at the Bowdo3 option, Bedrock runtime is probably the one you'll be using more often. And the Bedrock one is just for control plane stuff, management of Bedrock models, that kind of thing. And the new thing as well, since the last one we talked about, Bedrock is cross-region inference. This lets AWS root your request to whichever region has availability and capacity. And it's a pretty big deal, 'cause I think this is the first time, but it used to be like a adage that you could say, if you wanted to do something in multi-region in AWS, you had to specifically configure it in each region and configure this synchronization. This is the first time where you've got pretty much seamless rooting from one region to another. And we can imagine that this is just down to the fact that GPU availability is scarce. So it makes sense to distribute it to whatever region has capacity. And the way you do that is by using a model ID. You've got a model ID, which might be like something like anthropic.cloud, sonnet4, dash version one, but across region inference profile ID is something you can use instead. And it will have a rooting prefix, like US dot or EU dot, or it could be a global rooting, like global dot. That'll give you a maximum throughput, but no geographic restriction. So it depends on your compliance, data retention, data residency requirements. New prefixes and profiles might be added over time. And at the SDK level, they're pretty much interchangeable. I think if you look in our, I think it's in our pod whisper where we use Bedrock, we talked about that in recent episodes. You could see in the committee history when we started using these inference profiles. Newer models, like certain cloud and lama versions only work through an inference profile. That's why we had to change it in ours and we'll return a validation exception with on demand throughput isn't supported. I think we were a bit confused when we saw that. for the first time. If you hit this error, just add the routing prefix to your model ID. And the IAM permissions are different as well, so you'll have to make sure you set that up. If you're thinking about observability and monitoring these, you'll want to know where your requests got rooted to. So you might check Cloud Trail, re-rooted requests, include an inference region field, and you can set up a CloudWatch metric filter on this to monitor your routing patterns. So I guess by default, if you're not too concerned, just default to using inference profile IDs, but for everything, there isn't really much of a downside. You just might want to think about the region you want to use and where you want your data to go. So given that, you're up and running, you can do inference. We talk about the cost and see if we can make it clear how much it might cost. Yes, it's actually not that difficult in time, so just the added matrix of it, because we talked already about tokens. Togans have generally classified in tokens in and tokens out or input an output. Input is what the prompt you send to the LLM, output is the completion that gets generated by the LLM. And interesting enough, those get different prices, like a price for inputs, sometimes as a million of tokens, sometimes in the thousand, I think it was actually changed recently that now in the pricing base, pages by the thousand, used to be by millions, which confused me when we were writing the notes for this episode. But yeah, it doesn't change at the end of the actual pricing. It's just a way of visualizing the cost unit for input and output. And each model is different, so make sure to check what is the cost for the specific model you want to use. Some models are more expensive than others, and generally like the bigger, more capable models are more expensive, but those are generally the ones that can be more reliable if you're doing complex tasks. So again, it might be worth starting with the more advanced ones, just to make sure you can refine the first implementation of what you want to try to achieve, refine your prompt and everything. When you have something that works, you can try to see if cheaper models can also end on that task as reliably as the more expensive model, and that's just a strategy to reduce cost. The interesting thing is that there is not from commitment, so as many other AWS services which just pay for what you use, which is nice, because if you have very occasional use cases, or maybe you don't know exactly how much you're going to be using an NLM power feature, that gives you an opportunity to grow as you go. There are a couple of tricks that you can use to reduce cost if you have specific use cases. One of these is patch inference. I honestly haven't tried it yet, but my understanding is that basically you can defer the execution of a bunch of LLM requests. So you have a little bit of higher latency in being able to get the response, but that comes with a 50% discount on the cost of input and output tokens. So if you don't have like a real time type of experience where a user is waiting for a response in line, maybe you're doing some kind of, I don't know, overnight batch processing, and you need to do maybe analyze lots of documents, whatever it is, probably you can use patch inference to bring the cost significantly down. Then there are service tiers, which also is not something I have really invested a lot of time into really experimenting with, but effectively there are different tiers of discounts and costs that you can use to try to bring your cost down, or maybe you can have one or three months commitments with reserved, and they will give you more guaranteed capacity for predictable workloads. So just make sure also to check the service tier in the pricing page to understand what that's all about, because it could be important for your use case. Then there are a few other things that can be relevant here. For example, there is a concept of prompt caching, which basically allows you to reduce the amount of tokens that gets sent to the LLM, so that's another way that can save money on the input tokens cost. So it's almost like the way I understand this is almost like you are saying, if I'm going to be running always the same prompt for a specific interaction, then you could get almost like a snapshot of that. So that's what gets cached, and then you are resuming from that session with maybe an additional piece of text. So you are not paying for all the initial input, which gets cached and it comes with a discount, and that way you are effectively avoiding to resend that text over and over to different prompts. Yeah, I think if you go into bedrock and you start to use all the other features, of course, they come with their own pricing, but again, today we are focusing more on the inference part. If you're interested in knowledge basis, understanding flows, agents, fine tuning, all these different features of bedrock, they will have their own pricing and different dimensions you need to consider. So go and check those out if this is something that interests you. So now I think we should try to quickly attach on some of the issues that might be shipping you up, whether you think going. Yeah, yeah, we touched on one already, which is throttling. There's quotas at two levels, requests per minute and tokens per minute, their per model per region per account, and new accounts get shockingly low default quotas like two or three requests per minute for some models, and that can be a real blocker. Even established accounts can have conservative defaults, they're not giving this stuff, who hey, they're greedy rationing it out. So you might get 429 throttling exception errors, maybe plan for it by adding exponential black back off with jitter, AWS SDK can do that using adaptive mode in its settings, use cross-region inference, disparate load, monitor with load watch, and apply increased requests early. Don't wait until you're ready to go to production. And the max tokens parameter in your requests affects throttling. This is an interesting nuance. Bedrock reserves some tokens based on your max tokens setting up front, even if the model generates fire fewer. So setting that to high can burn quota faster than you expect, because quota mathematics reserves based on what you asked for it, not what you got. And some models like Claude, output tokens count more heavily against your quotas with a burn down multiplier, like 5x for Claude. Now model access isn't as simple as it looks as another gotcha, even though it's simplified, as we try to say, you still need the right IM permissions, marketplace subscriptions, all of that stuff. Different models might be available in different regions, and some might only be available in US regions initially. The marketplace, we mentioned IM permissions are required in order to get model access because of those serverless models going through the marketplace. So you might hit that when you switch to a new model from a provider that you haven't used before, or to a new model you haven't used before, and your function or your service role doesn't have marketplace permissions. If you need to resolve this, and you don't need to give your lambda marketplace permissions permanently, instead just get somebody with the right permissions to invoke the model once to trigger the auto subscription. So you could do that as we said during using the bedrock playground. The marketplace subscriptions are per account, so you'll need to do this in each account as well. Now for the anthropic form we mentioned, which is separate from the marketplace subscription, if you complete that at the management account, that's enough, and you might have to wait 15 minutes then for the model to become available. There is a really weird error you can come across, access denied exception, model access is denied due to invalid payment instrument, and this happens because some bedrock models are delivered through AWS marketplace, and the subscription process requires a valid payment method, and it documents what payment method issues and geo restrictions can cause this. You could typically hit this when your account has a payment method that marketplace doesn't accept for subscriptions. We've seen this with European accounts using CEPA or CEPA direct debit system, some India-based AISPL accounts, and certain AMIA credit card configurations. Everything in your else in your account works fine because those services don't go through the marketplace, but the moment you try to use the marketplace, such as using a bedrock model that requires it, it'll fail. And to fix it, you generally have to add a credit card as a payment method. Some users report that they need to temporarily set the credit card as the default payment method, then complete the subscription, and then switch back. And then, after 50 minutes, fingers crossed, it works for you. Another point to mention is the converse versus invoke model, invoke model means you have to format the request body for each provider's structure. That's why we were recommended using the converse one because it's standardized for all of them. Okay, then we talk about structured outputs. I think this is where it gets really interesting in English. How can we take this really cool topic? Yeah, summarizer. Yeah, we are in the process of publishing our entire article that goes deep dive into this, so I'm just going to briefly mention what we're talking about and then we'll defer you to the article if you want to deep dive. But basically, one of the main problems that you face when you try to integrate an LLM into something programmatic is that the LLM generates text, but what you want is generally something more structured, like a JSON object that you can parse and then reuse into the rest of your code. But there are problems like you can tell the LLM to respond with a snippet in JSON and then the LLM might get a little bit creative sometimes. So sometimes it's just is going to use markdown fences where you are. I don't know, Bactic, Bactic, Jason, then all the Jason inside and then Bactic, Bactic, Bactic, and then you need to write code that can remove the Bactics and all the map down routing and just take the Jason and do a Jason parse. Sometimes even worse, happens more rarely with the more capable models, but I've still seen it. Like sometimes the Jason that you get is not perfectly compliant. Like you might get a trailing comma or you, or even worse, you might get fields that you didn't define initially just because the LLM is getting creative. So this is kind of a common problem. Oh yeah, there is another interesting use case where the LLM actually gives you multiple Jason snippets. So it's kind of reasoning and say, this was my first attempt, then I realized that this didn't apply. Oh, now there is another more refined version of the Jason you need. So your parsing code might get more and more complex as you find out about all this different variation of text that the LLM can generate. So structure output is the solution to this and it's basically a way to constrain the model to follow a specific Jason schema. So you can literally instrument the model interaction to say when you respond, you cannot derail from this schema. So try to populate this exact Jason schema and give it to me as a Jason object, don't generate any other text. So that basically gives you a much more reliable way to get answers that then you can use in your code reliably and avoid all the retries or random failures that might trip you up. Again, there are lots of details on how you can define the schema, how actually works in bedrock. And effectively you need to learn exactly how to define good schemas so that you get the best results will have bunch of tips in our upcoming article. So watch out the episode notes down here because we'll put the link there once it's available. So with that, I think we get to the end of this episode. I think today we learned quite a lot about LLMs, what inference is why you should be considering bedrock and in general, want to summarize that our take is that if you're building anything new with LLMs, I think bedrock is really a solid default choice for production inference, especially because you get all the guarantees from regional availability, more legal concerns in terms of data privacy ability to make sure that better your data is not going to be used for training. Which is generally a communist plus all the other additional services that come with bedrock that probably might want to start using as you get more and more familiar with LLMs and bedrock itself. Now, our usual call to action is if you use bedrock, do you like it? What you didn't like? Maybe you found other random issues that we haven't encountered yet. So please share them with us because that's how we learn just by keep sharing and talking with the rest of the community. So we always love that you find our connection details in the links as always so feel free to reach out on socials. One last word, thank you to 40 RM for powering yet another episode of AWS Bites. If you want to help building AI power and applications on AWS that are reliable, cost effective and production ready, make sure to check out 40 RM.com and reach out to us. So thank you very much and we'll see you in the next episode.