Elasticsearch Operations Done Right with Pulse

[Music] So hi, welcome to Tech Tuesday Unbox. It's a chance to unbox the technology and the people behind it. Today we'll be speaking to Ita Mar Sin Herchko, right? I'm saying it correctly. That is correct. Great. So Ita Mar, actually we have kind of an epic story in the valley about this client that we came to save his elastic search cluster. And we didn't know what to do. I think our CEO, our CEO, had this friend and they called him and he's saying like, "There's a kid, he came with a disk on key. He put it inside the server and five minutes afterwards. It was fixed." And I'm understanding that you're the guy, right? I wasn't a kid. I'm not that young for a long time now. How old are you? I'm 38. Oh. I mean, not that young. Thank you. I'll take that as a compliment. You look very young. Thank you. So, yeah. It was actually a couple of years ago. I didn't know it's kind of a myth over here. A myth, yeah. Good to know. Yeah, that was it. So I've been consulting on the elastic search, now open search as well for over 14 years, contributing to the code base, doing a lot of things, speaking consulting to customers, and sometimes customers call us up or call me down. But now my team as well, when there's, when she thinks the fan, essentially. So when something is broken and nobody knows how to fix it, and we come in and help, obviously some customers, we help them with architecting and designing and everything. That specific key was the case where, yeah, they tried a couple of things. They were in soft down time. So basically things worked and didn't work intermittently. Yes. For a couple of months, I think. And then at some point, I know that they got a referral to us. And then I came in, helped them. It was me and another team member that kind of for a couple of weeks or about a week or two. We kind of worked with them on monitoring, getting understood what's going on, until we found the smoking gun, then helped them fix it. And yeah, it was actually one of an interesting achievement. And it taught us quite a lot of things on how to operate, and how to respond to crises. And then obviously we had tons of those over the years after. So welcome, Mitama. And can you tell us what is your job title? Where did you come from? So I'm the founder and CTO of right now two companies, actually two bootstrapped businesses. One is in the consulting space, big data boutique. We are consulting customers with their big data architectures. We architect, build and maintain modern data platforms. And the second one is a product that we have spinned off from that business. And we basically learned during our years how to do things in a certain area, as we will speak about it today. And then we just build a product that kind of automates and does it much, much better. How did you get to this business? What pulled you inside? It's a mentality. It's a personality thing. You either are a bootstrap or type or you're not. I've been working with a software. Your serial bootstrap? I guess, and I think you need the third to be serial. So basically I've been consulting for many many years. I started working on search engines, text search engines. So Lucine, Solor, Elastic Search, many years ago. I worked on Lucine several years before. I started working with Elastic Search, which wasn't in the day. And that eventually led me to also work on big data platforms. That was way before there was big data. I was an expert with Lucine-based systems. I consulted to many companies, worked on several very large projects. And then as that business evolved, I had more people to the business. We became a bigger consulting company. And then I just found myself a bootstraper of a consulting company and now also a product. Can you tell us about a project moment that pushed you to your limits? That made you think like, OK, I passed that. How did you pass that one project moment? That's interesting question. There's plenty of challenges. So sometimes it's, you know, the challenge is to find a way to work with a given team. You see something and then you are sure that you know what the problem is. But they think it's something different and you kind of need to find a way to work with them. As a consultant essentially, to either prove or disprove your theory. And then you need to kind of, you know, get them to work with you to do that. That this could be one challenge and we are, you know, we have this all the time. Sometimes it's a technological challenge. So for example, one of our more significant customers, a very big logo that everyone knows. They have a very mission critical elastic search cluster and they are also very technical savvy. So, you know, they come to you, they trust you and then you need to still prove your value. Yeah. And then there is the. Because you're not, you know what you're talking about. They know what they're talking about and you need to meet them at that point. That you can say, OK, I know what's happening here, right? Or you have a theory, but it's sometimes based on a gut feeling. But you still need to, to look convinced, right? So, you know, consultants talk a lot about the imposter syndrome sometimes. And I don't think I have it so often, but sometimes you meet that right in those scenarios. You basically say, OK, there are technical slavs, savvy. I see what's going on. It's a phenomena I haven't seen before or it's just, you know, very niche or something. And I would still deliver value. Yeah. And we have those. But I think, you know, conviction is really the way to kind of. Inspire trust. Inspire trust. And, you know, kind of wreck through that. You basically use. You're saying, I'm going to trust mainstinks. I'm going to rely on my gut feeling that sometimes fail me, but sometimes proved right. And I also need to remember that it's OK to make mistakes even when you're a consultant. And they brought you and they paid you a lot of money and all of that. But still, you need to still show conviction in. There is some stuff that needs to be tried before you are actually succeeding. And it's OK not to succeed at first. OK, so this is a lot of problem solving skills that you have, right? How do you manage this? Does this come handy on your day-to-day life? Being a problem solver, is this like something that you practice also at home? That's funny that you asked that. So, yeah, basically, as a father to four kids, with the eldest being almost six now, and the youngest being one, and a surgeon wife that's not at home at 6am already, yes. That is serious. Yeah, there's a lot of balancing, right? So, you have any hobbies, anything that you keep or everything these days is on the business level and maintaining the home. Fred, I don't have time for hobbies. But I really enjoy the stuff that we do. So, I'm consulting less today. So, sometimes I do have engagements that I'm involved with, but now it's more about managing the company, managing the business, finding things to do things to improve. It's more problem solving and things to fix on the business, and I enjoy those things, right? There's also still technical challenges like building things. We're building, we're trying out a lot of things, even things that do not mature. Overall, it's very, very fun to me, but also kind of good use of my time. It's very good that you still have the challenges in the work, right? That you meet something that you don't, you didn't know before, no, it's growing to somewhere else. Let's talk a little bit about inspirations. Do you have any role models people do you look after? Who inspired you when you started to do this? I don't know if anyone's inspired you when I started. I think this is something that I learned along the ways to kind of try to listen and try to learn from other people. So, it may sound corny, but many, many years ago, before it was cool, I was actually trying to find a lot of information about Elon Musk and what he's been doing. Many, many years ago, he's really the kind of challenging and breakthrough kind of person. I think what really people that inspire me are people who are dealing with real difficulties and not, and kind of finding ways, even to fail, right? But still recover and breakthrough. Those are the challenges and things that I'm meeting on a day to day. I just saw a character yesterday, basically, on the amount of people who do things and succeed versus the amount of people just talking about it. It's different kinds of people and different kind of methodology, so not every successful person is also able to speak about their success. So finding someone and people who do, that's what I'm trying to find to actually learn from their experience. That's a great, that's a great thought. Not every person that is successful is able to talk about success. If someone wanted to follow in your footsteps, what would you tell him? How would you encourage him? Let's talk about bootstrapping maybe. So bootstrapping is really about personality, and it's a very lonely place. Because you don't have investors to help you. I personally don't have a partner, I'm doing this alone, but if you have a partner, that's great. You pick the right partner, and then it's kind of, you need to do everything and still do it within budget. Let's call it. So it's really about personality, the ability to kind of understand what you don't know, have the courage or stupidity to try things and be okay with failure. That took me a lot of time to understand. So I would say before everything else, just find out if that's something you want to do. I also have a lot of friends, I'm consulting with them a lot, with our VC backed. So they do have startups, one where another, some got acquired already, so I'm just getting the series A, series B, kind of the various initial stages. You've got a small group that you consult each other. Yeah. Yeah. It's more of a lot of individuals, not a group session, but I'm actually thinking of finding the way to do it in group sessions. Yeah. It's more, again, when there's good chemistry and good vibes that we can actually talk about those things. So it's really about personality. I do have those friends who, you know, they're also alone. So even if you are VC backed, you still need that kind of personality. I know I can tell you that I don't have this kind of personality to go and create a startup that is VC backed from day one, because then you kind of need to run at a certain pace, doing certain things that are not necessarily the same order that I'm doing in a bootstrap business. So it's a very different kind of mentality. How many workers in the Big Data boutique today? Big Data boutique is nearing 20 people. And what do they do? You manage it and what are the people in the Big Data boutique? I love the name. What do they do? Yeah. Someone already told me they need to trade market to make up a few of yours, but then we did. So most of them are consultants. Again, being a bootstrap business, we need to be very cautious on that. And also we are using a lot of the, you know, the company resources to try more things, right? And we build this product, you know, we call pools and actually succeeded before pools with a couple of pure empty peas that we had that didn't make the cut. We weren't successful on creating them as a business. Now we are creating this AI rug helper that's called Shraga, because it has ragnate and also that it's a Hebrew joke for Shraga. Shraga. But we are going to open source that. So we have this kind of, you need this kind of of margin to do those things. So the people in the company are actually kind of doing many things. It's sometimes not just the CEO, where's the multiple hats. Most of their technical, most of them are working with customers, consulting, doing a lot of hands-on work as well. Either on the company staff or on customers work. Our model is more of a consulting, less of a headcount company. So basically we are not taking, you know, three people putting them in a company and that's the business. Our business is more of an expert consultancy. So you know, we put several hours per week per employee for a customer needing that. And then the customer can say, "I got enough stuff done. I'm going to take this with my team." And then there's a lot of context switches. So sometimes we relax those context switches by giving like a bigger task. And if we don't have it for a customer, then we have this for ourselves within, internally within a company. So there is a lot of hat-changing in the company. So you know, the effort is doing a product for Paul. Finds ourselves doing things that are more related to marketing and so on. But basically most of them are taking, most of them are consultants and developers. So you mentioned Pulse a couple of times during our talk. We're going to be talking more about Pulse in our webinar on the Tech Tuesday. It's going to be on the 24th of December. And maybe you can tell us a little bit about it. What is Pulse? Who are the end users of it? So end users, we are basically targeting three personas. But let me answer the first question. What is Pulse? You mentioned that story that Devaly Paz, again, didn't know that. The myth of Itamal. The myth of the guy with the discount key. So essentially that's where Pulse really originated from. We found ourselves getting called into emergency response team for too many customers. We needed you yesterday, we need this now, those kind of situations. And then we lend the customer side. But then they have these crisis with elastic search. Everything is not working. This is a mission critical. The house is on fire, but they have no idea what's going on. And then you come in and say, OK, show me your monitoring. And they don't have monitoring. And even if they have, it's a graph on it that they pulled out of the marketplace dashboard marketplace, which is good. But it's not enough. And you still need to dig very deep. Sometimes monitoring doesn't exist or stopped working and nobody knew. So it's always kind of the same story. And the way that the company works, the big data would take what we do is basically we sell you a package of hours and then you can just consume those hours. And then we ended up spending 20, 50 hours just installing monitoring. And around the houses on fire. So what we ended up doing is building this managed monitoring for elastic search. We didn't have open search back in the day. So we ended up building a managed monitoring for elastic search. So all you have to do is just take this agent, install it on our premises on whatever on your cluster. And it will turn as it metrics to us within, I don't know, 15 minutes. We'll have some dashboards within the hour or two hours. We'll just start to see some patterns. We'll quickly find the bottleneck. We'll figure out what's going on. We'll start fixing a house and fire and actual incident is also always trial error. Try this, try that until something actually clicks or multiple because they're in with the elastic search. There is often no one single smoking gun. So you kind of try this and then we need a couple of hours. You're done and you can go home, right? Everything is fine. So we kind of sped up the whole process, just connect our tool and everything will be fine. Then we understood that we, since we have control of the agent, this agent that we've built because we needed a couple of things that are not delivered by, you know, metric bit and another tooling that's already in existence. And then because we have control of the metrics that we're getting, we actually have a very holistic and very deep view of the cluster. And then why wouldn't we just, you know, try to automate that kind of thing? If you connect the agent, deliver metrics to us, the platform will just tell you what you could do to improve things, even if the house is not on fire. So let's try to be proactive about it or let's just, you know, automate it. So pro activities is actually in another leg or two of the story. Before everything else, we have dashboards and alerting that we can tailor them specifically to Elasticsearch. So it's not just a graphite dashboard that, you know, kind of was built for Elasticsearch, but it's based on a bigger platform that actually does a lot of things. We build dashboards for Elasticsearch. And when you go and look at indices, we group them for you by index patterns. So you can see metrics and issues within index pattern, not specifically in index. We show you nodes based on tiers. We do a lot of those things that are very, very specific. We do short hit maps to find hotspots, those kind of things. That's dashboards. And then we started doing alerting. Now, you know, when you do alerting, you do threshold alerts. If the CPU is above 80%, if the disk is below 10%, but we can do things that are much smarter because we know Elasticsearch, we have all the controls. So we can tell you that when there is an acute disk imbalance, for example, on a specific tier that affects performance. We can tell you when the indexing is being rejected, and we can tell you why. It circuit breakers is a thread pool. is it known 200 because version conflicts or something? There's tons of stuff that we can do. So that's just the basics. But what we found out, so that's for the debuggers. But those are just the 10% of 20% of our customers. What we found out, and that's during my consulting years, is that I would say, 80%, 90% of elastic search users in the world, now open search as well, really near the elastic search and open search, they rely on it, sometimes on mission critical stuff. But elastic search and open search are so complicated pieces of software that you can really get good deep enough understanding of them to actually handle that. So what they would do is they would throw a lot of hardware onto that. So the customer would just grow and grow and grow. They wouldn't care until it actually hits a very large number. We usually see this on the $10,000, $15,000 a month. And you have customers paying a lot more than that. But then it actually starts to ache. And then they will start trying to optimize. Or when again, she hits a fan. Somebody did something or they didn't scale out or up fast enough. And then things will start breaking or not work as expected. So what we understood is that if we can get you the bottom line kind of, this is your current situation, this is what you need to do. Forget about everything else. Don't look at the dashboards. Just look at what we call the health assessment. We'll get you the health assessment. Well, let you know when it changes. And we'll explain to you under four pillars. Stability, security, performance, and resiliency. We'll tell you exactly what's failing. And then we will also explain how to fix that using actionable recommendations, sometimes even providing the actual commands to run. And that targets, I would say, 80% of people that we meet. They want the bottom line. They are usually the managers, so CTOs, the pair in these SRE team leads, so on. But not necessarily. I mean, it could be just the DevOps actually maintaining things. But now we can dedicate this time not doing the quality of an DBAing in the last research. But they can actually go into the more important stuff for the stuff that they enjoy more. And then also, those people who actually understand the last research very well. And then they enjoy a pool as some sort of a power tool. And again, they don't need to worry about it. And then they can manage also dozens of clusters at a time. So you're saying that essentially, you don't need to have a special-- yeah, you don't need to have a specialist on your team for elastic search. You can use Pulse. Our DevOps teams, they have a lot of things they know, but not always they can't focus on everything. So this could be the great tool for them to save their time, save money for the company, and being able to focus on the big stuff. And do root cause analysis much faster? That's awesome. I want to ask you, what kind of impact do you want to live through your work or your personal efforts? I always say that this is a barmaid for question. I don't know. I just enjoyed the day to day. Right now, we do have a lot of focus on family, we're building a family. So it's actually more on there. How can I make sure that I have only have girls? So my girls are actually growing up to be smart and happy. This is really, really a lot of my focus goes there. In terms of the business, I think this is what I'm doing on the day to day, kind of building things, making things in a better way. Like the Pulse, how can we improve SRE in DevOps lives and kind of automate a lot of things there? Same thing that we're trying to do with Shraga on Ragh yet to be discussed and showcased. I think that I'm not actually kind of aspiring to send people to Mars or anything. But is there like a technology that you're hyped about right now, anything that is coming towards us that you saw on your work? OK, this might change the world. I'm still trying to rub my head around Jenny. Everyone. I try not to ask specifically about AI, but always AI, yeah? Yes, I'm trying to rub my head around it. I'm not psyched as I'm trying to understand as much as I'm trying to understand exactly what's going on there. You use tools? The Jenny AI tools? Not personally. I sometimes find myself kind of falling thing posts. I'm kind of admitting here. So basically, take this blog post, summarize this from me, and then you don't have to do the hard leg work, but not extensively. I don't know, more of an orthodox kind of guy, as you can say. No. Yeah, I think we're still not fully understanding how it works, not how it works, but more of how we can use it the best way. And we're always talking about we had a great talk about it, about being more of a generalist. It helps you being more like you can spend in other directions and not the AI work. And so we build a lot of rags for customers now. We have a team that was built zero to-- we're now two people within six months. This is like, we grew very fast in our terms. We're building multiple rags for some customers. And then, yes, the best way to think about it is rags, just to make sure that everyone understands, is retrieval, augmented generation, basically a chatbot. So you have this kind of database or data set that you have could be millions of documents in your system, and you want to be able to let your customers ask questions and get answers based on that data, not based on what the LLM knows because it doesn't know. It just regenerates tokens that it learned before, but based on actual data. So it's actually a multi-step process that we build a generator lot upon. And the best way to explain to customers what to expect and best way to build rags the right way is to kind of understand how that end person would have do it himself if he searched for the content and then build the response and then rebuild that and explain to the LLM exactly what needs to happen. So it's a generalist, but it's also very stupid. It's someone who doesn't have brains, and a lot of people confuse it with that. So we hear a lot of stories about people saying, yeah, I just use it and it just answered correctly. OK, until the day that it doesn't. So that's that. And then I had this thought actually, this point about thing about it. There is so much content being generated using LLMs right now. So for example, blog posts and things like that. LLMs are trained or blog posts. Basically. Yeah, they're building the blog posts and then you use them to train the new blog posts and then you get it's not real blog posts. But I think we can still sense what is real and what is not. I have a-- If you know the topic, if you know the topic. So basically there were two items in the news this week. One was about-- I don't remember his name-- the Rothschild descendant. That I don't remember the name, we can look it up. But if you type his name on chat GPT-- You can type it. Yeah. --with refused to respond. Nobody understands. I don't know if it's a conspiracy theory or not. But that's one way of kind of-- OK, there is some knowledge control or bugs or whatever that you cannot read trust. That's one. And second, there is this-- it's a well-known fact now that chat GPT and all of those models are in a witch world. But some of the limbs were trained on open subtitles on the data set. And it's kind of-- it's by proxy, it invalidates the rights or infringes the rights of Hollywood, actors, and writers, and so on. So it's not immediately, but it's like by proxy or by-- so that is to say, think about it. If the elements are trained on open data-- so open subtitles, stock overflow, all of that. And on the rule of large numbers, basically out of all of these data, 80% of it would be correct. And you can expect the tokens to be generated to respond to generate a good response that will also be correct without hallucinations and so on. But think about what's going to happen in two years time. When they will train the elements again on newer blog posts that were written in the past two years, and 80% of them were generated. It's going to be some sort of an echo chamber of the same information that-- let's say it's correct, but it sometimes kind of loses the balance of real data and reininformation. It reminds me of deteriorating gene pool, something like that, right? Yeah. It's-- No pun intended. Yeah. So-- I think we still need to do a lot of thinking for ourselves. We still need to build more data pools. And I think that's part of the paradox I'm seeing like companies that's hiring the teams that tags the data, but they're actually destroying their own jobs while they're tagging the data. But what is going to be new data? That's the whole fight. It depends on the company, it depends on how good they are. But yeah, it's so much fun talking to you. I gotta say, and you have a lot of knowledge about this. This was one of the times someone explained how the GNI works, how the LLM works, and it was truly clear and it was very fun to hear. Let me ask you, on a personal level, I'm always asking, maybe you can give me a good recommendation of a TV show or a good book that you read lately. A movie? So I didn't read it yet, but I read a trilogy on the middle of the second book that there was actually a Netflix show. I don't think it did a lot of justice for the books. I didn't read all the books yet. I'm kind of very slow reader. It's called The Three Body Problem. And if you just watch Netflix series, go read the books. The books are amazing, are very flowing and kind of unexpected turns and all that. But the good thing about it is the same reason why I read the Martian and to end really fast in my terms, is because it's kind of real science that makes itself into science fiction and good storytelling. So The Three Body Problem is a really great book. And then I saw another recommendation on the book. It's a book from 1981, about a team that was tasked with doing something for a company, building a computer or building a machine. And then they didn't listen. They kind of went to Cowboy and build their own stuff. And then it was successful. So if they would follow the company's rules, it would have been so again, I really didn't read the book yet. That's the kind of headlines that I got from it. But you could say I'm kind of not mainstream books as much as possible. But that's that. That's great. We love to get good recommendations. You can't get the same recommendation every time. The Three Body Problems. Yeah. Nice. Itamar, thank you very much for joining us today on Tech Tuesday Unboxed. We'll be hosting Itamar on Tech Tuesday on December 24th. It's going to be three o'clock and we're going to talk about polls. We're going to talk about data and we're going to see some live working. Hope so. Always carry with the live demos. I don't know why. Thank you very much. Thank you for having me. It was great fun. And see you on the other Tuesday. Okay. Bye-bye. ( tomorrow呢 ) ( tomorrow again ) ( tomorrow, tomorrow, tomorrow ) ( tomorrow, tomorrow, Dının Nora ) ( tomorrow, tomorrow, tomorrow )