HN789: How a Global Payments Processor Automates Firewall Changes at Scale

As your network automation practice evolves, so should your tools. Don't manage your collections, advancedable playbooks and Python scripts artisidally, move up to "Itential" and load your automations onto a scalable platform your whole team can use. Visit itential.com to find out more. That's itential.com. - Welcome to "Heavy Networking" from the Packet Pusher's, we're recording live from AutoCon 3 and Prague, we're taking the opportunity to collect real life tales of network automation to inspire and inform you about what's possible. I'm Drew Connery Murray, here with Ethan Banks, our guests today are Dan Peachy and Vlad Sgooba from Adyen. We're gonna be talking about a firewall automation project they've been involved in. We're gonna get into how they went from bash-crypting to a self-service model for updating firewall policies. We're gonna talk about business drivers for the project, the tools they're using, version control and more. Dan and Vlad, welcome to the podcast and just get a start of brief introduction of who you are, what you do. - I'm Senior Network Automation Engineer. I've been with the company now for two years. I've been mainly focusing on automation for around the last six years. But previous to that, my background is more network engineering. - Okay. - I started network engineering around 2000. So yeah, I've had quite a long career. In network engineering and then switched more to automation. - Okay. - In the last few years. - I'm Vlad. I'm actually product manager. I'm with Adyen for two years, but I spent whole my career in telecommunication, mostly service provided by ground. And yeah, as a product manager, kind of make sure that network brings value to business if you can. - Absolutely. - Yeah. - Yeah. - So you guys work for a FinTech. What can you just tell about what that means and sort of what environment you're working in? - I would say the main product is payments. So we're kind of a middleman for payments. So we have a bunch of merchants and we provide to them payment services through terminals in shops, restaurants, but also online as well. There's a bunch of other financial products around that as well. We have a banking license in some countries as well. It's a global business. So we have offices all over the world. Our infrastructure is fairly global as well. Yeah, so and maybe one of the interesting things on infrastructure side is that we mostly run everything internally. So most of it's on prem. We like to own as much of the stack as possible. That's kind of one of our selling points to our customers as well. - And so I'm guessing in the major priorities are speed, availability and security. - Yes, definitely availability and security. Resilience, I mean if we're down, we're losing payments. Transactions fail. That's a big problem. So resilience is a big one. - Yeah, I can only add that probably resilience security, resiliency, security and then speeds. Because normally when you swipe your card, yeah, you want it to be fast, but it's too late. You're already paying. But if you're not available, you might not be able to buy food or buy a hot dog for your kid in a stadium or things like that. So yeah. - Yeah, I have some experience when doing payment card stuff too. And speed was it mattered, but like you said, it's like if you got to wait a few seconds for the transaction to be approved or denied, whatever. - Fine, you know, it is what it is. Plus there's not a lot of data on the wire to do a credit card swipe. It's not like you're moving gigabits of data through the land and the way and it's really small. - Yeah, I also not sure how much interest in it will be, but we actually connecting multiple schemas. There is not only Visa and MasterCards and CreditCards. There is a VPay local payments method in countries. So young is a middle man for all those and we have struck them in a single package for the merchants. - Yeah, okay, right. So as a middle man, you're enabling your merchants to accept all kinds of esoteric payment methods, right? Like not just Visa MasterCard. Do you do, yeah, you do American Expressor. Here my car's been working great. - Yeah. - I think right now there are maybe thousands of them, even. - Yeah. - Like VPay, Rupé, India. There are all kinds of payment methods. - Yeah. So Dan, you said you've been doing automation for a while as that brought you to this team and what brought you on board? - Yeah, so previous to my current role, I was working for another company doing a kind of similar project of what I've been working on now for this company. It was, I mean, in the past, I've like lots of network engineers. I've done my fair amount of kind of shell scripting and expects a little bit of power here and there. But it was around six years ago that I took up a project which was heavily Python based Django. Building a similar kind of thing to what we're working on now with the security framework that we're building. So I kind of have some experience in that from my previous project. So yeah, after that project ended, I found that I really enjoyed the automation side of things a bit more than the network engineering, at least for now. I'll see how it goes. But yeah, then I kind of decided to carry on with the network automation and took this role within Adyen as a fully dedicated automation engineer. - Now you said Python and then you said also Django came out of your mouth at the same time. But yeah, I don't think everybody's, most people are going to know Python, but maybe not Django. - Yeah, yeah. So Django is a Python based web framework. If people know Netbox, that's what Netbox is using. So-- - Web framework is in components that are going to help the presentation later, which are actually looking at on the screen? - Yeah, so it provides you kind of an extract abstraction on top of the database. So it gives you an easy way. So if you have, for example, a Postgres database behind it, you don't need to use SQL to communicate with it. Through Django, it has a ORM on top of it. So it gives you an easy way to query the data and put data into the database. It also allows you to build a whole kind of web UI framework around it as well, and an API. You don't necessarily need to use the web UI part of it, which actually in our project, we're not. We use a different web UI. But yeah, you can build an API in front of a database around it, and it makes it quite easy to do that. And what's nice about it is it has a lot of components already baked into it to do with like a role-based access control authentication, all that kind of stuff. So it's quite easy to get going with it. OK, you need to know some Python. Yeah, so yeah, you're not going to get away with-- it's not low code, right? But it's one of the easier frameworks to get going with, I think, at least. Everybody says it's easy, and then you start working out a real Azure, a monkey looking at a helicopter going, I don't know what's happening, right? Yeah, that's right. So can you tell us a little bit about this Firewall Automation project, sort of like what problem were you trying to solve or problems? One of the interesting things about the way we do firewall management is the any team-- or not any team. But many different teams can create firewall policies, not just the networking team, or the request don't just go to the networking team. So how it's been working in the past is that we had a bunch of scripts. So we were basically doing get-ops with a bunch of scripts. So people would submit their firewall rules in plain text into Git, raise a merge request, have that approved. And then the deployment part of that was very manual. You had to jump on a server, you had to run a bunch of scripts to actually deploy that to the servers and to the firewalls. One of the key problems there was the deployment wasn't automated. People wouldn't necessarily do their own deployments. They might wait for other people to do deployments. And that would mean when you come to do deployment, you might have other people's changes in your deployment that you're doing. And you have to go and find who actually made those changes to check that, hey, is this OK? Can I push this? And it basically slowed everyone down. So I think that was one of the pain points. So it was different teams wanting to make potentially different changes or conflicting changes to firewalls and also just a number of changes piling up. And you had to reconcile all that. Yeah, a lot of changes piling up, I think maybe we had 20 to 30 changes a day or more, even in most requests coming into a repository, something like that. So the change frequency was high. And it was producing a lot of issues with people actually having to push the changes. Right. And if those changes are manual, then there's also fat-finger and keystroke issues. Exactly. Yeah. Yeah. Well, why so high? Do you have a lot of firewalls out there? Because we're doing multiple levels of firewalling, this is also pushing to end hosts. So each host has basically their own firewall, mostly, IP tables, because it's mostly Linux hosts. OK, that makes a lot of sense in a hurry now. So opposed to a bunch of central firewalls that traffic's going through, which you also have, it sounds like, but also endpoint enforcement. Yeah. So basically, if you want a flow from A to B, you might have to apply at least to maybe three different places to actually make that flow work end to end. And are you on-- you mentioned IP tables. Are you-- what other firewall platforms are? Or are you in a multi-vendor environment? It's multi-vendor. Yeah. But I think-- so the idea behind the system that we built is that it's relatively generic, so that we abstract away the actual definition of the data into a kind of higher level model. And then we have a translation layer, which means that we can take that data that's defined and then translate it to any type of firewall policy that we like. So the first, the key one for us was IP tables, because that's what we have most of. But it can be applied to other firewall types as well. We have other vendors in the network as well. Well, there are micro-vacars. segmentation solutions out there that can do something similar. Why did you guys build your own thing in house? I think we did talk to some vendors, at least one vendor, which required you to have their vendor everywhere, which we don't. We have multi-vendors, so we have at least three vendors, probably on the network side. So yeah, that ruled out that solution. We did look around the market. We didn't find anything that would cover all of our use cases. But then basically we kind of wanted, at least for a part of our infrastructure, to have an agent where we could push out policy to end hosts. We didn't come across anything that would solve all of these challenges that we had. And we also had things around like gradual, canary deployments, gradual deployments, not wanting to blast out policy everywhere in one go. So when we coupled in all of these kind of use cases that we had, it was, we didn't find anything on the market. It's kind of sat as far of our requirements. And so we started to look at building our own custom thing. Yeah, just to add to this, I think there are really two key things, which we were looking and were not able to find is one, we wanted to remove complexity from the end user who put in the firewall rules. So you kind of need to understand what you're doing, and then you will need to get approval and review, etc. But you don't need to be a network engineer. And you don't need to know topology of the network to make sure that all thousands of your enforcement points are updated. And the second, there was a set of really niche requirements when we needed to have a gradual route. So we release in firewall policies like people who release an application. So we compile in them for the network. And then we roll in it out on one enforcement point, then waiting, doubling in and amount, and two, then waiting, then four, then waiting, etc. etc. Until your route is complete. And I don't think that anyone is actually doing it like this. At least we didn't manage to find it. But you write it generally, there are solutions on the market, which you can use as a network team. And that's what you led with. You did look for one before you decided, okay, we're going to own this, and we're going to build the team internally to do it. Yeah, okay. So Dan, it sounds like the team did it start with you, or did you come into an existing team? When I joined around just over two years ago, there was one other guy who'd been working in the engineering team, network engineering. I guess he'd been doing more of the automation stuff. And at that point, they decided to kind of make a small squad to focus on automation. So when I joined it was me, and one other guy in the team lead, and after maybe six months or a bit more, as we started to develop this whole new project, we realized that we needed to take on more people. The way we were looking at things, doing something very custom in-house, fully Python based. We had some of the skills, but it was kind of bigger than just the people that we had. So we started to look at taking on more, or someone at least from a software engineering, or more of a software engineering background, who could kind of take some of the more complex things that we were doing. So you felt like you needed a developer to be part of the team, essentially? Yes. I would say I've developed some good developer skills, but I still feel like I'm more on the network engineering side. I wouldn't really call myself a senior developer, yep. So I didn't feel like I could maybe handle all of this myself. And the other guy as well was probably in a similar situation to me. So I think we really needed some software engineering help. Can you dig into that a bit? Because part of that sounds like a posture syndrome. But then part of that sounds like, no, actually there's some specific things that I thought a senior developer could bring to the table that I didn't have experience with. Yeah, actually one thing to mention is that we were fortunate in some ways that it were a FinTech company, but the company is like at least half software developers. So we were able to lean on other parts of the company and we borrowed a guy for six months, maybe, a bit less to help us bootstrap the project. And he was coming from a pure developer background, very, very good Python developer. So I had come from working on a Django project previously. So I thought that I kind of, I knew quite well Django and Python, but then he came in and made some quite complex stuff, which was a bit above my head. And then I kind of realized, okay, this is, you know, this is a level above, maybe what I can handle and our team can handle right now. So we need some more expertise around this. Just to add to this, I think that one of the really business drivers behind it is it, we would like network to be embedded as a part of a platform for developers and other teams. And it's great when you build in uncivil playbooks, but when you start exposing complexity outside, you need to build models and abstractions, which is, which should be easy for people to consume. And that's where the problem is. That's where you need system thinking, how you need to step outside if networking just to enable this. And that's what we want, we want networking to be invisible. We don't want to be in a people's way. We want like you provision a server, network appears, switch port configures, but your job is have a server. You don't want to deal with the rest. Well, it's a very cloud approach. It's more how that feels. Like you're provisioning part of the network as a part of the larger IT service that you're deploying. And the network is just as you said, you want it to be invisible. Yeah, it needs to do the things. It's got to be, you know, a part of a VLAN or have an access list applied or whatever it is. And you don't want anyone to care that that just happened. It's exactly what it's worth trying to do. Yeah. Okay. And now our brief sponsor message from I TENCHEL. If you're like me, your network automation skill set is any evolving one. You've probably done some Ansible and Python work and you've had some quick wins. But now you're getting to that point where you want other people in your team to use the automations that you've built. They're sort of okay with Ansible, right? They're definitely not 100% but they're okay. However, they do not want to figure out Python and they can't even manage the library dependencies. Most of the folks on your team, what do they want? They want you to give them a thing that they can run and it just works. And if it doesn't, any network task requiring automation comes back to your queue. I call this the platform problem. You need a platform that the whole team can use to run automations without them having to know anything about maintaining a Python environment or even how to run an Ansible Playbook. And this is the service. I TENCHEL provides across their product portfolio. You've created a bunch of network automations. All right. Now move them from your laptop into a repository and then let I TENCHEL execute them in a scalable, secure way that abstracts details from people that don't need to know, well giving access to every nuance for those people that do need to know people like you. I TENCHEL put you on the road to self service network automation, allowing anyone you like to run your automation all while integrating with all the other systems that your organization runs. And to me, this is the ultimate destination for network automation systems, self service. To find out more about I TENCHEL's products and how they help you build an automation platform, visit itential.com. You can test drive itential free for 30 days. That is itential.com. So how does the service that you've built, it feels like on the one hand you've got developers that are interacting with the service to deploy something. On the other hand, you're talking about there's a web UI that people end users can consume the service this way and get firewall services deployed. So can you talk about how the service you've built is interacted with? Yeah. So I guess you can put it into two parts. So there's a backend part which is based on Python, Django, we're using the Django Ninja API framework. We heavily utilize PIDANTIK models. The door of the API modeling. What is the PIDANTIK model? So PIDANTIK is Python library. I think it's gaining quite some traction in the community at the moment for doing modeling and schemas. So we basically build our APIs around that. It makes it super nice to be easy to build them and it gives you all the validation data kind of integrity that kind of stuff. So that's the backend part of it. And we have, there's a product called Backstage which was developed by Spotify. It's kind of self-service portal type products where you can build plugins for it. So it's basically a web UI and it allows you to build plugins for it where you can make a UI and let your internal developers and use it as a self-service portal. So again, that part of the platform, we had to borrow some people from other parts of the company to help us bootstrap that. Yeah, so we were quite lucky in that sense. But once that's now built, we are mainly focusing on the back inside. So within our team, we're more focused on the kind of Python API back inside of things. I think we set up a few like ground rules before doing all this thing. So the system is API first. We built a UI for people to consume it, but this decision was made simply to make it easier to people to interact with because in firewall rules, there are many, too many relations, IP related to rules and source to destination, etc., in the UI, It's simpler to navigate when you have more than 100 rules. We have thousands of them. That's why, but everything people do in the UI is API-based. Then the next requirement was we should be able to roll back to any period of time, in any state. And that's when we ended up implementing our own Git-like on top of a database. And yeah, Gradual Reload was the third, like, cornerstone of the implementation. Yeah, we went through a kind of decision-making process around, should we do this in a database, right? The Git versus database thing. Yeah, we went through quite some kind of decisions on that. Some people don't like the UI, some people don't like not having kind of Git-based text files to do this. But yeah, in the end, we went with a database. What we really like is the fact we're the database when you have relational data. You get this kind of immediate feedback loop of whether you can define something as you want. So, you go and make a firewall rule, for example. You get drop-down boxes for the relational data to that firewall rule. You can only, you're constrained to only use the options in that drop-down, whereas if you have this based in Git, you might define something in one file, go and define it somewhere else, do a copy paste error. You're not going to find out if that's a problem until you commit it to the repo, have a pipeline run, and it fails because we have some error in it. With a database, you don't have that, you get that kind of immediate feedback. So for us, that was really a key thing, I think, that we can enable people to do this stuff super quick, because we can constrain them from not making as many errors as they might do if they're doing this in Git. Okay. And you're storing the firewall rules in a database? Yes, yeah. The abstracted tile-able files. The abstracted, yeah. Okay. So basically from that produce artifacts for the underlying files, but those can change based on the changes to the high-level data. But the elements that somebody who wants a firewall rule change implemented, they've got essentially a small set of fields that they can operate within to get point A to be connected, which gives you a little bit more control and reduces concerns about opening or closing things that shouldn't be opened or closed. Yeah, exactly, yeah. Yeah, probably the easiest way to think is as a compiler. Yeah. You put in high-level definitions and we compile to certain implementation and then we can embed or enhance what you put with, I don't know, if you want an APS or you want some inspections etc. Yeah. But that intelligence to take these sort of high-level inputs and turn that into the syntax for specific firewall languages, how did that middle layer, how did you develop that yourself, did you buy that, where did that come from? We developed that ourselves and I'm not going to lie, end up being very complex. I can't believe it. Yeah. Yeah. One of the really complex things is to understand what's changed in the system because you have this relational data and you might say out of prefix, because we have a hierarchical relational data where we can say, okay, well, this prefix can be a source and this destination can be a prefix as well. And we have to then figure out what IP address is a part of that prefix and therefore what hosts do we need to deploy these firewall rules on. So actually figuring out what the impact is of a change in the high-level data, what host does that impact on the low-end is really tricky. Yeah, it got quite complex. We make it work, we can manage it, but yeah, this is one of the most complex parts of it, I think. Okay. Graphs? Did you get into graph theory? No, yeah, but this is another thing, I guess. When we started this a bit less than two years ago, InfraHub was not on the market then. I think Nautilbot had done something with version control, with Dolt, but they seemed to have kind of stopped it. That box had done something experimental in terms of version control. So we basically wanted version control as one of the key things, plus a way to have this high-level abstraction and figure out from the change, what do we have to actually, what does it impact? I think at that time these things were not on the market, so we decided to completely build ourselves, boring some stuff from open source here and there. Two years later, things have changed. InfraHub could well be a good solution for this, because it has graph DB, right? I think it handles all of this impacts kind of logic within the product itself. So yeah, I don't know if we were to look at it again today, I think we have more options, but again we have a lot of custom stuff built on top of it, only for our use cases. Yeah. I expect that if we use something like InfraHub, we'd still need to build a lot of custom logic around it anyway. So the effort that you have to put into maintaining this custom piece of software that you've written, how does that compare to the original problem you were trying to solve in terms of work and capacity and ability to execute changes quickly and reliably? Yeah, I mean, I guess we have now a dedicated team of five people managing this, plus general network automation. But we are enabling people to, we are reducing a lot of toil, I think, around the company. So do you have a metric that you have internally to say whether this project is a successor or not? We do. We actually measure in what is called NPS score. So we're quaterally asking all our customers about are you happy with what we build? And actually in the month, I will be able to answer it. But one of the feedbacks I got just in a coffee corner is the guy who is a database engineer, he was like a lot. I recently, some application needed access to our databases. And like a week ago, I spent full working day just applying, firewall policies, etc. Now it takes me like an hour. So that's pretty good. So we'll multiply it for hundreds of people. It's a lot of time and money and actually people doing what they're supposed to do and not managing firewall rules. I think also the other thing to say about it is the previous solution had been there for maybe five or six years, maybe more. And not had much love, I guess. And yeah, it was kind of one of those solutions that works, but if something goes wrong, no one wants to touch it. And I was building up a lot of legacy technical debt. So yeah, if we'd have had to try and keep that going, we could have, but it would have just bit us even more eventually, I think. I actually think that we missed one part of the solution, which we actually built because our tool, it also discovered the network topology continuously. And we integrated with a few databases in the company, one of them puppet database. So we know which AP leaves everywhere and that's how we know where the tool needs to leave. Yeah, you mentioned there's an agent component as well. Yeah, so part of the solution is we have a Python based agent, which sits on the end server. And it's basically responsible for grabbing the rules, the new rules and deploying up to the host. And it's also hooked into Prometheus as well, so we can get a bunch of metrics from it. So we know things like did the last apply, what did it fail, or was it successful? Yeah, what version of the config is it on? Yeah, a bunch of different metrics like this. That gives us quite some control around the visibility that we have of this policy on the end host, which is also super nice. Although sometimes people react poorly to agents because it's also now much more software to manage and monitor and upgrade. How did you decide, yeah, it's still worth it? Yeah, I mean, we build some of our own packages internally anyway. So we had some of this infrastructure kind of existing that we can package things in RPM packages and push them out. We use puppet. So we have various kind of ways of automating this on the system side. So yeah, I guess because we had that already in the ground mostly, then we were fairly comfortable that we could probably do this. In the start, we thought this agent would be super simple. Over time, it grew a bit more in complexity, so maybe we didn't kind of forecast that first, but it has given us like way more visibility into what's going on, which is really good. I think we ended up being like an internal provider of software. So we have OSS, we're supporting, we have a way to release it. We're fully responsible for life cycle of this piece of software, testing, etc. That also comes with a lot of overhead. We basically have three versions of the tool. One is production, another is development and staging, and we're going over all these release life cycle because blast radios can be surprisingly impressive. So how do you guys deal with scale? You're a FinTech. The assumption is you're going to grow. You're going to bring in more merchants. You're going to bring on more backend networks to authorize against. And you're just going to have more volume over time and points to support and so on. Is this system designed with that in mind? Yeah. I mean, we can't say exactly how many hosts we're supporting now, but it's It's, yeah, I wasn't right. It's going to grow. The way we do it at the moment is we use internal data storage for the rules, internal S3 clusters, we have those in many different DCs. So we can scale that quite well. If we need to add way more hosts, we can just add more S3 kind of endpoints where we can store the artifacts. So I think in terms of scale, it should be pretty good. And the kind of control plane side of it, scale of that is basically the number of users. So at the moment, that's fairly static. We have a fairly kind of predefined number of users that are making rules there. So I think we can scale that fairly easily. It's also running in Kubernetes. So we can throw more working nodes at the end. Yeah, we can throw more replicas at it and it should be good. So probably if someone would be there to repeat this, the biggest constraint is the database. Because every time we have in a branch or revision set of the colonit, you have a copy of a database. And if you have 20 branches, it can grow, especially if database is big. So you need to pay attention to scaling that. I guess. But as soon as you can figure it out, yeah, it should scale. You mentioned a lot of backend components that you're dealing with, Python and Django and Backstage and so on. The some other pieces that I don't think we've talked about yet. One of them is Ansible, I think. We mentioned before we hit the record button. What are you doing with Ansible? Yeah, so I guess this project, you could kind of consider it a bit separate to the, although it's firewall policy. At the moment, it's mostly focused around Linux hosts. We'll include other firewalls at some point. We use Ansible heavily for network configuration. So we have many leaf spine fabrics in our DCs. And yeah, most of our network configuration is stored in YAML files in Git. We grab some stuff from networks, so we have inventory in networks and we're getting some of the data from there. That's kind of our next project, I guess, is like what do we do with this automation stack that we have, which has lots of YAML files. But this does feel is a completely separate from the firewall policy management. I would say it's pretty much completely separate. Okay, I think for now it is, yeah, but the problem we will need to figure out is, and actually, we're trying to how to deliver changes to as many implementations as we can. And one, yeah, well, I guess that's true. Ansible is now staying alone, but we'll need to start to merge in them. Well, yeah, you're describing dealing with multiple leaf spine fabrics and using Ansible and Netbox to deal with that. The firewall policy management component is like an application that would ride on top of that infrastructure, but they're not directly tied, at least in my mind, the way I'm thinking about it. But Vlad, you're nodding your head like, yeah, but we got this plan. Yeah, I'm just trying to, you know, say it in a way that it will be understandable. I guess Ansible for us is a way to deliver configuration, but in order to compile configuration, there are many things should happen. For example, one of the workflows we have is a server provisioning. And part of that is people putting their magic YAML to puppet, and they define in server configuration. So, if you don't want people to configure the switch, so we have a piece of software which looks at the puppet, you understand which wheel ends are there, what should I put. And for now, we basically have another demon who goes and configures the fabric. But I think long term, we don't want to be there. So we're trying to figure out a unified way of compiling changes and delivering them to the network. If you follow in me, what I mean. We don't want them to be atomic and chaotic, because then amount of dependencies, it like multiplies quickly. So for example, I have a villain on the port, but is a villain on EVP and fabric? How to like, am I missing communities? Am I, yeah. Yeah, I guess we have, at the moment, we have various use cases for our kind of internal customers, for example, requesting a network on the VLAN and deploying that to the network. That's quite manual at the moment, because it would usually come in as a ticket and then someone would update the net box and then there'd be certain updates to YAML files and gear and then deployed through Ansible. So I think the thing we're kind of grappling with next is how could we automate this end to end? Either completely automated from the customer just requesting something like service. So sure, yeah. Possibly with one step in the middle where someone just hits a button, says, yeah, prove that's good. Yeah, so there's kind of BAU changes, low risk, that right now are quite manual. I would say that's probably our next thing that we need to look at, try and reduce the toil there. And then if you will do one step ahead of that is you have to give you an example and this is a really visionary for us, but think about automatic scaling in AWS. So for this to happen, you need to orchestrate firewall rules, switch configuration, provision of server, et cetera, et cetera. And that's what we're aiming. We want all these pieces to work together so you can have a platform experience. But again, you're going to build it in-house, you think, and you're not looking at a vendor that could, because again, there's a bunch of vendor solutions in that space. Some that are multi-vendor and that's what they're designed for. So you can have a service catalog that you're deploying out to your customers, folks like a new to networks come to mind, they've been doing that for a long time and others. But you're nodding, you're like, yeah, where were those? It's just a mirror of conversations we have. I think we will end up with some vendor solutions. Because simply to build this all in-house, it's really complex. Yeah. I think it's feasible for the even company of our size. There will be some custom built pieces simply because there is a certain business logic, certain way of doing things, which we would like to be like we doing it. Certain business logic, I guess, that's how it's called. Well, it's the same. Everybody's got this story where there's this commonality that all networks have. They're all doing VLANs and IP and VRFs and whatever they are. There's these core functions. But because everyone's network has unique business requirements, there is no out-of-the-box solution you just go by. No matter what, even if you invest in a platform from a vendor, you're going to put a significant amount of time into customizing it for your specific requirements that you have to deliver for your environment. So the choice becomes, I'm building something internally. How much of it am I going to build internally versus how much of it can I outsource and lean into the vendor to supply a bunch of bits and foundation for me that I don't want to have to deal with myself. And the answer is what's right for you, it depends. It seems like it depends massively. It is. I also think that the difference for us is we're not greenfield. We already have pieces of what we developed. Yes, they are a little bit disconnected and they solve in niche problems, but we already have them and they work in really good for us. I think for me, actually, that's what makes this space so interesting at the moment, network automation, because I think I see outside of the hyperscalus, you have a lot of this figured out right, the rest of us are trying to build these frameworks. We all have our own use cases and our own business requirements. Some of them are common, some of them not so much. So people are trying to put together these kind of frameworks that can be parts of vendor, solutions, parts, open source, custom built. There's no set rule around how to do this, which on the one hand is a bit frustrating sometimes. On the other hand, it is interesting because we have all these problems to figure out. I see people working together to figure these things out and come with good solutions. I see in the automation forum that people are trying to figure out a reference framework as well. Q about a few paid much attention to that development, if you had any thoughts? I have one, to be honest, other than dipping into the Slack channel now and again, I've not been following it. I'm looking forward to them coming with a nice recommendation at the end of it. You're seeing the need of the investor interested in the now. I'm interested in it. Yeah, at some point maybe I can have some input to it, but up to now I'm just kind of loosely following it. I think it's a great thing that they're trying to do. I've been on a lot of those meetings and just getting definitions agreed upon has been very difficult. It is a work in progress, but I think it's valuable work and those conversations need to be had. I think probably many businesses are having similar conversations to us around this that are trying to do network automation more than just some scripts and answerable, trying to take it that kind of step further and really build a framework around it. I guess maybe it's sort of two splitting hairs, but where does network automation stop and sort of full software development begin because it sounds like you've kind of blurred that, frankly, you've just built a software product. Yeah, I think. >> That's the some line, I guess, at some point where you're having to really build software, you're not just scripting anymore, you're really building software and you need to have all of these kind of software development processes around it to have a maintainable code base. And then I think you really benefit from someone who has this software engineering background, who's built these things from scratch that can come in and lay the groundwork for it and kind of maybe help the rest of the other guys in the team have come from a more network engineering background. It's great to have these people there that you can learn from as well, that they can teach you how, okay, this is actually how you develop software. This is how you do things properly, maintainable. Yeah, I think that helps a lot. If you're mostly kind of maintaining individual scripts that do certain things, maybe you don't need that. Once you start building things like, for example, a big Django project, internal project, then probably you benefit highly from some software engineering. >> I'm curious, you have a foot sort of in both words of development world and the networking world and it seems like a lot of the messaging to my perception is that network engineers need to take on more of development skills. Do you see anything that needs to go the other way, that we know and networking that could be beneficial to the folks on the development side? >> So I guess in my previous projects, I was in an interesting position where I was developing a similar kind of project based around Django and Python. But I was doing some of the development myself, but I was more managing a small team of developers that were outsourced into us. So we had three developers from a separate outsource company. And I was really kind of bridging the gap between network engineering and software engineering and translating to them. Mostly in pseudo code a lot of the time. Please make this and just giving them some kind of guardrails around what to do. And it worked fine, actually. They didn't really understand much about networking at all. And I think after the three years of the project, they probably didn't understand much more about it. But I don't know. It's hard to say. It really depends what you're working on. Where you have this purely kind of software project, the project in the end does something on the network. I'm not really sure software engineers. Software engineers really need to understand heavily about it. >> But it needs to be made consumable for them with guardrails. >> Exactly. In talking to various network operating system vendors and so on, we've discovered that there's a lot of software engineers that work for Cisco and Juniper and Arista. They don't necessarily know much about networking as such. Many of them do, but a lot of them don't, which really shocked me when I first realized that. I started thinking about it. It's like, well, no, they're delivering a software product. They know code and how to deliver that in a solid way and they don't necessarily need to understand. Or they understand networking from a very different point of view. Because they're worried about the minutia of a routing protocol, for example, how to deliver that code. But they don't have the big picture of how that's going to be deployed on a massive network for a service of writer or a fintech or an enterprise. And so they just think about it in ways that you're like, we're not talking the same language, but we are. It's so weird. There's this disconnect sometimes. But it's okay. >> At the end of the day, you end up with a workable product. >> Yeah. I think it's probably, depending on what you're working on, it's more, you have to get the team kind of, the skills within the team, correct, right? I don't think you necessarily have to have someone that has network engineering background and then is kind of shifted into software engineering. You can have a software engineer working with someone with a network engineering background that just understands how to work with the software engineer and describe what you need and make the requirements clear, this kind of stuff, right? So you-- >> And that's it. If you can learn to speak each other's language, that's very helpful. But right, there's no reason that you as a network engineer need to deeply understand the software engineering domain. But it helps, you know, the more link, more you can speak the lingo, the more it helps, and vice versa, as that software engineer understands. Especially if they're dedicated to your team from a net ops perspective and can speak the language of network engineering. It helps, but isn't strictly required. You just need to be able to explain to each other what's going on and how to make things work. >> Yeah, I also think like, our position is maybe not very common that we're doing this fully kind of custom in-house thing. And we really need some software engineer experience. I think that work engineers with some good scripting and software experience can go a long way, right? You can do a lot of the things that we usually need to do on the network in terms of automation now, not necessarily with software engineering skills in the team. But when you kind of take that step further and you fully go with something custom in-house, then at some point I think you're going to find you need someone with that kind of background. >> It's funny you were talking earlier, how you brought software engineers who really understood-- you know, I know Python and Django, then you worked with the software engineer that really, really knew it. You're like, oh yeah, I'll go. >> Right, yeah, we can all write a script and so on. But if it came, I need an adult in the room that knows how to manage the code and deal with, you know, bridging control and all that stuff. >> I think I saw that from my previous project actually that I was-- in that I was actually doing less of the development, the actual Python code. I was more, let's say, higher level and describing to the software engineers what to do. So I learned from them how they do things and I saw what I'd been doing, which was mostly scripting up to that point, was quite different to what they were doing. And yeah, so I think at that point I had the realization that, okay, this is, you know, someone who's worked as a software engineer purely for 10 years, you know, as a senior, I'm not that level. So I could become that level maybe if I do 10 years pure software engineering, then yeah, but I was not at that point at that level. But on another hand, I think what networking can learn from having and working with software engineer is, yeah, how to organize your code, the best practices. We, like, you will be surprised how small team, how far it can get these days without being a software developer, where we really benefited from having really, like, one guy at the beginning is how to make it scalable, you ask the question, right? That's where you really need someone to think in advance how the thing should look like if it's barely complex. Because otherwise you will hit the wall at some moment and it's going to be very painful. Right, I assume there's probably many more hours of conversation here, but we do need to wrap. So if folks want to find out more from you about the project or just connect with you online, are you on LinkedIn, are you social, Dan, where can people find you? Yeah, I'm on LinkedIn, Dan Pichy, or I'm also on this, there were a coordination forum Slack group as well. Okay. Deep pitch. So deep pitch. Okay. And work me there. All right. Vlad, how about you? This I am, I'm most likely active on LinkedIn. I think I left majority of other ways of communicating and Slack, yeah, you can find me there after I come. Okay, great. Well, thank you, Dan, and thank you, Vlad, for joining us and telling us about this project and hopefully giving people some ideas about things they could be doing. And thanks to you for listening for today's heavy networking. If you have topics you want us to cover or comments about this episode, you can send them to us at packupusures.net/followup and we really do appreciate it and we try to respond. And please do all the other things we have merch at store.packipusures.net. You can like, comment, subscribe, tell your friends. There's the network automation forum Slack group. There's also the packupusures Slack group. That's at packupusures.net/community. We also have a growing YouTube channel. You can watch our podcasts. If you want to do that, you can also see Ethan, Labbing and trying stuff and doing things in real time. You can follow us on LinkedIn, all those things. Help us keep going to do what we need to do in this world of fractured attention. Last but not least, remember that. Too much networking would never be enough.