Go back

Let's Get NUTS!

46m 7s

Let's Get NUTS!

The transcription discusses the Heavy Networking podcast recording at the Autocon 4 conference in Austin, emphasizing network automation. The guests, Urz Bauman and Marco Martinez, maintain the Network Unit Testing System (NUTS), which aids in network unit testing through predefined tests and YAML configurations. Nornir is utilized for parallelizing tasks in NUTS test execution. The deployment of NUTS can be in a virtual environment, Docker container, or notebook, depending on the workflow. The discussion also delves into test bundles, predefined tests offered by NUTS for easy implementation. The importance of inventory, credentials, and plugins in NUTS test execution is highlighted, ensuring secure access to network devices.

Transcription

7693 Words, 41607 Characters

Welcome to Heavy Networking from the Packet Pushers podcast network. We're in Austin, Texas, recording from the Autocon 4 conference, which is all about network automation. If you don't know about Autocon, the Autocon series runs twice a year, once in North America and once in Europe. If you're deep in the weeds with network automation or you're just trying out a few Python scripts, this is a conference you should very definitely attend. There are hands-on workshops, practical stage talks, vendors with network automation products on display. Personally, I think one of the very best hallway tracks that there is. I am Ethan Banks along with Drew Conroy Murray. You can follow us on LinkedIn, Blue Sky and the Packet Pushers community Slack group. Our guests today are Urz Bauman and Marco Martinez, and they are maintainers for the Network Unit Testing System Open Source Project, also known as NUTS. And today, we're going to discuss Network Unit Testing from the NUTS perspective with Urz and Marco. So guys, I don't think either of you have been on the Packet Pushers podcast network before. Welcome to Heavy Networking, both Marco starting with you just very quickly. Who are you and what do you do? Well, thank you, thank you, thank you. I'm Marco Martinez, working for Swisscom. Usually, in my day job, I work as an architect. At my night job, I'm the maintainer of NUTS. Very good. And Urz? Yeah, so I also work at Swisscom at the moment. So funny coincidence. And it's my fourth, fifth autocon. And so I'm quite off heavy in network automation and try to be active in the community. You did a talk last time. I did two talks so far. And the third workshop now. Speaking of talking, you have a podcast to Network Auto Magic. You do with Stainzy, right? Yes, he's doing it. And I'm the sidekick. The sidekick to Stainzy? Oh, my goodness. Okay, there must be... Okay, I need to dial in and listen to what's going on with that. All right. Yeah, so network.omagic.net, I think is the website for that. If you've got even more network automation content in your life, Urz and Stainzy will put on a show for you. All right, guys. So unit testing, network unit testing. Most of the people that listen to the show are network engineers. They're not software developers. And so when you bring up the concept of "unit testing," that means something that I think we got to explain to folks here. So what is unit testing? That's a good question. So that is also kind of a philosophical question that I've been trying to define for years. Because in networking, we kind of have this issue where everything is interconnected and we cannot really test the smallest, fine-grainest unit. But in a sense, a unit test is the smallest piece of a moving thing that you can test. And then if you fit all these things together, you get a system that should be working. So what we're trying to do is doing this as a very fine granular level. We're not doing something like, "Oh, my internet is working, so we're just checking if maybe I can have a user on my device." Do you have anything to add for that, yours? Yeah. So if you have a checklist, that's normal. If you're at 4 a.m., making a change, right? You want to test all the small steps. And that is what we defined as a unit test to see if the interface up, is the port speed okay, is the BGP neighbor here, all these small things. So it's not really integration tests, but the small pieces. Small pieces. Okay. So give some more examples. You're talking about, you are talking about very small things like interface testing and BGP neighbor relationships. Give me some more. When does, maybe the way we do it would be, where does it fit into my work for like, when am I running these tests? So that's different. How can I say? We have different use cases. So normally what I would do is pre- and post-testing. So before I do a change, I would test to just be sure that the network is in this state I'm expecting to. So I think we all had troubleshooting sessions, looking for errors. And we find out this error is more than one week old and was not from our change. So if we do pre-testing, we can make sure all the BGP neighbors are here. OSPF interfaces are okay. I do the change. I most likely also need to change tests. And then I run them again. And I'm sure that I did not break anything. So if I was a network engineer, like when I've done changes before, I would have a checkpoint before I made the change. I want to validate that I have X number of OSPF neighbors and I have this many routes in the routing table, do the change, and then I have a follow-up. After the change, I'm expecting to see this many neighbors and this many routes in the routing table. Unit testing could be used to figure all that stuff out for me. Yes, exactly. So if you would do, let's say, immigration of a PECE, you normally go there, you check how many BGP neighbors are half, I do the change, and afterwards I want to have the same number of BGP neighbors. Yeah, maybe to add here is just a snapshot of your current system so you can make sure that everything is working as intended. And maybe you also write a unit test for the thing that is not working currently. So after you executed your change, you can test with the unit test if the thing you wanted to do is not working. So you don't really have to go backwards all the time, where you just have the snapshot of how it was. You can also model the future a little bit. So it sounds like you're not introducing really a new concept to network engineers, but you're just kind of describing what it is and maybe thinking about taking changes and breaking them down into their smallest parts to make sure each part works as you go along. Is that the idea or? Correct, yeah. It's like building Legos. You have these one break and maybe something in the middle is missing that we're not aware of and that's what we're trying to fix. I think it's also sometimes close to monitoring, right? So at some point the question is how much do I do in the unit testing and what is done in the monitoring system and they many times overlap. Would you distinguish unit testing from other kinds of testing I would do on a network? Yeah, more or less. There's definitely more end-to-end testing. We could also do end-to-end testing, of course. We can automate this. Normally we do more unit testing. They are safe to run. So like say throughput is difficult to do it automatically. I think you used the term integration testing earlier. That's a software development term. From a software developer perspective, we have unit testing and integration testing and so on. Is it worth talking about the distinctions in the different tests? Yeah, I think the problem we have in software engineering, a unit test is really, let's say, a function you can test. But on the network, we don't really have this Python function or whatever we can test, but we can test the smallest piece. Yeah, maybe you could say that's a property like an interface speed or a mode or whatever. Maybe that would be a good analogy. The integration test afterwards would be the more if the communication end-to-end is up, like get the client, ping the other client. Yeah, usually maybe over two systems or three systems. So we're not doing the entire chain, but parts of it, that's what I would describe as an integration test. The unit test is more this fine granular thing that sometimes it also works if we have a piece of the chain wrong, right? Maybe the speed is just reduced and we have like some performance issues because of that, but it's not really something we can grasp in the monitoring at first. And that's where Nuts comes in very handy. Okay, so you mentioned Nuts. What is Nuts? So Nuts is a network unit testing system. It's an open source project you guys are working on? Yes, it's open source. It's one of the best projects just because of the name. The possibilities are endless with the name, so congrats on that. That's always nice to talk about the Nuts. It's a PyTest plugin actually. So the idea is to have the ecosystem that we are used to from software engineering and PyTest is the default to standard testing framework for Python code. And it adds the capability to fit defined test cases. Many are predefined, but you can also develop your own ones and then you can write YAML files or you can generate the YAML files. Who are the actually test executions then? Oh, test definitions, I have to say. Yeah, here's important to understand. We have like these two layers that we make it easier for network engineers. We have this one layer where you define a test, where you really need to code it, what you do. And then there's only this like application layer, I would say, where you define what is being tested. So that one is really easy. We just have like a few lines of YAML. We give it the target and the test is opinionated and the test then does what you define in the YAML. So it's reusable. We have this reusability that makes it easy. What do you mean the test is opinionated? Well, we want to make it as easy for the end user to use it as we can. So we just ask maybe for the system for the interface name and the test does it does its own thing. Of course, there are some optional layers you can expose. So the test will do whatever you intend to. But it's really the thing is that you write the test in a way that it does exactly one thing. And you just need to define this on the application layer and you give it like as little input as it really needs to execute it. So am I using nuts then as kind of both a repository for and a framework for these tests that I'm developing? Like what where does nuts come into the picture if I just got these tests that I'm firing off? You can definitely have it in a repository. You can run it in your pipeline as a normal test. You could also run it in a crown job if you want to. But you most likely would integrate it to your automation workflow. So if you have an automated workflow, you define the new function, you also write the test for it. And you run pre and post tests. Make sure that the network is in the state you want to have it. So you can write tests, but you also said there are predefined tests. We have test classes predefined that the opinionated. So you can definitely use them, right? We use Napalm as one of the library who makes the abstraction. So that was the easiest low hanging fruit to get the most vendors upon. I was wondering about that. Yeah, because you're talking about defining a test in YAML, but it's like the way you would model an interface or BGP or whatever it is, varies from vendor to vendor. So you're using Napalm and you use Nornier as well. What world does Nornier play? Nornier is the parallelization for the task and the inventory. So the Nornier is a framework who is similar to Ansible, just pure Python. And we use Nornier for the execution. So if you run tests on free machine, you can run them in serial sequence, no problem. But if you run tests on, let's say, 60 devices, you will definitely feel that the test takes long. So what we do now for each test definition, let's say we have a test for interfaces. What this does is in the end, you use Napalm to make a get interface. And we use Nornier to run this get interface in parallel on all the machines. We get the data and then we aggregate the result. I guess here you could say we use Nornier as the workflow engine, more or less. So our test that we define in Python, we create the test itself, where we define what do we need to get, what do we need to compare, and which input that we require from the user for the test to be able to compare these things that you'd want to. And then Nornier just goes and does the thing. You mentioned pre and post checks. Can you talk about how Nuts works through those stages? So Nuts is pretty simple from the execution. You have the YAML files defining your tests, and PyTest finds the file, execute the test. So what you normally can do is organize your tests in different stages, in different folders, and then you can run some tests. So you defined with the tree from your file system, which tests you want to run. And you can generate reports, for example. But it's in a pre and post test, you would just run all the tests. You see that everyone is green, and you do the change, and you run them again. So I would say if we talk about software engineering, you have to test to make sure that if you fix a bug, not another bug is introduced, and you have some side effects. Yeah, I guess on this point, it kind of depends on how large your network is. If you have 200,000 tests, it might take a minute or two. But you might want just the feedback of I change something in DMZ. So maybe I run like a subset of these tests. First I do a full run, then I run the subset, which just checks the DMZ. I do the change in the DMZ. I run the subset of my tests, check that everything is working fine. If it fails already, then I can intervene, right? And then after that, if I know that the subset is successful, I can run the full tests again. So you can kind of find granular also, depending on what you need, right? Sometimes monitoring takes a little while, and we need to be aware of that. I think that's what I was getting at is after I've run the test, how do I know that it worked and that it had the intended outcome I wanted? Am I using other tools for that, like a monitoring tool or something else to make sure this change that I've made has actually done the thing I need it to do? I think I would use the pre-test with nuts just to have a fast, detailed view. Monitoring normally monitors more on a higher level, and the test itself goes more into the details. I guess here, if you know what you're changing and you want to make sure that it was changed, maybe you'll write a test for it before that. So first time it fails, second time it's successful, or if you don't have a test for it at all, you don't see it and just everything is still green. So those are kind of these two layers that you could do. Just a practical boring question, but I'm curious, how do I get my Python environment ready to run nuts? Do I have to be careful about what version of Python I'm running, plug-ins, libraries, that kind of thing? As normal with Python, dependencies can be a little pain, right? But the easiest thing would be generate the virtual environment, pip install nuts, and everything should be up and running. The pyftest version is pinned, and yeah, normally for each kind of framework tool, run your own virtual length, and you will be fine. Or maybe just run it in Docker. I'm a big fan of Docker, it kind of gets rid of some of the Python headaches. Well, as in build the Docker container, then I'm going to run Python, and that's got everything. You like that as opposed to a virtual environment? I do, yeah. We also provide a container, I think, or there's one on the nuts organization, so you can use that one if you want, depending on what you need. Maybe you need some other Python tools in your tests, so you wouldn't need to install them as well. So I've got Python that I've updated, and maybe I'm running it in a container, maybe I've got a virtual environment. Where should I, where's my execution environment? Where's nuts actually running from, if I'm doing it right? Not my laptop, I'm assuming. Could also be a notebook, no problem. In the end, it depends. What kind of workflow you have, right? If you have a CSD workflow, it's probably in the CSD pipeline. If it's a workflow engine, it's probably in the workflow somewhere, multiple stages. But you can definitely also run it on your notebook, like your normal automation, but that also depends on the company. I mean, circling back to the thing I said before, maybe the subset you run on your dev machine, on your local laptop, just to check if everything is still fine, and you have this more bigger workflow that you have somewhere on the server centrally. Also, a little bit in the networking space when we think about security is hard that you reach every system. So you need to ensure that you're running that from a point where you're able to access everything. And that might be hard from a laptop. Yeah, so it's going to be your network management network, your dedicated VLAN or VRF, wherever you're running that from, maybe over your out of band network. But then also, where other people in your organization can get at the test, you want to be working on it as a group, right? Yeah. I think it's a normal Git repository. And you work with Git, you can have versions, you can make releases. So the kind of software principles coming into the play, I would say. So you are intending it to be part of a larger workflow with multiple users, not just like this is my own little private tool that I'm using. I'm actually not sure if anyone uses it. If you're hearing this, please reach out to us. We're desperately looking for users. I think it's open source. I have no idea how many people using it, downloading it. The only thing we know is that we are the only one who create issues and fix them. So yeah, no idea if anyone uses it production or not, besides of us, would be interesting to know. Yeah, this is a call. Just create an issue and say I'm using it, if you don't have any other issues. Just hello, I'm here. So how do, is there anything I need to do in terms of giving it access to my network? How much of my network does it need to see? Do I have to preload IPs? That kind of stuff. Yes, you need to have this inventory from non-linear. So if you never heard about non-linear, it would help to have an understanding of non-linear, of course. So you defined your inventory, you have your IP address or your host name there. You still need credentials to log in, right? So depending on which backend and napalm or your test is using, you need SSH credentials or you need API tokens. So yeah, the classical napalm call is probably the thing that we have to think about. But you're pulling that from an existing, I don't have to do it again, that is, because I'm using Nornier. I've probably got those credentials. What am I doing? Am I sharing those with nuts? It is in the memory then of non-linear. Non-linear is executed by PyTest. So the credentials are in the memory. So you have to be aware that you should not just let everyone run it. That would be fun. Maybe the person who has the credential can then fix also the box. Now it's like Ansible, any alteration, somewhere the credentials need to be, right? Of course. So what we have is the inventory plugins. So Nornier has a couple of plugins. If you have not about networks, in-drop, all this kind of stuff, you can pull the data out. You can also write your own custom integration for a wall system. You can use SSH keys, for example. So it's kind of a lot of freedom. Makes it also complicated to find the right approach. I was digging through your documentation and I was on the front page, to be fair. Not digging through too deeply. But you've got a notion of something called a test bundle. Could you explain what a test bundle is? Yeah, a test bundle is a predefined test where we have some tests that we already have that you can use that are already set and fixed. You just need to write a YAML and you go from there. So this is what the test bundle notion is. Talking about network testing, the analogy of tests, we call everything a test. The definition of the test is a test. Then the test that's testing the test is the test. And also the test that we're doing on the network is a test. So everything is called test. So with the test bundle, we just wanted to distinguish that language and have a certain term for the predefined tests. Now the nice part is you can also write the test bundle yourself using Python or your favorite language. You just need to hook it into nuts and execute it from there. So naming is halt, right? Yeah. We're still testing on that. You said there's a bunch of predefined tests that test bundles that come with nuts. Hit the highlights. BGP neighbors, interfaces, rogue users, ping, trace route, the classical ones, especially the thing that is supported by Napalm. Because yeah, as we said, it's kind of opinionated. If you are used to other testing frameworks, where you, for example, that's a PyATS, where you can make a snapshot. If you make a snapshot of BGP, you get thousands of parameters, even the memory usage for each neighborhood. And if I want to test, that's probably not making sense. So we only test some really key properties, because it's kind of a trade between how many tests and so better make better tests and not too much. I see the balance. It's a core. It's something that's common that just about everybody will be interested in. Not everybody's going to be interested in knowing exactly how much RAM is consumed by a particular BGP neighbor relationship. Some people might care, but that doesn't make sense to put that in a test bundle. So the test bundle, let's describe it as core tests that the vast majority of people using the service are probably interested in. I would say so, yeah. So for example, we have one test with how many BGP neighbors are half. If I have a leaf spine, I know on each device how many neighbors I expect. It's a really simple test. We have this mechanism called grouping, where you can say on all the leaves, I expect two neighbors. And that's, let's say, five lines of YAML. And they will test if all the leaves have expected the number of neighbors. Yeah. Or you can maybe say, I have this template for a port and just test it on every access port. So we have the security measures on there. We have the speed. We have whatever you want to define on there we can put in there. And we just, there's three lines of code. It's like interface, the group and something else. And you can go from there. Also, maybe going back to the test bundle, it's also pretty important to see that our vision is not that we provide a testing framework for everybody. You should be able to write your own tests. You don't want to install a nuts that is like six gigs big or even bigger. So we just try to keep it to a minimum in the main repository. And there's also another repository that has like some kind of more tests that are not more specific. Can community members contribute tests? Absolutely. The idea was to have a collection with different technologies. We have our own organization on GitHub where the idea is to have multiple reports because also dependency is hard in Python, right? So we want to have test bundles organized by technology probably. So far, I think there are two or three people I'm aware of who have developed test bundles. See, somebody's using nuts. You've got at least two or three people. They're even giving back to the project. They're all my ego personalities. No, there was, there was one, one guy I know, but not sure if he still uses it or time flies fast. Are there particular classes of network device or network operating systems that it's limited to or like what's? If you write your own tests, there's no limit. Okay. Out of the box, though, can I work with Cisco, Arista, Juniper, etc. Yes. All the stuff that works with Napalm is supported quite well. Okay. In the test bundle, we also have in the name which kind of framework we're using. There are community versions. So Napalm and let's say the community driver from Nokia as Linux works well. There's also an online demo because it's easy to test with them as they are providing the containers, right? But yeah, in the end, you can write your bundles. And one thing maybe we should mention is because it's a Python test plugin, you can write all the stuff also in classical PyTest. So you can write your PyTest Python and combine it with the test bundles and you can do almost everything. What is your opinion? I gotta get this from you guys. What is your opinion on so-called digital twins, right? There's a lot of talking about this and digital twins come up in the context of testing, right? I'm going to run my test against the twin and if everything goes good, then I'm going to run in production. What do you think about that? What is your definition, digital twin? What do words mean? No, I think there are different kind of digital twins. My opinion, the first time I heard about digital twin, it was academic and it was kind of a mathematic graphs, this kind of thing. And nowadays, if people are talking about digital twin, it more means like I'm running a similar network virtual. Just a speck of a second. So you were talking about like mathematical graphs. That's what forward networks does, for example. That's their whole thing. They're building a mathematical graph of your network, you know, front to back and they're bringing in state information from the network that helps populate even more detail and they're going as hard as they can to build you a real digital twin. Something that exists off to the side, but it is as close to a mirror of your actual network and network state based on, I don't know if they call them a snapshot or what they do. I know they take some kind of a poll periodically to capture where the network is at. Run tests against that and in theory, that means something. That's relevant. But yeah, then other people that have like, oh, we got to have something that's a digital twin. And so they want to be able to talk about it and it's just, we stood up some virtual images of routers and plumbed them up and it's like, it's the same thing. It's not the same thing. No, it's not. It's good. I don't want to say you should not run your virtual lab and test everything before. I would personally not call it a digital twin. It is just a virtual test environment. Maybe the digital broader. Well, maybe it's a broader, yeah. Not a full twin. They share some DNA, but it's not an exact copy. Well, okay, so let's assume I've got some sort of a digital twin and again, what do words mean? Digital twin means different things depending on who's using those words. Does nuts fit into a workflow that involves a digital twin? Which one? Any of them would speak broadly. So if you would have a mathematical digital twin, you could write and test in Python to interact with it. If you want to test, let's say, some graph queries or whatever, you could do this. I've not seen it so far. And most likely if you have a complex system who provides this, they will also have some tests and already this capability, I assume. And virtual labs, definitely I see running tests first virtual and then physically. But still, yeah, there is no replacement for physical testing. If I wanted to get familiar with nuts before I brought it to production, could I try it against like a container lab or a net lab first? Can you operate in those environments? Yes, absolutely. So there is even on the nuts organization and GitHub, there is a demo container lab. You can spin it up in five minutes and you can run tests. I would definitely use kind of a workflow where you first test virtually. But what I'm saying is you cannot test 100% virtually. Yeah, yeah, just talking about the digital twin notion, depending on what flavor you've got, it's not that it's valueless just because your version of a digital twin is more or less container lab of your network. It's that no, you can't test all the things because your production network and your lab network are different things and they're not going to be exactly the same. And so your test may work out in the lab, but in production, something you weren't taking into account falls over on you. I think it's usually just a cost constraint, right? You cannot have a perfect digital twin. If you could have, it's usually too costly. There was some software I worked with years and years ago that the point of it was to create that digital twin. It was long before they used that term digital twin, but they would take all these different virtualized versions of what we were running in production that was supposed to create that environment. It was ungodly expensive and we had to dedicate a person just to keep that platform up and running. And we never did get it working well enough that it was worth either the spend or the headache to make it go. I'm hoping modern digital twins are actually better than that. Even what we do in container lab is a possibility now that is better than what we had a decade ago. Yeah, I think it definitely compliments each other, right? But I don't think in realistic time that we'll have a perfect digital twin or that we can test everything virtually and not care about production anymore. I think it will still have to coexist for quite a while. Why did you guys create this project? What is your hope or intent behind it? Oh, history. I believe it was 2016 or something. At the time I was working on a small network who was not really critical, but still you have to make some maintenance. And it was a Friday night at 4 a.m. We did too many changes like change the core and the firewall and everything because it was, yeah, it's not such a, it's not a hospital, right? If someone cannot work the next day, nobody dies. Yeah, it's okay. And of course it went sideways, absolutely sideways. And at some point we did usual ping in the morning. We did send an email and we were thinking it should work. And then next day or Monday, everything worked surprisingly. But then one week or in the week, the first IP phone went offline. Then there's the next one. So when three phones were offline, there was something. A release renewal or something goofy. Yes, exactly. So what happened was from, let's say, 20 SVIs on two, the IP address helper was gone. No idea how this could happen. But the IP address helper was gone. The phone had a quite long release, a lease time of DHCP, but at some point they ran out without a flight. And I'm thinking like, there must be a better way. So that was kind of thinking, we do software testing all the time. Why can we not have this in networking? And the good thing is, when you work with university, you have three labels. So you just take some students, tell them. Would you like to get your master's degree? Wow, I got a project for you. That's on the bachelor level. So our bachelor students are quite good. So yeah, we can use them as free laborers. Of course, we want to give something back to the community, right? So you're not selling anything. It's all open source. So I think it's worth the free labor and they learn a lot. So it started. And there was a couple of projects and of course, maintaining by myself. So I believe it's a win-win. If someone uses it, let's start the question. But I think it's an important part that we talk about testing and that we have the tools. I think recently I did a talk on it where we even generated some automatic testing, which was quite nice. You could, from the model or like from the source of truth, you generated all the yamls based on the properties my devices had. Like with AI, using an LLM to generate the yaml or something else? No, it was just a ginger template. So I was going through all the devices in the source of truth. They generated the yaml for me and I was able to more or less automatically test my network because of that with nuts, which was pretty nice. It gave me like this baseline of testing, which is kind of desirable, right? It gives you some value. Otherwise, it's hard. You start from scratch if you don't have any tests. If I hadn't guessed wrong, we could have gotten through the whole show without saying AI. Why did he go there? How mature in my automation practice do I need to be to start using nuts? Do you mention source of truth? Do I need to have a net box or whatever in place first? Or can I just take nuts and start just unit testing on my own? If you're comfortable writing yaml files or copy pasting from the doc, I think you can start. I would definitely recommend to start with a container lab or something. Of course, if you go to production or integration, I would probably make sure that the user only has read/write, so this kind of thing. But you can start small. You can make one or two tests, like just test the BGP neighbors. It's worth pointing out that you guys have documentation. Yes, and we are kind of friendly. If someone approaches us and asks questions, we normally reply in the same year. Yeah, I think also to get on this point, just do an iterative approach. Start somewhere. If you have any issue, you write a test for that, and then you go from there. You build up your baseline of testing as you go, as you have problems. You don't need to start with all the tests you need. You just start with a couple of them, and they will help you. They will generate value for you, and as you go, you get more value out of it, I guess. I asked because I'm curious about how people get into automation and decide this works and is valuable, and it seems like unit testing is kind of a nice, safe way to start introducing some essential concepts. I think it's better than a notepad of having all these IPs you want to ping, and then just copy-pasting it from your notepad. Just do the pie test run, or do the nuts run, or whatever, and it started. I've done that, where I've written the change, and part of the change is ping this, and here's the file that has all the commands you need to run in the expected output. It sucks because it's 2.30 in the morning. You're like, "I just need something to tell me if it's good or not. I don't want to have to think about this." Yeah, or even worse, do a for loop and bash, and then ping all these IPs in the variable, or whatever. Just write the YAML, do that before, and you can test as you go. Of course, you can do it iteratively if you haven't done all the changes yet, but you can still see some outputs are getting better or even worse. Yeah, I think many of us have done this tackle command for pinging from the box, a list of IPs. It's not fun. One good thing is, with pie test, you can generate reports. You can visualize it in a Grafana dashboard, so you can even use the data afterwards and show your manager that you need to change. Who had thought that the manager was breathing down their Naked3AM and was like, "Are we done yet? Can I go to sleep?" You're like, "Just check out the dashboard. It's fine. Nuts visualizes everything there. Let me do my work." No, you guys said you're friendly, but I'm assuming there's some kind of a Nuts community? Is there a Slack group or a Discord or something that, if I got questions, I can hop on in there and ask? If you have Google Nuts, you'll probably find some weird results. There is not a dedicated channel or something. There is, I think, in the infrastructure as code. There was once a channel Nuts. There would be GitHub with the discussion and issues. That's probably the easiest, just open an issue or a discussion. I would also say just open a GitHub issue or open a PR if you're this friendly or just ping us on LinkedIn. Are you guys on LinkedIn? Yes, LinkedIn. Or if you want to have something fixed faster, come to Switzerland and invite us for dinner and beer, and we can discuss. What other contributions are you looking for? You said people can write test bundles. They can contribute. You're talking about being able to interact on GitHub. Are you looking for people to contribute to the project in other ways? Always welcome. Of course, first would be too interesting to know if someone uses it and how they use it. You just want to hear from the community. You want to hear from people that are using it or giving it a try. We see downloads from the package. We have absolutely no idea if it's just someone messing with us. Do they be adding stars to the project? Is that a thing that you care about? We lost those. Every time we see the star more, it's kind of a boost. I think at the moment, we have around 150. I'm doing a workshop tomorrow, so I was opening the positive quite often. I believe it's 150 stars. Someone's using it. Yeah. I think also, if you're looking at it, just look at the documentation. If you don't get it, it's probably not a you problem. It's an hour problem. We're not doing a good job on the documentation. You can also point that out in the issues or PR, whichever way you're willing to contribute. In the testing engine, you need to have a little line of code that pings something in the cloud so you know how many tests are being run using nuts as it goes. You can watch that counter. No, no, no. We don't do this calling home stuff. No, no. I think it's nice if someone uses it. It's nice to give feedback. But in the other way, I'm running it in an air-gapped environment, so I don't want to call it anywhere. Actually, there are people trying it out, definitely, because if you would open the repository in the readme on the lowest section, there are some talks who not only from us are mentioning nuts. But yeah, it would be nice to know if the effort is worth, because one thing with open source is maintaining itself is a lot of work. And yeah, at the moment, if more people would use it and we would have more feedback, I would have more motivation to work on it. Well, Ethan wants to know when it's going to have AI, so can you get on that? No, I do not. Drew and I actually sympathize with the problem. So Drew and I have both been doing podcasting full time for a lot of years. We've each done, I don't know, at least a couple thousand podcasts. You know how often we hear back from people that they downloaded and listened to the show on a percentage basis? Almost never. We hear from people, but it's not that awful. If we mess up, we'll hear about it. Yeah, if we mess up, we'll hear about it, actually, yeah. So we understand, people use things and they don't interact. So yeah, give your feedback, everyone, especially if you're using nuts. Urs and Marco want to hear from you. All right, so in our show notes for this episode, we're going to have a lot of links. We got the GitHub page, documentation, how you can find Urs and Marco on LinkedIn, the network auto magic podcast that Urs co-hosts with Steinsi. Are Steinsi co-hosted with you? I don't know which it is. Yeah, either way. I think the airtime is 80% Steinsi and 4% me. For me, perfectly fine. He's doing all the heavy lifting and I'm just enjoying the ride. So Urs, if people want to find you, we've mentioned your LinkedIn page. Are there any other websites, books, anything else that you wish to mention? There is a blog post, a blog infrastructure as code.ch, but not very often writing something. Besides that, LinkedIn is definitely the best. I think you also have some open source lab on nuts, no? Yeah, there are some labs. So my Git repo or my Git profile is also something to look through. Same question for you Marco. Yeah, for me, just connect with me on LinkedIn and maybe share some feedback. I'm happy to interact with whoever and that's pretty much all. Thanks guys for coming on. I know you got a workshop tomorrow here at AutoCon and you had a little time today. Thanks for making the time to sit with Drew and I and chat here in this conference room where I don't know what you guys can hear, but there's someone doing jackhammering or something outside the window. It's kind of loud. Hopefully it doesn't come through. Thanks everyone for listening to Heavy Networking today. Yeah, connect with Erse Marco and the Nuts Project, download it, give it a try, let them know. And if you'd like to meet Erse and Marco in person, you might be able to do that. The next AutoCon, AutoCon 5 is going to be happening in Europe, dates and city to be announced, but my guess late May, early June, 2026 timeframe and you can sign up for the mailing list at networkautomation.forum and you will know as soon as AutoCon 5 is official. I have been Ethan Banks along with Drew Connery Murray. You can follow us on LinkedIn, LinkedIn especially. Drew and I are like supernodes. We're connecting to us, puts you one level away from all the other networking folks that we're connected to. That makes it easier for you to find fellow networking nerds and make your LinkedIn feed more useful. Join the Packet Pushers Community Slack at packetpushers.net/community. It's free and there's over 4,000 people in there, lots of channels covering lots of networking topics including automation and there's also a jobs channel where people post about open positions. You can buy some merch at storeduppacketpushers.net, like, share and subscribe. Check out packetpushers.net for a dozen other engineering-oriented podcasts that we offer to you for free with no login required along with newsletters and YouTube. You're an excellent human. We care about you. We hope the work we do here is helping you in your career and we hope you have it a fantastic week. Speaking of which, we are taking next week off for the Thanksgiving holiday here in the US and then we will be back, last but not least, Remember that too much networking would never be enough.

Podcast Summary

Key Points:

  1. Autocon 4 conference in Austin focuses on network automation.
  2. NUTS project maintains the Network Unit Testing System.
  3. NUTS helps in network unit testing with predefined tests and YAML configurations.
  4. Nornir is used for parallelization in NUTS tests execution.
  5. NUTS can be run in a virtual environment, Docker container, or notebook.

Summary:

The transcription discusses the Heavy Networking podcast recording at the Autocon 4 conference in Austin, emphasizing network automation. The guests, Urz Bauman and Marco Martinez, maintain the Network Unit Testing System (NUTS), which aids in network unit testing through predefined tests and YAML configurations. Nornir is utilized for parallelizing tasks in NUTS test execution.

The deployment of NUTS can be in a virtual environment, Docker container, or notebook, depending on the workflow. The discussion also delves into test bundles, predefined tests offered by NUTS for easy implementation. The importance of inventory, credentials, and plugins in NUTS test execution is highlighted, ensuring secure access to network devices.

FAQs

Unit testing in networking involves testing the smallest pieces of the network configuration to ensure functionality.

Unit testing can help network engineers validate changes, troubleshoot errors, and ensure network components are functioning as expected.

Nuts is a network unit testing system, an open-source project that uses PyTest to define and execute test cases for network configurations.

Nuts can be run in various environments such as a CI/CD pipeline, local notebook, or integrated into automation workflows.

Nuts requires credentials stored in the memory of Nornier, ensuring security by limiting access to authorized users.

A test bundle in Nuts refers to predefined test cases that are ready to use with minimal configuration, simplifying the testing process.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.