The transcription discusses the Heavy Networking podcast recording at the Autocon 4 conference in Austin, emphasizing network automation. The guests, Urz Bauman and Marco Martinez, maintain the Network Unit Testing System (NUTS), which aids in network unit testing through predefined tests and YAML configurations. Nornir is utilized for parallelizing tasks in NUTS test execution. The deployment of NUTS can be in a virtual environment, Docker container, or notebook, depending on the workflow. The discussion also delves into test bundles, predefined tests offered by NUTS for easy implementation. The importance of inventory, credentials, and plugins in NUTS test execution is highlighted, ensuring secure access to network devices.
Transcription
7693 Words, 41607 Characters
Welcome to Heavy Networking from the Packet Pushers podcast network. We're in Austin,
Texas, recording from the Autocon 4 conference, which is all about network automation. If you
don't know about Autocon, the Autocon series runs twice a year, once in North America and once in
Europe. If you're deep in the weeds with network automation or you're just trying out a few Python
scripts, this is a conference you should very definitely attend. There are hands-on workshops,
practical stage talks, vendors with network automation products on display. Personally,
I think one of the very best hallway tracks that there is. I am Ethan Banks along with Drew Conroy
Murray. You can follow us on LinkedIn, Blue Sky and the Packet Pushers community Slack group. Our
guests today are Urz Bauman and Marco Martinez, and they are maintainers for the Network Unit
Testing System Open Source Project, also known as NUTS. And today, we're going to discuss Network
Unit Testing from the NUTS perspective with Urz and Marco. So guys, I don't think either of you
have been on the Packet Pushers podcast network before. Welcome to Heavy Networking, both Marco
starting with you just very quickly. Who are you and what do you do? Well, thank you, thank you,
thank you. I'm Marco Martinez, working for Swisscom. Usually, in my day job, I work as an
architect. At my night job, I'm the maintainer of NUTS. Very good. And Urz? Yeah, so I also work at
Swisscom at the moment. So funny coincidence. And it's my fourth, fifth autocon. And so I'm quite
off heavy in network automation and try to be active in the community. You did a talk last time.
I did two talks so far. And the third workshop now. Speaking of talking, you have a podcast to
Network Auto Magic. You do with Stainzy, right? Yes, he's doing it. And I'm the sidekick.
The sidekick to Stainzy? Oh, my goodness. Okay, there must be... Okay, I need to dial in and listen
to what's going on with that. All right. Yeah, so network.omagic.net, I think is the website for
that. If you've got even more network automation content in your life, Urz and Stainzy will put
on a show for you. All right, guys. So unit testing, network unit testing. Most of the people that
listen to the show are network engineers. They're not software developers. And so when you bring up
the concept of "unit testing," that means something that I think we got to explain
to folks here. So what is unit testing? That's a good question. So that is also kind of a
philosophical question that I've been trying to define for years. Because in networking,
we kind of have this issue where everything is interconnected and we cannot really test the
smallest, fine-grainest unit. But in a sense, a unit test is the smallest piece of a moving thing
that you can test. And then if you fit all these things together, you get a system that should
be working. So what we're trying to do is doing this as a very fine granular level. We're not
doing something like, "Oh, my internet is working, so we're just checking if maybe I can have a user
on my device." Do you have anything to add for that, yours? Yeah. So if you have a checklist,
that's normal. If you're at 4 a.m., making a change, right? You want to test all the small steps.
And that is what we defined as a unit test to see if the interface up, is the port speed okay,
is the BGP neighbor here, all these small things. So it's not really integration tests,
but the small pieces. Small pieces. Okay. So give some more examples. You're talking about,
you are talking about very small things like interface testing and BGP neighbor relationships.
Give me some more. When does, maybe the way we do it would be, where does it fit into my work
for like, when am I running these tests? So that's different. How can I say? We have different use
cases. So normally what I would do is pre- and post-testing. So before I do a change, I would test
to just be sure that the network is in this state I'm expecting to. So I think we all had
troubleshooting sessions, looking for errors. And we find out this error is more than one week old
and was not from our change. So if we do pre-testing, we can make sure all the BGP neighbors are here.
OSPF interfaces are okay. I do the change. I most likely also need to change tests.
And then I run them again. And I'm sure that I did not break anything.
So if I was a network engineer, like when I've done changes before, I would have a checkpoint
before I made the change. I want to validate that I have X number of OSPF neighbors and I have this
many routes in the routing table, do the change, and then I have a follow-up. After the change,
I'm expecting to see this many neighbors and this many routes in the routing table. Unit testing
could be used to figure all that stuff out for me. Yes, exactly. So if you would do, let's say,
immigration of a PECE, you normally go there, you check how many BGP neighbors are half,
I do the change, and afterwards I want to have the same number of BGP neighbors.
Yeah, maybe to add here is just a snapshot of your current system so you can make sure that
everything is working as intended. And maybe you also write a unit test for the thing that is not
working currently. So after you executed your change, you can test with the unit test if the
thing you wanted to do is not working. So you don't really have to go backwards all the time,
where you just have the snapshot of how it was. You can also model the future a little bit.
So it sounds like you're not introducing really a new concept to network engineers,
but you're just kind of describing what it is and maybe thinking about taking changes and
breaking them down into their smallest parts to make sure each part works as you go along.
Is that the idea or? Correct, yeah. It's like building Legos. You have these one break and
maybe something in the middle is missing that we're not aware of and that's what we're trying to fix.
I think it's also sometimes close to monitoring, right? So at some point the question is how much
do I do in the unit testing and what is done in the monitoring system and they many times overlap.
Would you distinguish unit testing from other kinds of testing I would do on a network?
Yeah, more or less. There's definitely more end-to-end testing. We could also do end-to-end
testing, of course. We can automate this. Normally we do more unit testing. They are
safe to run. So like say throughput is difficult to do it automatically.
I think you used the term integration testing earlier. That's a software development term.
From a software developer perspective, we have unit testing and integration testing and so on.
Is it worth talking about the distinctions in the different tests?
Yeah, I think the problem we have in software engineering, a unit test is really, let's say,
a function you can test. But on the network, we don't really have this Python function or
whatever we can test, but we can test the smallest piece. Yeah, maybe you could say that's a property
like an interface speed or a mode or whatever. Maybe that would be a good analogy.
The integration test afterwards would be the more if the communication end-to-end is up,
like get the client, ping the other client. Yeah, usually maybe over two systems or three systems.
So we're not doing the entire chain, but parts of it, that's what I would describe as an integration
test. The unit test is more this fine granular thing that sometimes it also works if we have
a piece of the chain wrong, right? Maybe the speed is just reduced and we have like some
performance issues because of that, but it's not really something we can grasp in the monitoring
at first. And that's where Nuts comes in very handy. Okay, so you mentioned Nuts. What is Nuts?
So Nuts is a network unit testing system. It's an open source project you guys are working on?
Yes, it's open source. It's one of the best projects just because of the name.
The possibilities are endless with the name, so congrats on that.
That's always nice to talk about the Nuts. It's a PyTest plugin actually. So the idea is to have
the ecosystem that we are used to from software engineering and PyTest is the default to standard
testing framework for Python code. And it adds the capability to fit defined test cases. Many
are predefined, but you can also develop your own ones and then you can write YAML files or you
can generate the YAML files. Who are the actually test executions then? Oh, test definitions, I have
to say. Yeah, here's important to understand. We have like these two layers that we make it easier
for network engineers. We have this one layer where you define a test, where you really need to code
it, what you do. And then there's only this like application layer, I would say, where you define
what is being tested. So that one is really easy. We just have like a few lines of YAML. We give it
the target and the test is opinionated and the test then does what you define in the YAML. So it's
reusable. We have this reusability that makes it easy. What do you mean the test is opinionated?
Well, we want to make it as easy for the end user to use it as we can. So we just ask maybe for the
system for the interface name and the test does it does its own thing. Of course, there are some
optional layers you can expose. So the test will do whatever you intend to. But it's really the thing
is that you write the test in a way that it does exactly one thing. And you just need to define
this on the application layer and you give it like as little input as it really needs to execute it.
So am I using nuts then as kind of both a repository for and a framework for these tests
that I'm developing? Like what where does nuts come into the picture if I just got these tests
that I'm firing off? You can definitely have it in a repository. You can run it in your pipeline
as a normal test. You could also run it in a crown job if you want to. But you most likely
would integrate it to your automation workflow. So if you have an automated workflow, you define
the new function, you also write the test for it. And you run pre and post tests. Make sure that the
network is in the state you want to have it. So you can write tests, but you also said there are
predefined tests. We have test classes predefined that the opinionated. So you can definitely
use them, right? We use Napalm as one of the library who makes the abstraction. So that was
the easiest low hanging fruit to get the most vendors upon. I was wondering about that. Yeah,
because you're talking about defining a test in YAML, but it's like the way you would
model an interface or BGP or whatever it is, varies from vendor to vendor. So you're using
Napalm and you use Nornier as well. What world does Nornier play? Nornier is the parallelization
for the task and the inventory. So the Nornier is a framework who is similar to Ansible,
just pure Python. And we use Nornier for the execution. So if you run tests on free machine,
you can run them in serial sequence, no problem. But if you run tests on, let's say,
60 devices, you will definitely feel that the test takes long. So what we do now for each test
definition, let's say we have a test for interfaces. What this does is in the end,
you use Napalm to make a get interface. And we use Nornier to run this get interface in
parallel on all the machines. We get the data and then we aggregate the result.
I guess here you could say we use Nornier as the workflow engine, more or less. So
our test that we define in Python, we create the test itself, where we define what do we
need to get, what do we need to compare, and which input that we require from the user for the test
to be able to compare these things that you'd want to. And then Nornier just goes and does the thing.
You mentioned pre and post checks. Can you talk about how Nuts works through those stages?
So Nuts is pretty simple from the execution. You have the YAML files defining your tests,
and PyTest finds the file, execute the test. So what you normally can do is organize your tests
in different stages, in different folders, and then you can run some tests. So you defined with
the tree from your file system, which tests you want to run. And you can generate reports, for
example. But it's in a pre and post test, you would just run all the tests. You see that everyone
is green, and you do the change, and you run them again. So I would say if we talk about
software engineering, you have to test to make sure that if you fix a bug, not another bug is
introduced, and you have some side effects. Yeah, I guess on this point, it kind of depends on how
large your network is. If you have 200,000 tests, it might take a minute or two. But you might want
just the feedback of I change something in DMZ. So maybe I run like a subset of these tests.
First I do a full run, then I run the subset, which just checks the DMZ. I do the change in the DMZ.
I run the subset of my tests, check that everything is working fine. If it fails already, then I can
intervene, right? And then after that, if I know that the subset is successful, I can run the full
tests again. So you can kind of find granular also, depending on what you need, right? Sometimes
monitoring takes a little while, and we need to be aware of that. I think that's what I was getting
at is after I've run the test, how do I know that it worked and that it had the intended outcome I
wanted? Am I using other tools for that, like a monitoring tool or something else to make sure
this change that I've made has actually done the thing I need it to do?
I think I would use the pre-test with nuts just to have a fast, detailed view. Monitoring normally
monitors more on a higher level, and the test itself goes more into the details.
I guess here, if you know what you're changing and you want to make sure that it was changed,
maybe you'll write a test for it before that. So first time it fails, second time it's successful,
or if you don't have a test for it at all, you don't see it and just everything is still green.
So those are kind of these two layers that you could do.
Just a practical boring question, but I'm curious, how do I get my Python environment ready to run
nuts? Do I have to be careful about what version of Python I'm running, plug-ins, libraries, that
kind of thing? As normal with Python, dependencies can be a little pain, right? But the easiest thing
would be generate the virtual environment, pip install nuts, and everything should be up and
running. The pyftest version is pinned, and yeah, normally for each kind of framework tool,
run your own virtual length, and you will be fine. Or maybe just run it in Docker.
I'm a big fan of Docker, it kind of gets rid of some of the Python headaches.
Well, as in build the Docker container, then I'm going to run Python, and that's got everything.
You like that as opposed to a virtual environment? I do, yeah. We also provide a container, I think,
or there's one on the nuts organization, so you can use that one if you want, depending on what
you need. Maybe you need some other Python tools in your tests, so you wouldn't need to install them
as well. So I've got Python that I've updated, and maybe I'm running it in a container, maybe I've
got a virtual environment. Where should I, where's my execution environment? Where's nuts actually
running from, if I'm doing it right? Not my laptop, I'm assuming. Could also be a notebook, no problem.
In the end, it depends. What kind of workflow you have, right? If you have a CSD workflow,
it's probably in the CSD pipeline. If it's a workflow engine, it's probably in the workflow
somewhere, multiple stages. But you can definitely also run it on your notebook, like your normal
automation, but that also depends on the company. I mean, circling back to the thing I said before,
maybe the subset you run on your dev machine, on your local laptop, just to check if everything
is still fine, and you have this more bigger workflow that you have somewhere on the server
centrally. Also, a little bit in the networking space when we think about security is hard that
you reach every system. So you need to ensure that you're running that from a point where you're
able to access everything. And that might be hard from a laptop. Yeah, so it's going to be your
network management network, your dedicated VLAN or VRF, wherever you're running that from, maybe
over your out of band network. But then also, where other people in your organization can get
at the test, you want to be working on it as a group, right? Yeah. I think it's a normal Git
repository. And you work with Git, you can have versions, you can make releases. So the kind
of software principles coming into the play, I would say. So you are intending it to be part
of a larger workflow with multiple users, not just like this is my own little private tool that I'm
using. I'm actually not sure if anyone uses it. If you're hearing this, please reach out to us.
We're desperately looking for users. I think it's open source. I have no idea how many people
using it, downloading it. The only thing we know is that we are the only one who create
issues and fix them. So yeah, no idea if anyone uses it production or not, besides of us,
would be interesting to know. Yeah, this is a call. Just create an issue and say I'm using it,
if you don't have any other issues. Just hello, I'm here. So how do, is there anything I need
to do in terms of giving it access to my network? How much of my network does it need to see?
Do I have to preload IPs? That kind of stuff. Yes, you need to have this inventory from non-linear.
So if you never heard about non-linear, it would help to have an understanding of non-linear,
of course. So you defined your inventory, you have your IP address or your host name there.
You still need credentials to log in, right? So depending on which backend and napalm or your
test is using, you need SSH credentials or you need API tokens. So yeah, the classical napalm
call is probably the thing that we have to think about. But you're pulling that from an existing,
I don't have to do it again, that is, because I'm using Nornier. I've probably got those credentials.
What am I doing? Am I sharing those with nuts? It is in the memory then of non-linear. Non-linear
is executed by PyTest. So the credentials are in the memory. So you have to be aware that
you should not just let everyone run it. That would be fun. Maybe the person who has the
credential can then fix also the box. Now it's like Ansible, any alteration,
somewhere the credentials need to be, right? Of course. So what we have is the inventory plugins.
So Nornier has a couple of plugins. If you have not about networks, in-drop, all this kind of
stuff, you can pull the data out. You can also write your own custom integration for a wall system.
You can use SSH keys, for example. So it's kind of a lot of freedom. Makes it also complicated to
find the right approach. I was digging through your documentation and I was on the front page,
to be fair. Not digging through too deeply. But you've got a notion of something called a test
bundle. Could you explain what a test bundle is? Yeah, a test bundle is a predefined test where
we have some tests that we already have that you can use that are already set and fixed. You just
need to write a YAML and you go from there. So this is what the test bundle notion is.
Talking about network testing, the analogy of tests, we call everything a test. The definition
of the test is a test. Then the test that's testing the test is the test. And also the test that we're
doing on the network is a test. So everything is called test. So with the test bundle, we just
wanted to distinguish that language and have a certain term for the predefined tests.
Now the nice part is you can also write the test bundle yourself using Python or your
favorite language. You just need to hook it into nuts and execute it from there.
So naming is halt, right? Yeah. We're still testing on that.
You said there's a bunch of predefined tests that test bundles that come with nuts.
Hit the highlights. BGP neighbors, interfaces, rogue users,
ping, trace route, the classical ones, especially the thing that is supported by Napalm. Because
yeah, as we said, it's kind of opinionated. If you are used to other testing frameworks,
where you, for example, that's a PyATS, where you can make a snapshot. If you make a snapshot
of BGP, you get thousands of parameters, even the memory usage for each neighborhood.
And if I want to test, that's probably not making sense. So we only test some really
key properties, because it's kind of a trade between how many tests and so better make better
tests and not too much. I see the balance. It's a core. It's something that's common that just
about everybody will be interested in. Not everybody's going to be interested in knowing
exactly how much RAM is consumed by a particular BGP neighbor relationship. Some people might care,
but that doesn't make sense to put that in a test bundle. So the test bundle, let's describe it as
core tests that the vast majority of people using the service are probably interested in.
I would say so, yeah. So for example, we have one test with how many BGP neighbors are half.
If I have a leaf spine, I know on each device how many neighbors I expect. It's a really simple
test. We have this mechanism called grouping, where you can say on all the leaves, I expect
two neighbors. And that's, let's say, five lines of YAML. And they will test if all the leaves have
expected the number of neighbors. Yeah. Or you can maybe say, I have this template for a port
and just test it on every access port. So we have the security measures on there. We have the speed.
We have whatever you want to define on there we can put in there. And we just,
there's three lines of code. It's like interface, the group and something else. And you can go from
there. Also, maybe going back to the test bundle, it's also pretty important to see that our vision
is not that we provide a testing framework for everybody. You should be able to write your own
tests. You don't want to install a nuts that is like six gigs big or even bigger. So we just try
to keep it to a minimum in the main repository. And there's also another repository that has like
some kind of more tests that are not more specific. Can community members contribute tests?
Absolutely. The idea was to have a collection with different technologies. We have our own
organization on GitHub where the idea is to have multiple reports because also dependency is hard
in Python, right? So we want to have test bundles organized by technology probably. So far, I think
there are two or three people I'm aware of who have developed test bundles. See, somebody's using
nuts. You've got at least two or three people. They're even giving back to the project. They're
all my ego personalities. No, there was, there was one, one guy I know, but not sure if he still
uses it or time flies fast. Are there particular classes of network device or network operating
systems that it's limited to or like what's? If you write your own tests, there's no limit.
Okay. Out of the box, though, can I work with Cisco, Arista, Juniper, etc. Yes. All the stuff
that works with Napalm is supported quite well. Okay. In the test bundle, we also have in the name
which kind of framework we're using. There are community versions. So Napalm and let's say the
community driver from Nokia as Linux works well. There's also an online demo because it's easy to
test with them as they are providing the containers, right? But yeah, in the end, you can write your
bundles. And one thing maybe we should mention is because it's a Python test plugin, you can write
all the stuff also in classical PyTest. So you can write your PyTest Python and combine it with
the test bundles and you can do almost everything. What is your opinion? I gotta get this from you
guys. What is your opinion on so-called digital twins, right? There's a lot of talking about this
and digital twins come up in the context of testing, right? I'm going to run my test against the twin
and if everything goes good, then I'm going to run in production. What do you think about that?
What is your definition, digital twin? What do words mean?
No, I think there are different kind of digital twins. My opinion, the first time I heard about
digital twin, it was academic and it was kind of a mathematic graphs, this kind of thing. And
nowadays, if people are talking about digital twin, it more means like I'm running a similar
network virtual. Just a speck of a second. So you were talking about like mathematical graphs.
That's what forward networks does, for example. That's their whole thing. They're building a
mathematical graph of your network, you know, front to back and they're bringing in state
information from the network that helps populate even more detail and they're going as hard as they
can to build you a real digital twin. Something that exists off to the side, but it is as close to
a mirror of your actual network and network state based on, I don't know if they call them a snapshot
or what they do. I know they take some kind of a poll periodically to capture where the network is
at. Run tests against that and in theory, that means something. That's relevant. But yeah,
then other people that have like, oh, we got to have something that's a digital twin. And so they
want to be able to talk about it and it's just, we stood up some virtual images of routers and
plumbed them up and it's like, it's the same thing. It's not the same thing. No, it's not. It's good.
I don't want to say you should not run your virtual lab and test everything before.
I would personally not call it a digital twin. It is just a virtual test environment.
Maybe the digital broader.
Well, maybe it's a broader, yeah.
Not a full twin. They share some DNA, but it's not an exact copy.
Well, okay, so let's assume I've got some sort of a digital twin and again, what do words mean?
Digital twin means different things depending on who's using those words.
Does nuts fit into a workflow that involves a digital twin?
Which one?
Any of them would speak broadly.
So if you would have a mathematical digital twin, you could write and test
in Python to interact with it. If you want to test, let's say,
some graph queries or whatever, you could do this. I've not seen it so far.
And most likely if you have a complex system who provides this, they will also have
some tests and already this capability, I assume.
And virtual labs, definitely I see running tests first virtual and then physically.
But still, yeah, there is no replacement for physical testing.
If I wanted to get familiar with nuts before I brought it to production,
could I try it against like a container lab or a net lab first?
Can you operate in those environments?
Yes, absolutely. So there is even on the nuts organization and GitHub,
there is a demo container lab. You can spin it up in five minutes and you can run tests.
I would definitely use kind of a workflow where you first test virtually.
But what I'm saying is you cannot test 100% virtually.
Yeah, yeah, just talking about the digital twin notion, depending on what flavor you've got,
it's not that it's valueless just because your version of a digital twin is more or less container
lab of your network. It's that no, you can't test all the things because your production
network and your lab network are different things and they're not going to be exactly the same.
And so your test may work out in the lab, but in production,
something you weren't taking into account falls over on you.
I think it's usually just a cost constraint, right? You cannot have a perfect digital twin.
If you could have, it's usually too costly.
There was some software I worked with years and years ago that the point of it was to
create that digital twin. It was long before they used that term digital twin, but they would take
all these different virtualized versions of what we were running in production that was supposed
to create that environment. It was ungodly expensive and we had to dedicate a person
just to keep that platform up and running. And we never did get it working well enough
that it was worth either the spend or the headache to make it go. I'm hoping modern
digital twins are actually better than that. Even what we do in container lab is a possibility now
that is better than what we had a decade ago. Yeah, I think it definitely compliments each other,
right? But I don't think in realistic time that we'll have a perfect digital twin or that we
can test everything virtually and not care about production anymore. I think it will still have
to coexist for quite a while. Why did you guys create this project? What is your hope or intent
behind it? Oh, history. I believe it was 2016 or something. At the time I was working on a
small network who was not really critical, but still you have to make some maintenance.
And it was a Friday night at 4 a.m. We did too many changes like change the core and the firewall
and everything because it was, yeah, it's not such a, it's not a hospital, right? If someone
cannot work the next day, nobody dies. Yeah, it's okay. And of course it went sideways,
absolutely sideways. And at some point we did usual ping in the morning. We did send an email
and we were thinking it should work. And then next day or Monday, everything worked surprisingly.
But then one week or in the week, the first IP phone went offline. Then there's the next one.
So when three phones were offline, there was something. A release renewal or something goofy.
Yes, exactly. So what happened was from, let's say,
20 SVIs on two, the IP address helper was gone. No idea how this could happen. But the IP address
helper was gone. The phone had a quite long release, a lease time of DHCP, but at some point
they ran out without a flight. And I'm thinking like, there must be a better way. So that was kind
of thinking, we do software testing all the time. Why can we not have this in networking? And the
good thing is, when you work with university, you have three labels. So you just take some students,
tell them. Would you like to get your master's degree? Wow, I got a project for you.
That's on the bachelor level. So our bachelor students are quite good. So yeah, we can use them
as free laborers. Of course, we want to give something back to the community, right? So
you're not selling anything. It's all open source. So I think it's worth the free labor and they
learn a lot. So it started. And there was a couple of projects and of course, maintaining by myself.
So I believe it's a win-win. If someone uses it, let's start the question.
But I think it's an important part that we talk about testing and that we have the tools.
I think recently I did a talk on it where we even generated some automatic testing,
which was quite nice. You could, from the model or like from the source of truth,
you generated all the yamls based on the properties my devices had.
Like with AI, using an LLM to generate the yaml or something else?
No, it was just a ginger template. So I was going through all the devices in the source of truth.
They generated the yaml for me and I was able to more or less automatically test my network
because of that with nuts, which was pretty nice. It gave me like this baseline of testing,
which is kind of desirable, right? It gives you some value. Otherwise, it's hard. You start from
scratch if you don't have any tests. If I hadn't guessed wrong, we could have gotten through the
whole show without saying AI. Why did he go there? How mature in my automation practice do I need
to be to start using nuts? Do you mention source of truth? Do I need to have a net box or whatever
in place first? Or can I just take nuts and start just unit testing on my own?
If you're comfortable writing yaml files or copy pasting from the doc, I think you can start.
I would definitely recommend to start with a container lab or something.
Of course, if you go to production or integration, I would probably make sure that the user only
has read/write, so this kind of thing. But you can start small. You can make one or two tests,
like just test the BGP neighbors. It's worth pointing out that you guys have documentation.
Yes, and we are kind of friendly. If someone approaches us and asks questions,
we normally reply in the same year.
Yeah, I think also to get on this point, just do an iterative approach. Start somewhere. If you
have any issue, you write a test for that, and then you go from there. You build up your baseline
of testing as you go, as you have problems. You don't need to start with all the tests you need.
You just start with a couple of them, and they will help you. They will generate value for
you, and as you go, you get more value out of it, I guess.
I asked because I'm curious about how people get into automation and decide this works and is
valuable, and it seems like unit testing is kind of a nice, safe way to start introducing some
essential concepts. I think it's better than a notepad of having all these IPs you want to ping,
and then just copy-pasting it from your notepad. Just do the pie test run, or do the nuts run,
or whatever, and it started. I've done that, where I've written the change, and part of the
change is ping this, and here's the file that has all the commands you need to run in the expected
output. It sucks because it's 2.30 in the morning. You're like, "I just need something to tell me
if it's good or not. I don't want to have to think about this." Yeah, or even worse, do a
for loop and bash, and then ping all these IPs in the variable, or whatever. Just write the YAML,
do that before, and you can test as you go. Of course, you can do it iteratively if you haven't
done all the changes yet, but you can still see some outputs are getting better or even worse.
Yeah, I think many of us have done this tackle command for pinging from the box, a list of IPs.
It's not fun. One good thing is, with pie test, you can generate reports. You can visualize it
in a Grafana dashboard, so you can even use the data afterwards and show your manager that
you need to change. Who had thought that the manager was breathing down their Naked3AM and
was like, "Are we done yet? Can I go to sleep?" You're like, "Just check out the dashboard. It's
fine. Nuts visualizes everything there. Let me do my work." No, you guys said you're friendly,
but I'm assuming there's some kind of a Nuts community? Is there a Slack group or a Discord
or something that, if I got questions, I can hop on in there and ask? If you have Google Nuts,
you'll probably find some weird results. There is not a dedicated channel or something.
There is, I think, in the infrastructure as code. There was once a channel Nuts.
There would be GitHub with the discussion and issues. That's probably the easiest,
just open an issue or a discussion. I would also say just open a GitHub issue or open a PR if you're
this friendly or just ping us on LinkedIn. Are you guys on LinkedIn?
Yes, LinkedIn. Or if you want to have something fixed faster, come to Switzerland and invite us
for dinner and beer, and we can discuss. What other contributions are you looking for? You said
people can write test bundles. They can contribute. You're talking about being able to interact on
GitHub. Are you looking for people to contribute to the project in other ways?
Always welcome. Of course, first would be too interesting to know if someone uses it
and how they use it. You just want to hear from the community. You want to hear from people that
are using it or giving it a try. We see downloads from the package. We have absolutely no idea if
it's just someone messing with us. Do they be adding stars to the project? Is that a thing that you
care about? We lost those. Every time we see the star more, it's kind of a boost. I think at the
moment, we have around 150. I'm doing a workshop tomorrow, so I was opening the positive quite
often. I believe it's 150 stars. Someone's using it. Yeah. I think also, if you're looking at it,
just look at the documentation. If you don't get it, it's probably not a you problem. It's an
hour problem. We're not doing a good job on the documentation. You can also point that out in
the issues or PR, whichever way you're willing to contribute. In the testing engine, you need
to have a little line of code that pings something in the cloud so you know how many tests are being
run using nuts as it goes. You can watch that counter. No, no, no. We don't do this calling home
stuff. No, no. I think it's nice if someone uses it. It's nice to give feedback. But in the other way,
I'm running it in an air-gapped environment, so I don't want to call it anywhere.
Actually, there are people trying it out, definitely, because if you would open the
repository in the readme on the lowest section, there are some talks who not only from us are
mentioning nuts. But yeah, it would be nice to know if the effort is worth, because one thing
with open source is maintaining itself is a lot of work. And yeah, at the moment, if more people
would use it and we would have more feedback, I would have more motivation to work on it.
Well, Ethan wants to know when it's going to have AI, so can you get on that?
No, I do not. Drew and I actually sympathize with the problem. So Drew and I have both been doing
podcasting full time for a lot of years. We've each done, I don't know, at least a couple thousand
podcasts. You know how often we hear back from people that they downloaded and listened to the
show on a percentage basis? Almost never. We hear from people, but it's not that awful.
If we mess up, we'll hear about it. Yeah, if we mess up, we'll hear about it, actually, yeah.
So we understand, people use things and they don't interact. So yeah, give your feedback,
everyone, especially if you're using nuts. Urs and Marco want to hear from you.
All right, so in our show notes for this episode, we're going to have a lot of links. We got the
GitHub page, documentation, how you can find Urs and Marco on LinkedIn, the network auto magic
podcast that Urs co-hosts with Steinsi. Are Steinsi co-hosted with you? I don't know which it is.
Yeah, either way. I think the airtime is 80% Steinsi and 4% me. For me, perfectly fine.
He's doing all the heavy lifting and I'm just enjoying the ride.
So Urs, if people want to find you, we've mentioned your LinkedIn page. Are there any other
websites, books, anything else that you wish to mention? There is a blog post, a blog infrastructure
as code.ch, but not very often writing something. Besides that, LinkedIn is definitely the best.
I think you also have some open source lab on nuts, no?
Yeah, there are some labs. So my Git repo or my Git profile is also something to look through.
Same question for you Marco. Yeah, for me, just connect with me on LinkedIn and maybe share some
feedback. I'm happy to interact with whoever and that's pretty much all. Thanks guys for coming on.
I know you got a workshop tomorrow here at AutoCon and you had a little time today. Thanks for
making the time to sit with Drew and I and chat here in this conference room where I don't know
what you guys can hear, but there's someone doing jackhammering or something outside the window.
It's kind of loud. Hopefully it doesn't come through. Thanks everyone for listening to Heavy
Networking today. Yeah, connect with Erse Marco and the Nuts Project, download it, give it a try,
let them know. And if you'd like to meet Erse and Marco in person, you might be able to do that.
The next AutoCon, AutoCon 5 is going to be happening in Europe, dates and city to be
announced, but my guess late May, early June, 2026 timeframe and you can sign up for the mailing
list at networkautomation.forum and you will know as soon as AutoCon 5 is official. I have
been Ethan Banks along with Drew Connery Murray. You can follow us on LinkedIn, LinkedIn especially.
Drew and I are like supernodes. We're connecting to us, puts you one level away from all the other
networking folks that we're connected to. That makes it easier for you to find fellow networking
nerds and make your LinkedIn feed more useful. Join the Packet Pushers Community Slack at
packetpushers.net/community. It's free and there's over 4,000 people in there, lots of channels
covering lots of networking topics including automation and there's also a jobs channel where
people post about open positions. You can buy some merch at storeduppacketpushers.net, like,
share and subscribe. Check out packetpushers.net for a dozen other engineering-oriented podcasts
that we offer to you for free with no login required along with newsletters and YouTube.
You're an excellent human. We care about you. We hope the work we do here is helping you in your
career and we hope you have it a fantastic week. Speaking of which, we are taking next week off
for the Thanksgiving holiday here in the US and then we will be back, last but not least,
Remember that too much networking would never be enough.
Podcast Summary
Key Points:
Autocon 4 conference in Austin focuses on network automation.
NUTS project maintains the Network Unit Testing System.
NUTS helps in network unit testing with predefined tests and YAML configurations.
Nornir is used for parallelization in NUTS tests execution.
NUTS can be run in a virtual environment, Docker container, or notebook.
Summary:
The transcription discusses the Heavy Networking podcast recording at the Autocon 4 conference in Austin, emphasizing network automation. The guests, Urz Bauman and Marco Martinez, maintain the Network Unit Testing System (NUTS), which aids in network unit testing through predefined tests and YAML configurations. Nornir is utilized for parallelizing tasks in NUTS test execution.
The deployment of NUTS can be in a virtual environment, Docker container, or notebook, depending on the workflow. The discussion also delves into test bundles, predefined tests offered by NUTS for easy implementation. The importance of inventory, credentials, and plugins in NUTS test execution is highlighted, ensuring secure access to network devices.
FAQs
Unit testing in networking involves testing the smallest pieces of the network configuration to ensure functionality.
Unit testing can help network engineers validate changes, troubleshoot errors, and ensure network components are functioning as expected.
Nuts is a network unit testing system, an open-source project that uses PyTest to define and execute test cases for network configurations.
Nuts can be run in various environments such as a CI/CD pipeline, local notebook, or integrated into automation workflows.
Nuts requires credentials stored in the memory of Nornier, ensuring security by limiting access to authorized users.
A test bundle in Nuts refers to predefined test cases that are ready to use with minimal configuration, simplifying the testing process.
Chat with AI
Loading...
Pro features
Go deeper with this episode
Unlock creator-grade tools that turn any transcript into show notes and subtitle files.