2.5 Admins 291: UPS for LiFePO4

This late night Linux family podcast is made possible by our patrons. Go to latenightlinux.com/support for details of how you can join them. Support us on Patreon for access to ad free episodes and early releases. That's latenightlinux.com/support. 2.5 admins episode 291. I'm Joe. I'm Jim. And I'm Alan. And here we are again. And before we get started, your customary Clara article plug Alan is designing Open ZFS storage for independence. Yeah, so in this article, we talk about some strategies you can deploy to help build systems that aren't locked in and to make sure that you'll be able to move your data around however you want, both from your existing system to the ZFS system and then to your next system, whether that's ZFS or something else. Open source and OpenStand, it's a. Basically. Right, well, and can the showroom says usual? Please, please, please stop using past keys for encrypting user data. Right. Tim Kapale. The major issue here is that if you have not taken precautions to make sure that you can get back into any past key security counts and you lose or accidentally destroy the device that is your past key, you get locked out all those accounts. And a lot of folks don't really realize that aren't really planning ahead. And if they just set up a past key because they're told a past key is what you should do. It seems convenient to do this and this and this and this and the other all with that past key. You might have the same situation I've recently complained about with multifactor authentication when end users are just their end users. They're not IT admins. They're not always going to get it right ahead of time. Think, well, I need to make sure I've got scratch codes or you know, fallback methods to get in. They're just going to do what they were told to do and was told were safe. And then when they drop their phone in the lake on a weekend, they can't get in anything anymore. Yeah, I think in particular in this case, they're talking about encrypting a backup of this stuff from the app using the past key. And it's like, well, the main reason I would need to restore this backup is if I lose the phone. And if this past key is tied very specifically to this phone, then maybe that's not the right combination of security factors. I don't know how we get this message really across to the people who need to hear it. But I think the simplest version of the message that really needs to get through to the end users of the world is you must configure fallback authentication methods. You must do that. You can't just be like, well, I set up the MFA and I'm done. Nope. You need an answer for what happens when you no longer no longer have access to that multi-factor device or that past key device because it's going to happen. And it's not the kind of thing you prepare for after the catastrophe. Yeah. The main point of this article specifically is past keys are designed to solve the authentication problem to prove that you're the right person. We shouldn't be conflating that with using it to drive the encryption keys for maintaining the confidentiality of data. That shouldn't be tied to the same thing as the authentication. Right. Because maybe you have multiple different ways to authenticate, but you need one more recoverable way to get to the encrypted data. And so to Jim's point, they're saying in the article, we cannot and should not expect the users to understand this. The wider authentication industry needs to stop trying to push this one solution to do more than it was meant to do. Right. It was meant to solve the problem of people having a lot of different passwords, but it isn't meant to solve the key management problem of encryption. Yeah. And Tim is very clear in his piece that he thinks that past keys are a very good idea for authentication. He has nothing against that. It's just this particular use case. Yeah. And that the recovery he's and so on that you need to make sure that you can restore your encrypted backup when specifically the most likely thing you've lost is the device that this past key was tied to. And if you fear Tim might have no problem with past keys. So as authentication devices, I have some issues. The quantifiable value add of using a past key rather than using simple MFA with a login is that unlike two-factor authentication in which a scammer might be able to convince a victim to give them the one-time code that then the attacker can use or convince the attacker to just go ahead and tap the button with the right numbers on it on their phone, you might be able to confuse granddad or grandmother to do that if you're a nared well. But if instead, grandma or granddad has a past key, then a remote attacker can't convince them to unlock the session that remote attacker is operating because the past key is not in the physical location, it would need to be to do that. That is a genuine security ad, but I would like to point out that's really kind of only a genuine security ad for terminally clueless people. And again, if this is a security solution that is specifically most useful for terminally clueless people, it really needs to be designed around the idea that that's who's going to be using it and we need to make sure they can use it safely. Yeah, and not overload it and try to make it mean other things that it wasn't meant to do and cause those same users more grief. Isn't it wild for how many generations we've had computers now as a species? And yet we still haven't figured out how to build an actual large scale computer system that's usable by the humans who we intend to actually use it. We haven't figured out where on that dial things should go. The dial that is on one end security and on the other convenience. Well, that's the thing. There is no one place it goes. To hijack your analogy, it's a knob because you need to change the position on it pretty frequently. There is only one position. It wouldn't be a freaking knob. You just started to resist her into the board and called it a day. Well, I suppose what I'm trying to say is we haven't figured out how to figure that out for each application. That is certainly arguable. I'm feeling a little cranky today after dealing with a lot of different things and my take tends to more be that, you know, we just don't care enough to bother and we never have and we still don't. And I'm not sure when we're going to start. I mean, none of this stuff is anything that can't be figured out ahead of time. We're not sitting here doing like pages of advanced like tensor math to come to the conclusion that something was designed wrong. It just like, we're sitting here and pointing out obviously freaking dumb things. So why did we do the obviously dumb thing in the first place? The only answer that makes any sense to me is, well, we didn't care about it being dumb. We're not enough to schlep out the door so we did. About a year ago, we talked about anchor, solar, exportable power stations and how you can use them as a UPS. We've all got them and now Alan, you've bought yourself a new one, a C1000 Gen 2. We're not sponsored. You've used your own money for this. You're pretty happy with it, I understand. Yeah. So when we were talking about it a while ago, you know, Jim and I have been complaining about the price of new UPSs or specifically batteries. You know, I have a bunch of APC backups, 1500 G's and you know, it's been two to and a half years. The batteries are all dying and it's like, why am I spending more money on lead acid batteries that are just only at the last two years? So about a year ago when we talked about it, I tried out a blue tea. I think it was 400 watt hour or so. One of these portable power stations, but it turns out with that one, if you're pushing more than about 60 watts through it, a little fan turns on and makes a lot of noise, which would be really poor for podcasting. So that had been relegated to powering the switch and the modem and so on in the laundry room kind of my utility closet. And I hadn't really found a very satisfying solution. And then I saw an ad for this C1000 Gen 2 that specifically mentioned they had cut the response time down. So with the older generation of these power stations, the UPS cut over time was 20 milliseconds, which seems good enough and I think almost any computer will be fine with it, but it's a lot higher than the response time on a regular UPS. But with the new Gen 2's, the response time is cut down to 10 milliseconds, which is a lot more interesting. It's still, I think a UPS is normally around 5 milliseconds, but it's a lot closer. And if you're talking about 60 hertz, it's less than a whole cycle. But it's not up to 20 or up to 10. Yeah, I think it's up to 10 or at most 10 milliseconds for the cut over. Yeah. It was on sale on Amazon and it looked interesting and I was really, you know, my UPS was beeping because the battery was dead. So I'm like, I'm going to give this a try. So I got it and I installed it and set it up for the router, which is actually a whole like 2U server in my house. Of course. The 2.5 gigabit/10 gigabit PLE switch that powers via PLE all the access points at my house and fans out to a bunch of stuff. And the 10 gig switch that connects all the servers together. So it adds up to 250 watts current total draw all the time. And one of the very nice features that the anchor app has for it is peak shaving. So where I live, we have time of use power billing where I pay three different rates depending on the time of day for power. So the battery can actually say, all right, it's on peak time now. I'm going to use the battery instead of the wall to get power to power my computers up to the point where the battery hits 20 or 25% or whatever I can figure it to. So that was me to kind of shift the constant load of my server to a different time of the day when power is cheaper, which is just a nice add-on. And of course, you could do something similar with solar panels and stuff. Yes. And it has a solar input where you could actually you know, charge the battery not off the grid during the day and so on. But just the peak shaving, you know, I'm only going to save like 20 cents a day. So it's not going to add up to pay off the power bank anytime soon. But it's every little bit on top of my UPS wasn't ever going to pay for itself. So, you know, this seemed pretty interesting. And then I actually experienced a longer term power outage. One of the reasons why I hadn't bit the bullet on getting something really big in a big battery system for a whole house or something is, you know, in the 15, years I've lived in this neighborhood, there's been one prolonged power outage and it was caused by like a car physically tackling the telephone pole at the corner and knocking the power out for like 12 hours. I was upset that the person who crashed into the telephone pole was out of the hospital before the lights were back on. In this case, it was just a fault with a transformer or something. The power went out at 9.20 pm and didn't come back on for a couple of hours. Originally, they estimated it wouldn't come back on until 3 in the morning. So, I was glad when it came on a bit early. But this power thing suddenly became a lot more critical where I was like, "If the power doesn't come back on soon, I'm powering off the router and the internet and moving the portable power station to power my refrigerator instead." Yeah, that's my plan. I've got the C300, not the C300 DC, which doesn't do the UPS stuff. That's only for kind of charging laptops and stuff. But yeah, the C300, I've got it on my backup box and my access points and everything. But if the shit hits the fan, then it's going to potentially power my heating system, my fridge, and the key stuff that I need. I want a slightly different direction than Alan when I bought my anchor stuff initially. I bought a handful of C300s and one C800 to go in the rack downstairs. The C800 powers my big work station, my server, a power of Ethernet switch, a couple of monitors, stuff like that. The C300, which is the first one that I tested and tested very extensively. It's only got a 288 watt hour capacity. I say only, but that's sufficient to give a, it was a 9th generation i7 workstation, a 24-inch monitor running an artificial workload pulling 100 megs a second random off a solid state drive. It kept that up and running for three and a half hours. An APC 1500 volt amp UPS would be lucky to keep that set up running for half an hour when it was brand new. After it's a year old, I guarantee you it's not going to have a half an hour of runtime on that. Now we look at the newer stuff, the C1000 Gen 2. You can pick that up for about $430 where at the time I was spending about 250 for my C300s. The C1000 Gen 2 has a 1024 watt hour capacity on it. That's a lot. Anyway, the thing I want to get out with my slightly different strategy, I do have a lot of really ugly power outages where I live in the American Southeast. We have all aerial power lines. We have tons and tons of trees with very iffy arbor maintenance keeping tree branches away from lines. So if there's a lot of wind or a heavy storm, it is not uncommon to have power outages. And when Hurricane rolls through, well, last year, my family was at a power from more than a week. Luckily, I already have this giant grab bag of high capacity power banks with AC capacity. And just down the road at the assisted living home that my parents live in now, they did have power. And that meant that anytime we wanted to, we could grab a handful of very easily portable devices with really easy grab handles, their own trouble lights, everything. We could just go visit mom and Scott, plug them in for an hour or so over there, charging back up, bring them back home, run whatever we wanted to off of them. It makes a really, really big difference not only for the brief power outages that you normally think of having UPS for, but if you've got enough of these things and enough capacity, it can be a real big disaster survivability piece of equipment. Yeah, try doing that with an old lead acid UPS. And also the lead acid UPS, you're not going to have USB ports on it to directly charge, you know, phones, tablets and whatnot. Instead, you would have to plug your AC charger adapter into the UPS to then convert back to DC to charge your devices, which is just brutally inefficient. And slashes, you know, that runtime slash capacity even further. Whereas here, you've got direct DC charging for devices that want DC. You've got nice AC out for devices that want that. And you've got a really nice trouble light with multiple brightnesses, you know, for when, again, you know, the power is actually out. Turns out that's pretty frequently a really nice feature to have baked in. It's amazing. You mentioned that the one thing you lose with the Gen 2 of the C1000 is it doesn't have a light anymore, but it is like 11% smaller and quite a bit lighter. It's comes in at like 24 point something pounds. I was almost surprised with how small it is considering I was getting, you know, a kilowatt of a kilowatt hour of power in it. Of course, having had the power outage and kind of wished to have more power and especially watching it peak shave each day and noticing that when it also powers my media server as well, the battery doesn't last long enough to survive the whole peak time. And so I'm still paying for a bit more of the peak time power. I started looking at the bigger one and they have the C2000 Gen 2. I think it's just around $800 US, so not double the price for double the capacity. And that looked interesting, but then I started looking at previously, we had talked about their like F3800 kind of whole home but portable version. And I know at least one of their listener has written in about their experience. I actually bought a pair of those and wired them up to the house and they were pretty happy with them. But I also noticed that anchor announced their new like E10 system that's like a Tesla's power wall type mechanism. It's something big that you install in your garage to power the whole home. And so I'm still looking at options. The other interesting thing is the integration with home assistant. I can actually pull all the stats out of anchor into home assistant. And like I can actually see graph of the battery charge and see the peak shaving happening and all that integration and even configure it. So I can have automations is that like, hey, a storm is coming. Don't peak shave today. Save all the battery in case the power goes out and things like that. How are you connecting it to home assistant then? They have an integration that's not great, but there's an MQTT based one that works very well. So I just got the anchor solics thing out of the community store and it works very nicely. Okay, this episode is sponsored by Automox. If you're evaluating endpoint management solutions, the real risk isn't which tool you choose. It's choosing one without a proven blueprint for success because while most endpoint tools promise automation next to none promise outcomes. That's why Automox introduced turnkey results. Turnkey removes the uncertainty that holds endpoint automation back. It delivers a personalized results blueprint so Automox is configured and operated to achieve real outcomes from day one. No trial and error, just clarity. Each blueprint is validated across millions of endpoints managed by Automox and tailored to your specific environment, your risk tolerance and your operational goals. Instead of guessing how aggressive to be with automation or how to configure policies safely, start with the plan you can trust and you can choose to implement the plan yourself or have Automox do it for you. Either way, you get faster time to value, lower risk and predictable endpoint outcomes. Automox turnkey results. Know the plan, trust the result. Learn more at www.automox.com. Let's do some feedback. Christian writes, "I know nobody on this show would like to put their mission critical data on butter FS, but what about using it for the root file system? That way you can still take snapshots on the root and you can still use ZFS for everything else, right?" As long as it's a single disk file system and you're not using butter raid and I would recommend avoiding compression and don't rely on replicating it anywhere, but with all that said, if that's still something you want and you're looking for my personal blessing, you have it. Be free. Run butter on your root. Take the occasional snapshot. Be happy. It just seems like you're going through a lot of contortions to try to get a little butter in your file system when it's probably fewer contortions to do ZFS boot menu. I don't know about that. There's a lot of distros that default to butter FS on the root, like for Dora, so is a, you know. ZFS boot menu is really not an end user easy level solution. Is it reasonably simple for an experienced, you know, IT person to set up? Absolutely. My entire family are full-time Linux users and I would not tell any single one of them, "Hey, I want you to go install a machine with, you know, ZFS boot menu for me." No, that's not going to work out. Yeah, but they could probably work out the Fedora installer, which would just buy default, give them butter FS. Any of them would be able to do that, yes. Before I ever met my wife, she had actually had a friend in roommate have a Windows installation go completely bonkers. I think it got malware to hell and back. I don't know. This was long before the two of us met and she went on the internet and discovered Ubuntu and downloaded it and installed it on her friends laptop and her friend became an Ubuntu user. Janice is not an IT person. She's just, you know, a competent, reasonably self-confident person who saw a problem and resolved it. So yeah, that's totally doable. I don't want to dog on ZFS boot menu. I think it's a really compelling product and I think within the limitations, they're forced to operate under. They've made it reasonably easy to use and you wind up with something really awesome after you go through those steps. But calling it easy, nah, I got to kick up my feet on that one. Well, yeah, I guess my point was so much it was easy. It was like, if you want the features, maybe it's worth a little bit of extra pain than getting the. First implementation you would get from butter FS. It's like at what point do you just stick with the default like EXT file system? I guess it's a bit of a different story if you're distro defaults to butter FS and you'd have to opt out of it But if you're opting into butter It's like how much more effort would it be to opt into ZFS instead a fair amount Because you you have to go through a deb bootstrap, you know command line installation You're not using like the normal installer at all You got a you got a monk you're in with charutes and deb bootstrap and you wind up with like a weird system that doesn't quite look like a typical desktop or server from any of the builds available from the normal install And like can you can you work with all that if you know what you're doing? Yeah, you can but I'm not even calling that simple and easy for me Duable absolutely simple and easy. Oh, hell no. I would argue that it's Honestly, it would probably be easier right now if what you really want is you want an easy way to go ahead and get ZFS on root with boot environments Make switch free BSD Let's do some free consulting then for first just a quick thank you to everyone who supports us with PayPal and Patreon We really do appreciate that if you want to join those people you can go to 2.5 abbins.com/support And remember that for various amounts on patreon you can get an advert free RSS feed of either just this show or all the shows in the late-night Linux family And if you want to send any questions for Jim and Alan or your feedback you can email show at 2.5 abbins.com Quentin writes "I've been wondering about restricting internet access for some of the IoT devices on my network that don't normally need it As a home automation enthusiast I have a lot of them Obviously a VLAN is the standard way to do this but I've been wondering about a tactic that would be easier to implement Suppose my DHCP server was configured not to give out a usable gateway address to the MAC addresses I'd like to restrict This is just security by obscurity and a dedicated attacker could bypass it easily by searching for routes to the outside world But I doubt that your average IoT device wanting to call home to a Chinese server would do such a thing Unless my brilliant solution becomes more widely adopted of course You don't really need VLANs for this but I don't really approve of the idea of just handing out a bogus gateway address As Quentin mentioned himself that's very easily bypassed security through obscurity A better and arguably just as easy way to manage this maybe even easier and more discoverable later Is you just set up static DHCP leases that always hand out the same IP address to the devices that you want to restrict And you set up firewall rules not to allow those devices to talk to the internet. You don't need a VLAN Yeah, so what you could do here is as Jim said create static entries for these servers in your DHCP configuration Our dot service IoT devices so you get the bonus of all your IoT devices having static IP addresses Which is handy for your home assistant to manage them and so on But if you were to do that in all in a certain range of IP addresses in your subnet or as a whole separate subnet But that gets complicated so if we just say you know everything from 120 to 150 in your subnet is IoT then you just have a firewall rule that says anything in that address is blocked and can't go out to the internet And your DHCP server only gives out that range to those mac addresses and and you're good to go It's just change the range in your leases for regular devices to not include that set of IPs And statically assign them to those devices and block it and it'll be a lot better The only reason that you need the VLANs or otherwise implement multiple subnets Which is really all VLANs are there a way of implementing multiple subnets without needing an actual router in between The only reason to do that is not to restrict devices from getting to the internet To restrict two different devices on the same local network arguably from being able to talk to one another Because one way or another when you leave the subnet is when you have to go through the router And therefore the router gets a chance to implement a firewall rule You can't just block off one range of IPs from talking to another range of IPs on the same network Because they are allowed to just speak to each other directly and that's the way they're going to do it They don't send their traffic through the router in the first place But again if all you're worried about is saying you don't get to go to the internet That's what firewall rules are for no extra subnet required Yeah, and I would caution against the DHP lease that purposely breaks things If you give out a bogus IP you're going to keep seeing ARP traffic where all the IoT devices are being like What MAC address is the router and there's nobody there answering so they're just going to be spamming the network all the time And if you're on you know your 2.4 gigahertz Wi-Fi There's going to start eating up more and more airtime and making your that side of your Wi-Fi even worse But also some of the devices have a failsafe where it's like A, they'll either try to guess and try to find the internet another way Or they'll be like this isn't working. I'm going to reset and try to find a connection again And that might mean that they fall back into a safe mode or other things that just cause the IoT devices not to work the way they're supposed to So I think Jim's suggestion is a lot better And would have saved me so fast so I was doing it with VLANs and I forgot to change the tagging on one of the VLAN ports And my IoT circuit transformer clamps they go on in my circuit panel to measure the power of all the different circuits I had fixed the config but didn't copy Run start on my switch and so when I had a power outage my switch went back to the old config where That ethernet port wasn't in the right VLAN and suddenly I had no monitoring of my electric panel and I had no idea why The other thing that we should warn you about kind of long the same lines is you know Alan talking about devices you know kind of freaking out and trying things to get the internet Is these days an awful lot of things even by default right out of the factory they'll they'll spoof a Mac They'll give you a random spoofed Mac every time And you can't use any of these tactics with a device that isn't always reporting to you with the same mac address In those cases if you get a device that randomizes the mac address and you don't have a way to disable that Or you don't trust the control that says it's letting you disable that Then you're back to like you really do need entirely separate networks whether it's via VLAN or via you know literally separate physical segments You have to break them out Because you're reliant on that mac address being unique and not changing to allow you to identify the device on the network And you know thereby do different things to it than you do to the other devices Yeah, depending on your situation it might be easier to Enumerate and give a static IP address to all the machines that should have the internet and leave the default Everything does not in this blessed range of the subnet not having the internet But that might might guest Wi-Fi really annoying at your house This is why I have a separate SSID with a separate subnet on a separate VLAN for all my IoT stuff Because I want to keep it all separate and it's that SSID is 2.4 gigahertz only and my phone knows it's not to ever try to connect to it I know for some listeners the idea of setting up VLANs seems relatively trivial For some listeners they already have Wi-Fi gear that supports you know arbitrary numbers of VLANs And they can spin up whatever they want and they're comfortable with it and those folks sweet You got all the tools you need you got the knowledge and you need you ready to go I also know we have a lot of listeners that are like oh my goodness that is a lot. I barely know what a VLAN is my gear doesn't support that Ah, I don't want to learn all that well. There is a much easier answer Which is just by a completely separate Wi-Fi access point on a different SSID And now you can control things because everything is coming onto your real network from that other Wi-Fi network So you've essentially created a guest network just by buying you know a different AP and putting it with a different SSID and password And that's what you put your IoT devices onto and they have no way to jump off of that and get somewhere else where they won't be tightly controlled Yeah, and that's a great use for your old AP when you've upgraded the new one to have six gigahertz and so on Maybe keep the old one around and use it just for IoT crap Most of it only supports 2.4 anyway Yeah, a lot of folks in our industry like to laugh at stone-ac solutions Not me stone-access work We use those things for a few thousand years for a reason Yeah, I'm definitely done with Jim here the idea of Doing this with just a firewall rule is a lot nicer especially when you have switches that aren't managed and you can't easily do a bunch of VLANs and get over complicated Even if you wanted to it's like I Wicefenn all that money on smarter switches when I can just write a firewall rule and and I have to bother upgrading all my infrastructure Right, well, we better get out of here then remember show at 2.5 admins.com if you want to send any questions or your feedback You can find me at joarest.com/mastodon you can find me at mercenarycissadmin.com and I'm @AllenJude. We'll see you next week