Go back

Best Of The Render Farm (Compilation)

107m 36s

Best Of The Render Farm (Compilation)

Mark is building a home-based render farm to reduce reliance on expensive online rendering services, using cost-effective, refurbished computer hardware sourced from eBay during a tech industry hardware clearance period. He discovered a suspicious pattern where five of his Dell computers simultaneously suffered boot drive corruption, leading to speculation of a targeted malware attack or security flaw, and he criticized Dell's lack of support and forensic investigation. To build the render farm, he modified cooling systems, reduced GPU power consumption to avoid overheating and fuse failures, and successfully installed eight 4090 GPUs into a server. The setup includes innovative cooling concepts like phase-change materials such as Glauber salt, though these remain experimental. Mark acknowledges the risks of buying discounted hardware, emphasizing poor quality and lack of support, and points to the need for better reliability in DIY systems. He is also collaborating with Houdini and Stagger Dynamics to optimize the farm’s network and processing power, with a focus on CPU performance for physics simulations. The project demonstrates a deep technical dive into computer hardware, cooling, and rendering, driven by personal passion and a desire to create high-fidelity 3D content, while remaining grounded in practicality and cost-efficiency. Despite setbacks and technical challenges, Mark has achieved functional results and continues to refine his setup.

Transcription

19996 Words, 103990 Characters

English
This episode of Distractable is present to you by T-Mobile 5G Home Internet. The folks over at T-Mobile have some big news for you. They now have the fastest 5G home internet according to the experts at UCLUS speed test. If you want the fastest 5G speeds with a 5-year price guarantee, visit t-mobile.com/home-internet-check-availability. Price guarantee exclusions like taxes and fees apply, fastest based on UCLUS speed test intelligence data, 2nd-F 2025, all rights reserved. So, I have a new hobby, a new expensive hobby. No, we know about the lenses. No, no, no, no, no, no, no, no, no, no, you don't understand. All right, who's our next guest going to be who like breaks a house trying to get you your thing? No, it's not like that at all. Is it mobile games? They can be expensive. They can be, but it's not mobile games. I want to build a render farm. That's not as exciting as you're trying to make it seem. What come on? I am personally. I do think that's interesting because I'm interested exactly at what the infrastructure for that is and how you set that. But, like, to normal people, that's anything even tougher self than lenses. I'm going to be honest. Oh, come on! I'm going to let you cook here only because I filled up my Synology the other day. I've had it for, I don't know, 10 years. I've had it for a very long time. And it's got 30 terabytes of storage. Last night, I went to transfer some files. It's full up. Damn. So, if you've got space for me, Mark, I'm in. Yeah, so I think, wait, you'll like this because you know when I was doing that whole hard drive stint, which people forgot about when the lenses took over, but I was doing a hard drive thing there for a bit. Yeah, when you were Thunderstruck, whatever it was called. Yeah, it's exactly Thunderbolt. So, I built a server that has 400 terabytes of storage, right? Because I was like, I'm going to future proof this. I went on a whole hard drive kick. I got some refurbished hard drives. If there's one thing I love is that doing these hobbies and trying to find a good deal for them, even because it's so expensive, a lot of these things. So, I always try to find like the good deal for it. So, I bought refurbished hard drives. And in every way, that's a big no, no, no, don't do that. But it just so happened in a specific situation that I did this, if there was some cryptocurrency, those based on hard drives ways I talked about before. Yeah, yeah, yeah, yeah. But, render farm is different because it's extremely difficult to get server infrastructure at a cheap price. Unless you just so happen to be in an era where everyone with a data center is replacing their hardware with this new fancy AI bullshit hardware. And they are throwing everything out at incredible discounts. And you can find the most amazing server hardware ever. On eBay, four steep discounts. Okay, how much is a steep discounted 400 terabyte? Oh, you're not going to like this. I'm talking like 50% plus discounts, right? How much is a 4090 right now, right? 49, a couple grand, isn't it? 2200 bucks or something? 4090s right now is like two grand, like around that, it's still very expensive. You can get some things with a 4090 in it, an entire CPU, RAM, like power supply, motherboard, case, all this shit for $2,400. So 400 more than the card and a 4090 in there. 400 more than the card itself for the entire fucking computer. With an NVM, two terabyte NVME, it's got 64. Look, even my camera agrees. What the shit is, why is that happening? Why is there a thumbs up on your face? It's so positive. You can get all that for $2,400. And that's still a lot of money, but you talk about getting the computer with a 4090 in it. For that, now here's the real kicker. That's just, keep talking. That's just domestic level computer hardware, right? At steep discounts, 'cause now is a better time than ever in the history of forever to get a computer because there are so many companies that are trying to clear shit out to get this new shit in. What do I search for on eBay for this? Give me some keywords. What you wanna look up on eBay is you wanna find, you wanna find the specific server grade processor that you want because what's happening is all these companies realize they need GPUs right now. So actually GPU deals are hard to come by for workstation level graphics, but you can still get good deals because people are still trading out computers. But you can get like 64, 9628 core processors in their server chassis with all with like a terabyte of RAM. I don't even know what you would do with a terabyte of RAM for like 80% off of the cost, and they are perfectly functional, very modern, like only a year to old. But because every company in existence is trying to get rid of what they have and replace it with these bigger GPU-based servers with like eight GPUs in 'em. You can get, you can get crazy shit. You can get the craziest shit for so much less. There's processors out there that are like $12,000. You can get, you can get two of them. You can get two of them from like a 25% of the price, which is a lot. I haven't bought these, but this is why I'm like, now is the greatest time in the history forever to build a render farm. I wanna see your half a million dollar server set up. You don't need to spend half a million dollars. What was I just telling you? I don't know, I had left. Oh yeah, you weren't here. Where did you go? I had important business to take care of. (laughing) I'm with you, Mark. I see what you're laying down. Yeah, you see what I'm laying down, you see? What, so what? You're gonna buy, well, you're gonna lease space and just buy a bunch of second hand servers and set up a render farm thing for cheap for people to use? No, just do it in my garage. If I have to lease space, that's gonna cost more money. But if I do in my garage, where's the cargo? I don't, my car hasn't been in my garage for a year, man. Yeah, I mean, you know, he had movies set in there and stuff or whatever. Yeah, I have a whole set in there. But once that's gone, now that movie's done, I have space for everything. And I put air conditioning in there because filming in there in the summer was such a pain because it wasn't air conditioned. So I put an oversized air conditioner in my garage so I can have a server rack in my garage. I can't believe you filmed a whole movie in your garage. I didn't realize that. I picked up a lot of shots in my garage. I was basically talking about it for the past year, wasn't it? He did talk a lot about that, yeah, I feel like. Huh, that's crazy. You know what, Mark, points to, oh, that's someone. So I wanna build a render farm. I wanna get some sweet deals on some sweet computers and put them into my garage. And I'm sure that buying this shit off of eBay is definitely reliable. Well, so you're saying eBay, these companies aren't going out of business, but they still probably like, offload these to just like wholesalers or something, right? Can you go to like in-person auctions where this stuff would be even cheaper probably? For sure. I haven't scratched the surface of where these sales are. I've only really just started looking and finding like incredible deals. If anybody knows the secret ends of where to get this shit for the cheapest prices, put it on the subreddit, let me know about it. I'm about to start a rendering revolution because online render services cost way too much. And if I can do it from home, because I got solar panels. So I don't even need them from the electricity, okay? - Man, if we have a zombie apocalypse, you're gonna be able to film movies from your house. (laughing) - Mark's never gonna have to stop making content. - Yeah, one can only help, one can only help. Anyway, that's my small target. - My latest hobby, have I told you guys about it? - Yes. - Render farm? - No, yeah, yeah. - Render farm? - My main man, Tugum4470, you're the real, you're the real real. You get me the best discounts on possibly good server hardware. - I'm sure they're all okay. - I haven't even gotten it yet. I'm sure it's great. - Where are they coming from? - Shenzhen, China. - Well, is that where all electronics come from? - I feel like that's going into possibly a prejudice territory, but keep going. - Oh, yeah, honestly, there's just like the manufacturing in Shenzhen is like every piece of technology. - Where was the place where the glycine factory was? - Medical grade glycine. - Medical grade glycine, did you hear about this? - Of course not, it's me. - On TikTok, there was a huge trend where there was this nonstop talk about medical grade glycine. And there was this one glycine. - I don't know. - I don't know and I don't know what it's for. - Okay, I just know that medical grade glycine was very important. - So there's certain things that you can do in CG is like computer graphics. A lot of it comes down to rendering, but there's a whole world of like art there that I know I'll never fully get. I won't be an expert at it, but at least the technology size is like, okay, for the sake of rendering, I can render and I can figure out how to do that. I can build a render farm, right? I know how to build a computer because I've done it before. I've done it for you. I look into it like a server can't be that different. And so this is where my brain goes because from one to the other, I have a basis of knowledge. This is a problem that I'm facing. I think I can figure out puzzle away to a solution using my knowledge. You never have any desire to do that? - My knowledge is to socially outreach and find someone else who can solve my problem. I was gonna go into the spiel of all the hardware that I've come back to and talking about that and talking about the latest thing that's gonna make people so mad about me talking about it. I can't wait for people to be pissed off. (laughs) - Is it a render farm? - I've heard that a drink. - You can't say that word in front of an render farm? - Did render farms kill his family? - Yeah, probably. I'm gonna go with yes. - You ever swallow and immediately know you did it wrong and you're fucked? - Did you just do that unrelated to me saying the word render farm or was that? - That was completely unrelated. I was just taking a sip and my body was like, I should breathe that. - I've been building my render farm. I'm a render farm expert. I'm an expert. - You're still on about that. - I've discovered the magic that is blown fuses again. So it turns out computers draw a lot. of power, a lot of power. And so for those that aren't aware that even just running one computer all day long is an incredible draw of power. It's like running a microwave constantly. So one of the things that I'm looking into is getting a power into my house, like figuring out the power since I'm trying to get like a better battery backup. But I was doing something yesterday that I was, okay, so I was up on my roof, right? And I realized how cosmically stupid what I was doing was I climbed up on the roof and I called for it. I mean, I mean, I was like, can you pass me the hose? So she throws me the hose. And there I am standing on my roof, watering my solar panels because they're dirty and encrusted with grime. But I just like, I just was like standing there and thinking about it as this like wonderful era that we live in with all this technology and able to just grab electricity out of the air with solar panels. And there I stand with a garden hose being like, so fix it. Well, if you want to grow your solar farm, you've got to water it. Yeah, that's true. That's true. It just, it just felt so dumb. It was happening. But then I, I thought you were going to tell you were cleaning out your gutters or something. I was not expecting the twist of watering your solar panel. No, no, because I could understand like cleaning out the gutters. And what I should have done is I, I have a ReoB power washer. And what's great about it is it has a hose attachment on it. But my hose isn't long enough. And I didn't have the battery charge and I was just like, let me go up see what they are. And they looked pretty crusty. Were you using the solar panel attachment for the hose? Shit. You know, they actually make a ReoB solar panel cleaner. It's basically the same as a power washer, but it's designed to work way better on solar panels. Yeah, because that stuff is crusty because I got done. I wash it off. And I, you know, I felt like I was playing power wash simulator because I didn't have enough pressure because I had one of those little handles. But you know, at the distance, the wind alone was beating me. So I got done. Everything was wet. And I, I thought I washed it thoroughly. I saw some gunk runoff. And by the time I got to the end, I went back to the beginning and the beginning was dry. It was just as dusty. It was like super like just as crusted over. Couldn't see a single cell. And I'm like, damn it. Uh, so I bought, I Amy bought a big mop with a big handle, so to deal with my gun, mop my solar panels. If people out there want to know what the celebrity lifestyle is like, it is mopping your solar panels on your roof. So yeah, that's what I get to look forward to. Um, and so I, I'm realizing that with all of this, uh, computer power, that man, I, uh, I, uh, um, yeah, things are going good. Tinker going real good. I have all the parts assembled there, but I realized that I got the wrong, uh, case for it because I'm not building a server rack. I'm basically just building a few computers for, for my Houdini render farm, um, but I got the wrong case size because, you know, it's been a while since I built a computer. I used to always do it. And then I kind of fell into, um, people doing the custom water cooling solutions because I was like, I don't really want to fiddle with all that. So I'd rather get it from a company that's gonna do it. But for these, for these things, like the prices are so stupid for like business level stuff. That's why businesses don't make any sense. I don't know how business should ever make money when they pay like business prices for things because you got like home internet. You pay home internet. It can be like up to $100 for good internet. And that's, that's pricey. A business internet line starts in the thousands. And I'm like, wait a minute. Where does this jump occur? Why, why is this? But that's just what business pay for. They just do that. And, and other people charge businesses like these exorbitant rates. And it's, it's like, it doesn't make any sense because the price of these, these big workstation computers from a, from a pre-built manufacturer, like there's a markup for pre-built computers for consumer level hardware. What should be like a $10,000 computer if you bought all the parts in a very good computer? So I'm going to mark up up to $50,000 for, for the entire thing to be assembled. If you look at some of the higher end workstations, it doesn't make any sense. So I'm building it myself again because I've done that before and it's not much different. But I got the wrong case because it's an SS, SSI-EB motherboard, not an EATX motherboard. And even though the case says it's EATX and the motherboard I bought was listed as ATX. Have you tried the EATS motherboards? But what if we had in our movie theater? We had a LAN area. We had a little pods where people could do like if they wanted to record YouTube videos or stream or something. We had little rooms segmented off where there was like one computer with a camera already set up and a light already set up. And also all those computers are part of my render farm. Yes. And we can farm Bitcoin while they play. No, no, no. That's not, you can't do it on normal hardware. That's not an efficient way to get Bitcoin. Not with that attitude. Oh, that's why you're the handsome one. Ah, do you want to get me Bitcoin? You just don't know it. Maybe. I don't know. I don't know what to believe anymore with you. But, right? Isn't that a great idea? It's the best idea. No, well, I mean, I do have little things that I've been working on, but you know, I don't want to bug anyone down with my really cool ideas and things. Is it lenses and/or render farm? No. No, no, maybe you'd actually appreciate it. Maybe I actually appreciate it because it's only a adjacent to it. I want I'm trying to get back into water cooling. So when you're building a computer and this, this applies to people out there, you have, you can have the option of water cooling components. And it's kind of like a, it's a higher level way of building a computer, but it can yield it like a lot better results than just air cooling. Because if you have a bunch of fans in your computer, the computer that I built for you was air cooled, you know, it wasn't water cooled. There was just like a big ol' air cooler on it, I think unless I did put it in there. Probably an AIO, didn't it? That one with the thread ripper was the last one that I built there. Oh, maybe not. No, maybe not. Yeah, I have no idea. But either way, I've been getting into water cooling again because there's a lot of really funky new developments in the world of water cooling. And I've really wanted to try immersion cooling. Have you guys heard of this? I have actually. No. So immersion cooling is literally what it sounds like. You take your whole computer and you stick it in a liquid. You go with, and that's okay for it. That's okay for it. It's so good for it actually. It has to be the right kind of liquid. Yeah, yeah, yeah. Can't just be spit or piss or something like that. Yeah. So you get a special fluid that's called dielectric fluid. It's non-conductive. So that when it's in there, it's not going to like zap. Because the thing is water, pure water is actually not very conductive. It's the impurities and water that cause it to become conductive and that can cause it to short out and then it'll go could bloom. So you need a pure or whatever this fluid is. You can't have like dust getting in it or anything like that. It's take some cleaning required and there's some extra things. But the way you can do is you can take your whole computer, really dense, like get the density way up and stick it in a, in a vat of fluid and it'll circulate the fluid and it'll cool all the components in it, right? Do you run the computer while it's in there? Yeah. Okay. Go on, expound. So it's dielectric, right? So the fluid is entirely non-conductive. So all of your, your plugs and stuff in the computer, you power plug. It's in there. You know, it's, it's powered. It's got all the peripherals in there. This is usually for data centers and stuff like that. But there are people that, you know, their home computer, they basically get a fish tank, right? And they just put their whole, they build their whole motherboard and stuff inside the fish tank and then they fill it up with this fluid and you'll see it. It's pretty cool. I can actually pull up like a, uh, because I'm picturing like your tower, you just shove it in a kitty pool and then all the wires and stuff. It's like, does it just like bend up to come out of the tank or they're like, how do you turn it on? If it's in the tank, how do you keep it from getting impure? It's real tricky, but this is it. Okay. So like the tank is also the tower. It's not like the tower in a tank. The tank is also the tower. So this one's trickier. I think this one is at a higher risk of problems because the top is open, right? So you see the cables going into the top. The back of the motherboard is at the top of this fish tank that's in front of everyone. And so the plugs go on the top. Now the problem with this is dust and impurities can get into the fluid this way. It's not sealed off. Ideally, you want something that's more sealed. And the other issue here is there's probably not as much circulation. I think this is, it's, uh, it's just more of a kind of a funny build that actually looks like a fish tank. But in industrial applications, you put your entire server rack inside this fluid, you can get the density up. You just need to circulate things and it's weird. It's super weird to look at. Your mind goes like, that's not right. But it works. It works and it works well. It cools things really efficiently. How pure are the rocks in the fruity pebble? Oh, essential. Essential for that. Yeah. You want your 90s ass color scheme glow in the dark rocks at the bottom so that you can really get all the gaming juice out of your components. Yeah. So I'm looking into this and I'm probably going to break some things for a computer or for a server size stuff for server size stuff. But the problem is, you know, everything server size takes a premium. So I'm trying to like see if I, if I just get it big enough, you know, horse drinking trough if I do that. How annoying is this to change out? Like, let's say a part fails. You have to replace a part. Pull it out. You shake it off a bit. I mean, it's about as annoying as like you can imagine it is. It's, it's literally dunked in a fluid that's probably slightly toxic. But like, can you put your impure fingers in this liquid down to replace the whole liquid? No, you should wear like gloves and stuff. Ideally, you know, you're doing this in a very professional way. But, you know, I'm not the most professional person. I mean, they make like filtration systems. There must be a type of filtration that you could use to filter your dielectric juice. Is it expensive? Is dielectric liquid water? Whatever. Because it's a special thing, right? You don't really buy it in 50 gallon barrels. You buy it in, you know, smaller containers than that. You know, like paint where they have that special paint that's so dark. It's like only 1% reflectivity. Sure. That is very expensive, but you can get black paint that is like 2% reflectivity. Still extremely dark for way cheaper than that, right? It's the same with these fluids. Some people have proprietary fluids that are crazy expensive that they have patents on and they sell because it's that much better at non-conductiveness or non-cooling or cooling capacity or stuff like that. And so these companies want to sell them at premiums, but you can get there is dielectric fluid that you can get that will not be as expensive as expensive. That's cool. So if everything blows up, a lot of my things seem to like have a high risk of electrocution and explosions, but that's how you know you're really, really starting to get somewhere. You're also starting to push your your drown man's saga on yourself because like your power washing on your roof and now you're getting liquid on your computer, you're starting to really toy with liquid and as someone who deals with liquid, it's a dangerous game. I had a crazy idea to what if I pumped air through my pool, you know, because I thought Linus, you know, when he did his pool cooling for his whole server, where he ran tubes, his problem was that he did liquid cooling underneath the pool. I'm like, what if you just pump air through the pool and then blow that air in and you got a cheap air conditioner, right? Yeah. Yeah. Sure. Yeah. Yeah. Is there okay, perpetual energy? Not perpetual energy. What if is there a way that you, you know how your water has is pressurized, right? So you got water. What if, what if I have my water pipe, I build an illegal extension to go up really high? Have it pour onto a giant water wheel that's going to generate electricity, but at the bottom of it, I just have it pump back into the pipe where it came out of. That way the water company never knows. They never know. Yeah. No, they have, they don't have anything for that. They have no idea. That's how it works right. I just have the open pipe at the bottom. It just, it funnels right back in. It would work. I just put the negative pressure terminal. Yeah, I just put it in reverse. Yeah. I have some questions about it, but I'm just going to not ask them and assume you're correct. This episode is brought to you by T-Mobile 5G Home Internet. You already know that T-Mobile 5G Home Internet is ridiculously easy to set up with a price that doesn't feel like you're getting scammed every month, but here's something new worth noting. T-Mobile now is the fastest 5G Home Internet according to the experts at UCLUS speed test. No, seriously. Their 5G Home Internet speeds are officially the fastest, which means that when you're downloading all your movies that aren't mine, every single one except for mine, you'll get them at the fastest speeds possible, along with all your video calls and uploading files and maybe even streaming something in the background, you know. And yeah, it is still a great value. Back by a five-year price guarantee, so your build doesn't slowly creep up when you're not paying attention. So if you want the fastest 5G Home Internet, with a simple setup and savings that actually stick head over to T-Mobile.com/HomeInternet to check availability today, price guarantee exclusions like taxes and fees apply fastest based on UCLUS speed test intelligence data a second half 2025. All right, it's reserved. So I got a mystery. This isn't the episode. This is just a random side that I want to talk about. So as you know, I've bought some refurbished computers, right? From Dell. And I had them running. I was running a little render farm. I've been doing small scale experiments with these computers as I tried to build out what I'm going to do. So I had five computers running. And for weeks now they've been running fine. They've been working perfectly. I've actually done some experimentation with underclocking or undervolting the graphics cards. Yeah, they got it. It doesn't matter. Two nights ago, or yesterday morning, rather, I woke up all five computers that I had running were offline, all five of them. And I'm like, well, that's strange. Not one of them. I load up each one of them. I load up the first one and I discover, oh, the boot drive is corrupted. It cannot boot. Can't boot windows. It does itself diagnostics. All the hardware is fine. It can't boot windows. Can't boot at all. I can't even find the boot drive. So I load up the next one. Two of five. Same thing. Boot drive corrupted. And here's the thing. I don't know if me even talking about this is painting a big target on my back. Oh, it definitely is. But I'm going to keep talking about it because it's just fascinating. All offline at this point. Third one. Same deal. Same exact problem. Fourth one. Fifth one. Boot drive is corrupted. What time was this that you found this? I don't know what time they all went offline because I couldn't find any logs or anything like that. I didn't have anything robust to set them up. These were just render notes. They didn't have anything on them. They were just running two pieces of software basis is all they were running. So they were technically connected to the internet. I didn't even have monitors connected to them. So I had, you know, I was remodeling into them just to even get there. But I had to like, I have a little like clamshell KVM so I can plug into them and see what's going on. And I'm like, that's super weird. What are the odds that five computers in different parts of my house all running the same software. But also other computers in my house were running this software as well. All of the only the Dell ones shut down and got corrupted. Can I pose a theory here? Yeah. What's up? Okay. Do you remember the story we just told the bird with chipmunk wizard theory? What did we talk about yesterday on this podcast? We were recording. Talked about Adele and Boots. Go on. And now you're Adele's won't Boots. Oh, okay. I'm just saying put the 10 foil hats on because what day did that issue occur? Oh yeah, during the Adele Boots day. Bob gave one hint all day. You know that it was Boots. Whoa, whoa, whoa, didn't apply to this. This is my fault somehow. Remember that song? Never mind. I will find a render farm like you. Adele. All right. So this is very strange, right? So I contacts Dell customer service and I've got I've got a customer service gripe to pick because. All right. Clearly this isn't normal, right? No, that's pretty weird. It's very strange. My only guess is that it was some type of malware that somehow got into the computers and there's again, these like maybe painting a big target on my back. I have no idea. But something got into the computers and corrupted the boot drive. It's very easy to cause these things to become unstable if you like delete system 32 or something like that. You can basically render a whole computer by doing that and it would just cause a bunch of problems. It wouldn't it would not be able to boot anymore. It wouldn't be able to repair itself. I'm not saying doing that would do that, but you know, a lot of things can be done. That just doesn't have a lot of value to the malicious party other than fucking your shit up, I guess. Yeah. I have no idea either. So I reach out to it like Dell customer service and I knew something was wrong when I was being answered by the Dell social media team, which was a little strange, but I hoping that that's Dell's customer service on the social media team and I go through this all day. I'm like, this is weird, right? Can you help me find it? Figure out why this is happening. I was like, okay, let's troubleshoot it. So they run me around for hours and hours doing all these troubleshooting tasks that I keep saying, like, yeah, I've done the basics. I've done your diagnostics. There's nothing wrong with the hardware and then it goes on and on. They keep leading my like, you're going to find out why this happened. Like, no, let's just do the next step. You're going to find out why this happens, right? Let's just do the next step. We're going to find out what happens, right? All right. Create a thumb drive with a Windows installation on it and I'm thinking in my head, I'm doing anything to not do this because if I format that drive, any evidence that of what happened on that computer is going to be erased or become much harder to find if I do a fresh install of Windows on it. And I keep asking them like, just and they're like, please just do the next step. We are going through this. This is the process we need to get in like, okay. And so I reinstall it. Do that. And then I tell them like, okay, I'm in Windows. Now what? And they're like, great. Let us know if there's any problem. Keep an eye on it. Have a good day. And I'm like, you mother fuckers. You sons of bitches. You had no intention of finding out what went wrong with it at all. I was like, just take this computer back and look at have have compete. There's such a thing as computer forensics. You can look into the hard drive and find out some logs. You can dig in there and you can, but they had no desire to do any of that. They were never going to do it. And they strung me along the entire time. Did you do all five? Or did you say do you have any that are still? No, I only did one. Okay. I only did one because I have all five with the same problem, but they would not admit. They would not admit that five going offline for the same reason all independently of each other wasn't weird, right? And they kept thinking like, it's just like a random software corruption. I'm like, you gotta be kidding me. The odds of random software corruption occurring across five machines at the same time is statistically most likely improbable or not impossible and nothing's impossible, but whatever. It seems much more likely that this is some kind of security vulnerability that was exploited or something like that. And it's only these. It's my other computers are fine. Only these. What's up with that? It was kind of brilliant. Now they've got you as a complicit party and covering up the evidence. If you do get a forensics as a specialist, they're going to see that you covered up one of the five. What crime is being? Okay. I don't know, man, but it's pretty bad if Dell's getting you involved in the cover up. You're right. And they got the social media team involved. So that way the moment you try to say to them, they're like, they've already got the posts ready. Markiplier covers up corruption at Dell. You're right. Inside job. That's it. That's it. That's the one. That's it. And look, it's like, I know what can they do? All I wanted was them to just escalate it to the next level of being like, yeah, this is weird enough. Maybe you should look into this. It's probably nothing. It's probably something like as I was running them, like it was some port was not forwarded correctly in the computers. It allowed someone to get in or a virus to get in. I'm not saying this was like a specific attack. It just seems like this is malware riding across the software for the render farm thing that I was using that allowed like some piece of malware to get in deleted a system thing and boop, all the thing go. Yeah. It's more likely that that happened, but it's also worth exploring. I can't use these computers if that's possible. That was my point to them is like, I can't rely on these to do what I need them to do if there is Something like that in them. I don't think it's the software I'm using because I'm using it across my other computers But maybe again, I mean it might be I like a driver compatibility Where it's the software and some specific thing in the Dell machines and they all have it because they're all the same machine or something But yeah, exactly now it's weird to corrupt the boot disk like that is like not what would normally happen is They would just blue screen and then you could you could get it to post and maybe you continue having issue You have to do drivers or brainstorm or whatever but like yeah, I'm not an extra but as far as I know that's so much Weirder outcome. Yes. Yeah, that's not something where it's like whoopsy doodles and then your whole boot drive is just corrupted That's a different kind of issue. It's very strange You guys are basing the too much stuff on facts here I think Bob's earlier theory about the wizard chip monk He's trying to get his revenge because you caught him like practicing his bird, which wasn't perfect So he's like half chip monk and then he fully went chip monk and he's like fine I just want to be a bird fuck this guy. It is strange and then he followed you home as a chip monk and attacked your computers to get back at you I did think I was almost positive that when I first saw that bird it was a chip monk first That's what I saw in my mind. I might have seen a shape-shifting chip monk. I think Bob was originally right Just a glitch in the matrix here. Let me give you let me give you a Wade solution here. Hang on. Okay. I'm ready All right, Mark. I know how to solve this problem. It's not the Dell computers themselves What you need is a Costco-sized jug of lighter fluid and then a lighter to light the lighter fluid And then what you do is you take all of your computers and you set them in a smaller area Small enough where you could draw a circle on the ground around them in lighter fluid And you want to use like the whole bottle like as much as you can't you might need multiple walls I don't even know, but you basically want all of the wires all of the networking stuff all of the computers all inside this circle And before you turn them on to do anything you light the lighter fluid So you've got a wall of fire around all of your computers. Those are important to protect you I know this about technology The firewall will keep you safe, Mark. I've heard of this mythical firewall. I've got an alternate Okay, wizards are sometimes like the male equivalent of witches and we know from even dating back to like the 1940s or even earlier Witches don't like water. So if you pour water on each machine you'll melt the witch out of them. Oh, oh, okay I have a pool. Can I just throw them in the pool? Yeah, I'm theory. Yeah. Okay. All right. Cool. No, you know what Mark? You were telling us about how you wanted to do water computers in water, right? You were talking about that. We talked about that What you do is because you said the water needed to be a special kind. It needed to be thick or something Get a bunch of the leftover thicket that you have from redactin Get a get a bathtub filled with thick water and then just submerge all the computers in there That'll melt the witch out and keep it out Plus then they'll be real cool because of what you were saying about water cooling. Yeah, you you listened or Get a snake big enough to eat a chipmunk or a small bird because clearly something the wizard can turn into Mm-hmm release a snake into each computer. So that way if the wizard comes back, it's ready to be gobbled up Mm-hmm. That's good defense. That's great. If you could find a species of snake that can survive underwater You could combine some of these solutions. That's true, too If you can find a fire that can burn underwater. Are you drawing these down for later? Yeah, yeah I'm really I'm really keeping track of everything that we're doing today. I appreciate that well That's I really appreciate you guys helping me out and I appreciate being able to rip about something else someone out there It's like oh my god. Mark is under attack. I need to get him a message. Don't totally happen. Yeah. Yeah, yeah definitely anyway Glauber saw g-l-a-u-b-e-r Apostrophe S. Glauber salt remember the name. It's gonna change the world Also, well, that's what for us discuss. Just chucking it out there. Glauber salt. I feel like generating heat is not as much of an issue in the world as Disupating heat and avoiding excess heat generation is You're it. Yes. Yes. Go on. Go on. Well, this doesn't seem to want his Glauber salt used in like heat exchangers or something Like what are we talking about like is this cause almost almost so here's a thing you boil water You put your Glauber salt in it and that mixture alone creates a concoction that would freeze at 90 degrees Fahrenheit That's scientifically true, but that doesn't mean it's cold. Now now here's the thing if you add Regular table salt to your mixture you will lower the freezing point of that mixture same way as if you put salt on ice It'll start to melt because the freezing point is now lower Okay, it should be you can if you put a certain ratio into it I don't remember it off the top of my head. You can create a mixture that will freeze at 70 degrees Fahrenheit Exactly what most people might consider room temperature and if you put a zanthan gum it'll turn into a gel So it's easier to you know handle than just like pure liquid and then you have a gel like a gel pack that'll freeze at 70 degrees And what that takes advantage of is the fact that phase change It's called a phase change material. It had it takes so much more energy to get it to change its phase Then it does to get it moving a degree so it'll stay at 70 degrees I'm way longer and all this circles back to the concept of my render farm So I figured if I make enough of this shit and stuff my render farm full of it it'll stay at 70 degrees because there's so much ice but warm ice Glauvers ice so I don't know enough about thermodynamics to like really contribute to this But what you're saying sounds right so basically what you're saying is you would you would put all this stuff where the the phase change point Is right about 70 degrees. It would constantly be in and out of phase like it would be kind of part liquid part frozen because the Racks would generate heat so they would heat but then the phase change of all the Glauvers out salt solution Goop it would extract but they're put but Glauvers out is still still a poor Thermal conductor, right? So would it really when mixed with water it becomes as good of a thermo conductor as water is so by itself So here's the thing right the weird thing about 70 degree phase change material is if you think of the world we live in at night It usually gets below 70 degrees most places even in the desert it can get down because the sun is not shining and it's cold It will freeze overnight if it's in an environment where it can for free anyway Glauvers salt is cheap so the thing is if I get an industrial cook thing pot Wolf an industrial boiler pot. I mean I do industrial boiler probably implies that it's a very large pot for boiling stuff I don't know. Hey Siri, what's a kitchen? So Mark if Glauvers salt is such a magical cheap material that is so simple to implement Why hasn't anyone else done this? Well, I don't know. That's fair. I don't know what it is about it But ever since I heard about Glauvers salt, I can't stop thinking about it It's not even that it's the Glauvers salt really. It's like I don't know if there's a name for the mixture of what it is afterwards But that science like you you can do the science. I just need to figure out a way to make a 422 kilograms of it I think the thing for me that I'm sticking on is the point is not I'm with you that it freezes at a such a high temperature that basically freezes for free But then it's a solid right? Yeah, or it's a it's a goof anyway The way water the way water cooling works is the heat is not dissipated just because water absorbs heat It's dissipated because water absorbs heat and then is cycled away from the electronics. Yeah, but that's not what I'm doing If I put it in the room and then put a fan on it and blow the air But you're but but heat heat saturation is a thing the whole the whole thing Well, if you don't shut down server farms overnight you part of the point is they sort of run 24/7. Is there a downtime? It'll be fun. It'll be fine. Are these like you change out the Glauvers salt packs or something with no That'd be too heavy. Oh, so Mark. What if you make your Glauvers salt solution your high high freezing point solution with Mineral oil and then you submerge your stuff in a Glauvers salt mineral oil tank So that then you don't have to cycle it the phase change occurs and as the heat builds up the phase changes out But then it cut like it frees it. It's like an expanding contracting water cooling rig I love that and I think that's incredible. However, I feel like it would mess up with the freezing point if it was mixed in Mineral oil. Yeah, I have no idea what the freezing point of mineral oil actually is. I don't know how that's different I know it's not water. So what I was gonna do is I'm gonna make it at an evaporative cooler So I'm going to make master car and I'm gonna buy a bunch of duct work And I'm gonna get some ventilation fans And it's gonna build a little evaporative cooler and the cool air from there It's gonna blow over the Glauvers salt mixture and I'm gonna have a heat exchanger inside there That's going to be like basically encapsulated by all of the Glauvers salt mixture and then some working fluid running through that Going to another heat exchanger inside the the render farm room and a fan on that blasting all of that delicious cold Free air conditioning baby Not free it cost a lot to build it and all that stuff Yeah, so it's it's essentially a normal water cooling set up with extra stuff that's definitely gonna go wrong What do you mean it's Glauvers salt? It's Glauvers salt? You got to have faith in the Glauvers Are you getting custom like heat sink blocks for all each of your little chipsets and cards and things or are you just sort of setting it near it? No, I'm not you know you keep thinking about a slather my electronics and Glauvers salt. No, it's just air. It's just air Okay, it's just air circulation. You're just using the the salt solution to cool the air to a crisp It's a cold battery. It's a battery of coldness a room temperature coldness. How is small for you guys? How's it been you know how it be every day another phase? every day I rise grind full flat on my ass this render farm is never happening I swear to God everything I've tried I I fuck you not it is not going good I thought I knew computers I've assembled three different computers let me just say don't buy cheap shit on eBay half of it doesn't fucking work no you don't have any support you don't have any backup you can't return it because it's coming from a city I can't even pronounce and then late the latest adventure was in my adventures with water cooling right I've never built a water cool computer with a custom loop before but I was like hey I went to engineering school I can do this and I I set up everything I made sure that I actually got parts that I could verify like it counts still on eBay still discounted but there was like someone I could communicate with and I got parts and then I sourced all of my water cooling stuff from a company that's actually here in L.A. who was very helpful I asked for the lot of advice in the gaming guidance and I put it all together and I turned it on and I started filling the reservoir with liquid and I go through two bottles and I'm like wow the guy only said I need one this is a thirsty machine and when I get through the second bottle this pump just keeps sugar in this liquid I'm like damn it's going and then I finally notice it's a clear liquid so I know it's so dense with components that I didn't notice the entire bottom of the case is swimming in the fluid somewhere in the loop there was a catastrophic leak and it was just gushing so you go through the pump you never occurred to you that no point to it start coming in the return to the reservoir you know now that I'm thinking about that it should have been obvious but I never built one before so I was like wow this is crazy so you doubled the amount it recommended you're just man this is thirsty I don't know the server right now I took all the electrical components out of it besides the power supply I just so happened to grab the power supply that was makes a clicking sound with the fan like I don't know what kind of trailer all of my shit fell off of that I bought don't buy discount shit on eBay it's not worth it it's not going to be worth it it won't get you it's so many nightmares I've wasted so much money I should just trust this company because holy shit I just gave you points for the drowned farm also for listeners out there he pointed to a shirt that says stagger dynamics yes stagger dynamics oh you like the listeners now and what what sucks also is like I've been I've been doing this I run into all these little problems so this is actually a 4090 he's holding a 4090 4090 graphics card that I have unmounted from its previous cooler and put this nice thin one slot cooler right it's one slot that's what it's it says there's only one it'll miss a little wide it's no it's one slot this son of a bitch is not one slot by two millimeters it was all everyone said this was a one slot cooler the company that made it then came out with another for this exact board the reference board that this is a more expensive one that is actually one slot and I contacted them and I said hey is this available and they said yes but you have to order a minimum of a hundred Jesus Christ a hundred I said I didn't respond I haven't responded how expensive are these if I may have the 4090 themselves are fairly expensive so this is I mean the one slot one the one slot thing I got for this thing was a hundred and I believe 60 plus tax and something like that so 160 bucks for the cooling block itself but if you know a 4090 it goes from a three-slot card down to one and I was like that which would be awesome it would be awesome if it fucking did the thing that it said if you buy a hundred of them it will you're right I sure will the solution is to buy more stuff mark than the new stuff you buy will work probably and if it doesn't then you could just go buy other new stuff and that'll be the stuff that works and it's like all of this is an attempt to save money everything I've done is an attempt to save money I have lost so much money well I haven't lost it right because I could sell this again and that's fine it's just work to do that the the other computers that all crashed at once Dell also extremely unhelpful and then the episode that where I talked about came out suddenly they wanted to help so badly and I'm just like I'm interesting whatever guys you can't have eggs without breaking a few eggs yeah so I can technically sell this and this is still useful to people it's still useful to me like it it's still like okay you can put this in a loop this still works fine yeah that's a normal 4090 right that's not like a like a workstation one or something weird where it won't do video games or computer shit perfectly normal 4090 and and actually I could probably sell it for more than I I paid for it because I've modified it in a way that's valuable yeah with the water block that's actually pretty sweet which is so you know anybody if I sell it it's not one slot only if your slots two millimeters to thin like mark slots right one to get girthier slot if you got the wrong slots mark why don't you just increase the gap a little bit I don't really have a story to tell other than we're so back baby buying random computer parts off of eBay is the sixth way to do everything you know how I say that it wasn't it is it we're back in oh shit we're so back we're so back because I have not had one success wait that sounds bad I have had not just one success I have not had one success not a single goddamn success oh I've not had a single I've had a double I have now two functioning dual processor computers that I built myself one of which is right here and I'm gonna show you guys I'm so excited for this Jesus Christ that's enormous they're so portable Marcus lifting up a very large looking computer for the listeners it looks like he stuffed a wig inside oh wow that's that's one of them that's large that slides into our server rack of some port some time yeah so it's a server case it doesn't seem to be plugged in to anything how you know it works I plugged it in before I just don't have it on right now because in this room if I have both it and this computer on it'll blow circuit oh fun how many how many did you build total so far to this one is dual processors that I got a fee bay are those like xenon processors or whatever like server ones yeah so these are they're actually these ones right here so these oh I got these for like less than 10% of the price of what they retail for I don't know how and I don't care well just wait two weeks when I stop working no no didn't you build like five of these new to send them to begin I'm sorry what to look at them you like help me are you trying to say stagger dynamics our favorite computer company God fucking damn it dude I hate words yeah that's it didn't they didn't they help yeah they're helping okay so is this a separate computer one of the ones they looked at this is a separate one yeah so we're working in tandem I'm just trying to follow the timeline yeah so what I what else I did the second one I have an eight GPU server out in my storage there was no way that I could fit eight 4090s unless I change the cooling block right so what I've been doing is I've been putting these water cooling blocks on them the ones that were like two millimeters too thick or whatever it was yeah those are too thick but that's not it it's not too thick for that one water cooling that whole thing would take a whole thing so what I realized is the actual 4090 I have if you take off the fan off the top of it the actual radiators less than two slots there's no fans on it but it's less than two slots there are these stands on the sides of it that go up past two slots all I got to do is cut those off that I take some scissors and I'm like it must be plastic not plastic metal grab a dremel put the metal cutting head on it well metal dust everywhere all in the electronic components that's really good for electronics yeah airbrush blasted out first one done not gonna do that again for the seven others go to Home Depot grab tin snips because I'm like I bet they're aluminum not aluminum steel so you squeeze really hard with the tin snips and you bend it back and forth it'll break eventually they got these real jagged scrappy posts on the edge of your cheapy oh so sharp so razor like after it would break off it wouldn't break off like it would limply fall over it would go and they would go flying in a random direction it was crazy so seven of those went by and I fit eight GPUs into this server and at this point I decided to Google if I was doing this right nope I tried to calculate the thermal capacity of these cards if they were just purely passively cooled what the airflow would need and then I couldn't find an answer so I said I bet it'll be good enough. And I turned the server on. And it worked. It fucking worked. I was watching the temperatures. So long as I kept the power of the cards individually, you can limit the power. They're normally 450 watt a card, which at 8 cards is about 3600 watts or 3.6 kilowatts. But if you limit the power to a reasonable level, like 33% of what it should be, it works. Oh, though, that's fine. You don't need to use all the power. Why not just use three cards at full power? No, the only reason that I have it at 33 right now is because it'll blow a fuse. But the reason with 4090, that math weighed a lot of people on my member mixer pointed out it doesn't math because the 4090 is already an overpowered card. So at 450 watts, it's actually not as efficient as it could be. If you lower it and there's tech YouTubers that have done analysis of this, you can lower the power target to 60% and only lose 10 to 15% of performance. So you gain a huge power saving up to 33% for about 10 to 15% performance loss, which is great. It's actually phenomenal. And it's because the 4090 was overpowered, they put the power target so high. It's like it's already overclocked as the design comes in. But if you bring it down, you'll get barely a dip in performance. If you're gaming, it would be imperceptible. If you're doing rendering tasks, it's the matter of like a few seconds, like longer. And you can make it work. And I'm making it work. The render farm is happening. Yay. What are you going to render once it's all done? What are you making? My movie stuff like balls and water and stuff. I'm going to make the greatest, most high fidelity 3d porn the world has ever seen. Yes. Yes. I might still have the computer tower that has what was that game? Yifalicious on it somewhere. If you want it back. I mean, what's porn? Is that Jake Gyllenhaal's new movie? Well, congrats on your functional render farm. How many, how many more are you going to make? Like how many, how big will this get? So they stagger dynamics is working on six computers that have are going to be like the base render notes. I've actually been talking to the people at Houdini so that they can know how best to set this up. Sagger dynamic can know how best to set this up in terms of networking. But I've also been working with them because I'm trying to get more CPU power. CPU power is like the thing for fluid simulations. The GPUs are great for rendering the actual final images, but the CPUs are necessary for any kind of physics simulation stuff. There might be a time in the future where that's not true. But right now you can get CPUs a lot cheaper than anything else because of the way that it's it's structured for simulations right now. And because everyone's crazy for GPUs right now. So that's why the professional level GPUs that actually fit in those cases are way too expensive. So if I can fit a 4090. Oh, the back of the the back of the server can't close because the 49ers are too tall. It looks real jank, but it works. That just includes airflow. I'm pretty sure. It sure doesn't, especially when you got only passive legal cards, but I'm going to I'm going to like tape off the sides where like the triangle of it goes up like this. And and I'm going to tape off the side so it actually does kind of still go towards the back. Oh, is it? Oh, it's supposed to be drawing and that opens up a big leak in the I see. No, all you need is just get some box fans and just aim those right in there. You're right. You're right. We're going to spray bottle to get too hot. Just yeah, get the stuff you're allowed to put your electronics into. What's it? What are we? What was that called? Dialectric fluid. Just spritz it with a little dielectric juice. Cool right off. So you know, I just know how I'm trying to do this stupid server rim render farm thing, right? You have my genius. I was like, I'll buy some more solar panels because right now, you can actually get them for fairly affordable rates. The prices have come down. Dude, everything is so cheap in Mark's world. What's where do you live? That every every servers are cheap, processor are cheap. He's Markiplier. So he has his own version of eBay. He can log into where everything's just discounted for him. If you buy refurbished things, you you can get him at cheaper prices way cheaper and they're only partially used. Solar panels are good for 25 years and they lose, they have a rating of how much power drop that they'll experience over a period of time. The quality is so much worse though. I bought used dinner and it was terrible. Yes, I'll have the refurbished ribeye and the potatoes off that lady's plate over there. The thing is like if you get things, you get them refurbished and you do you put some DIY spirit into there, you can get things at pretty good rates. Now that being said, I am I am spending more because I'm investing in building something. That's why I'm doing it. I bought solar panels, right? And so you can read online that this solar panel is like 65 inches by 44 inches is about the dimension of this panel and I'm like, okay, that I got that size in my head. I ordered you can order them by the palette that you get bulk pricing and I'm like, okay, I'm at that. I think that this area could cover about 28 to 30 maybe solar panels depending on how I arrange it. There's a palette here says 31 refurbished high quality barely a few years old. I'll get it. That palette weighs 1,680 pounds and my hero of a FedEx freight person got it delivered down my driveway. So how? Look at one guy is like looking at his chart. He's like, oh no. Your driveway is treacherously steep. I cannot imagine. I was ready. I told them. I was ready. Yeah, just put at the top. I'll bring them down one at time and even then if I had to deal with that, I would be looking at a nightmare because 65 inches is five foot six. That's taller than you. No, no, it's shorter than me. Okay, it's it's half an inch shorter than you. You're right. It's close enough where I had to think about it. But they're very large, right? If it was in this room, the palette would take up like 70% of the room. Oh, Jesus. That's enormous. Well, yeah, 31 things that are basically five and a half feet by what four feet ish? They're almost as tall as Mark and wider than I am, which is to say, extremely wide. Yeah, they're huge. They're enormous and I have to haul them up into my roof. It's really nice of you to put all that power right back in the grid and not get any for yourself, though. Yeah, you're not going to hook those up to your house, right? You're just going to keep, you're just going to send that juice into the grid. Yeah, that's what I just do in my public duty, you know, just really, really got to do it for the people. I imagine some poor guy working on a power line and then Mark just plugs this thing in like a Christmas tree and all of the power goes flying. This guy like, uh, hermit scream. Well, we'll help. The line bulges as the tsunami of power comes down the line at him. Like a cartoonist is like, oh, one of the guys like, oh, no, I gotta get down. Lightning bolts wearing sunglasses are coming out the sides. Some of them have guitars playing rock and roll songs. That was small talk. That was all small talk. That's what I did yesterday, Mark. What do you think we're talking about? You bought 10 kilograms of some unknown metal. And I can't wait for the subreddit post. I can't believe Mark bought 10 kilograms of silver. It's how you turn on the render farm. You have to put it in the slot. It's going to operate. It's very expensive. Oh, yeah. They got me. And you pull a big lever and then you render. That'd be great if I could set it up that way. I'd like to render a ball floating in water. Chunk, cherry, cherry, oh, bar. A dummy crashed. Fuck. I'm doing good. My render farm is rendering. It's very powerful and it takes a lot of electricity. You're in the future now. So maybe it takes less. No more. But you've got your solar all fixed probably, right? I didn't. I bought a 30 foot pole. But then I hesitated or I didn't hesitate. I didn't hesitate at all. That's the wrong word. Forget I said that word. I underestimated how unwieldy a 30 foot pole is because when you have extended all 30 feet, it's not only extremely heavy at that mechanical this leverage disadvantage, but also the the the pole itself just goes, you know, just droops all the way everywhere. And so it's really easy to whack it on things. And you know, solar panels aren't exactly the most. They've got things to help with droopy poles. What? That's sponsoring the middle thing. You don't think? I don't know. It's a future man who knows. I've been also working on the render farm. And I know I've disparaged Linux users before. And I will continue to do so. It doesn't make a goddamn bit of sense. You open source bastards. I hate it. I can't even log into my NAS from Ubuntu. I tried to install Fedora and it blew up on installation. And then I then I I installed Fedora's server client. And I after the full installation process, I'm greeted with a DOS command prompt. And I'm like, I left this behind in 1994. The commander. So yeah, I'm doing that because the alternative is Windows server. And I don't know if you know how much Windows server costs, but it can cost up. upwards of $7,000 per license. Do you need a license per machine that's on it? So you don't necessarily need that one. And I don't know if that includes multiple licenses for multiple computers, but it's like with servers, I have many small versions of it. And I don't know what, I don't know any of it, but I know that Linux is free, so I don't have to deal with that. But also it's. It's Dracula. Oh, I love Count Dracula. Bluh, blah, blah. I audition for the movie. You think it would be good? Bluh. Bluh. Bluh. It's like a Christopher Nolan drag. I go, we're going for dark and gritty. Dracula's been through a lot. And whenever you're ready. I'm Dracula. Bluh. Jared Lido has Dracula. So my small talk is the same as it's always ever been. It's just about the render farm. I have solved water cooling. Wow. Don't talk about cloud bar salts again. Don't go there. Why not? I don't need to right now. Right now, I don't, because it's winter, so I don't need to worry about cooling too much. Water cooling is a lot like factorial. I was just playing that last night. Yeah, I know, right? So, factorial is about like optimizing your distribution and automating things so that it all flows in a good direction. And, you know, you can get clogs in your factorial factory because you can have things going and then it won't go out. You have an end to an out and an out to an end. And these, the belts need to be perfect. And also, you don't need extraneous belts because you'll run out of space and stuff like that, right? So, water cooling has always been a very scary part of computer building to me. And I've never done it because I always worry about catastrophically exploding my computers with water. To be honest, I did that the first time. I think I told you guys the first time I did it. I had to put it outside and drain it. Well, that's just because I didn't know what kind of plug I was using. It was an insect plug resource. I've now successfully water cooled eight servers. Damn. I'm really proud of that. I've got custom blocks while I didn't make them. But they're blocks for the CPUs. I put them in there. I created my own routing with the loop inside a very elegant way of like the tight angles and stuff like that. Because if you go to a server manufacturer and you buy their proprietary water cooling solution, it's thousands of dollars, thousands of dollars. But the block I got is 60 bucks, right? So, it's 60 bucks for that, 60 bucks for another one. And then the pieces, the connectors are like five bucks a piece. You need dozens of them. You need way more than you ever think. And if you get the quick disconnects, they get pricier. But I did it all. And it works. And I set up the, the quick disconnects properly so that if I ever need to do maintenance, I pop them out. No water goes anywhere. Like try breaks, you mean? Like specifically. Exactly. Yeah. So they're, they're just like couple and they uncouple really easily. So I pop all those out. The only thing that I didn't, I don't understand is there's some liquid physics going on that don't make any sense to me. Because I know the general principle is like you need to put the fluid in when you're filling it. And then air rises to the highest point in there. And it'll, you know, you want to vent that. So you can always fill it. So you want where you're filling it to be the highest point. And then it goes down. But what they don't tell you, well, they probably do. But I didn't read it. Is the way the, the loops can sit and go up and down. Also create like many high and low points in the loop. It's like the false vacuum thing in the universe where they say, we're resting at what we think is the base point. But actually we're going to all collapse into nothingness. And so what happens with that is pressure can start to build up explosively inside these tubes. I haven't had anything explode except where I was filling it in. So many times, it's like old faithful. I watch as it goes in. I see an air pocket come back up. And it sounds, it sounds like it's rumbling. And I have to put my finger over the thing or else it's going to push water up from behind it before the air gets there. So I push that. And then once I look at the tube and I see the air is like filtered to the top, then I let it go. It's a very strange. It doesn't feel like building computers at all. It feels like a, it feels like a car mechanic just trying to convoy the other system. It's very fun and water is going everywhere. I've ran through so many rules of paper towels. I've lost more fluid than I've put into the thing but not in the computers. You've probably spent thousands of dollars on this. On the parts. No, I was joking. Oh. Man, I'm at like negative 10 points to start this episode. It's not a great start for me. No, come on. It was funny, man. I laughed. I thought it was very funny. I was just being respectful and being quiet. Thank you. We love this episode. I've not even done anything. I've never built custom water loop stuff but I've watched lots of videos on it. That's, that's very funny. Well, see you have dry breaks. So you could use like a, like a pick or so they just sort of like open the dry break and bleed the air from the your little pockets that you have. Technically. I could, I just worry about breaking any kind of o-ring or seal inside there or jamming a mechanic or mechanism in there. And then suddenly I got like, oh, o-ring break. No. What idea of trauma about o-rings? Probably they make like a dry bleed connection like a dry break bleeder where it's like you can connect it to the dry break. And it's so the right connection. But then it's just a bleeder that opens to the atmosphere or something like that. I bet they do. One of the problems with water cooling is that there's so many different manufacturers. There's so many different standards of parts. Most of the tubing that I do is 13 is 10 13. It's 10 millimeters inner diameter, 13 millimeters outer diameter. Most of the actual threaded screw holes are quarter inch. Some things are very standardized. But all of them are tiny, 80-bitty parts that cost like five bucks a piece. And so you're like, oh, it's cheap. Ah, man, I need 96 of these. I wasn't wrong by adding up to thousands of dollars. It sounds like by the end of this. It does. But when you think of the server thing, it's like thousands of dollars per actual server on the rack. So it adds up way quicker for there. And in the end, you would still have to hook it all up and get other stuff. So it works out to be cheaper, but there's something very fun to it. It's like it's the exact same difference of buying your own computer and building, buying your own parts and building your computer or buying a pre-made computer. What if you buy your own parts that someone else builds your computer? That's cost money. Unless I do it and then I do it for free. Unless you trick Mark into doing it. Hey, you guys want to come over? No, I felt very cool because I had a pressure tester. So whenever I would build a loop, I would have a pressure tester, plug up one in, put the pressure test, pump air in to test for leaks. It felt very, it felt very official. I felt very confident doing it and it was actually really fun. I don't work with my hands a lot. And I don't know. Something about wrestling the pressure was very fun and super frustrating and sometimes, but very fun. Good work. I listened. I paid attention. Thanks man. I have a disaster. Oh, go on. So, you know, on the last episode, I was saying that my renafarm is working. I don't want to ruin the illusion for people. But you mean, do you mean like 40 minutes ago? Yeah, well. What do you practice about your renafarm? Well, I said it in there because I knew that if I got into it, it would be a longer thing. So, I didn't want that to be part of it. It has not been going as good as I may have suggested it was going. Uh-huh. Oh, no. Go on. This is an issue that me and there's, there's an IT guy that's helping me out with setting some of it up. He's doing more of the like the software side, getting things to actually. He's arranged in that capacity and so one of the persistent problems is a server that I bought, which is a density, like a high density server. It's four nodes in like a two unit thing. They're long, but it's meant to cram as much process power into small paces as possible. The issue with that is not in me putting it together. It's not in the water cooling that I set up because in the compact space, you need water cooler else. It's going to overheat. They're so crammed in there. And so you need an external water cooling solution. All that works. After all that work that I've put in and I probably put in like 100, 200 hours into just setting up those parts of it with all the water cooling pieces in there. There's still a persistent performance problem. The chips themselves are fine. They come back and they should all the checks come together. But what it's boiled down to is the specific server brand that it is has not put out an update to the chipset for those processors. And that's just something that is I don't know if there's a way to just go directly to the Intel makes a processor and go like, "Hey, you got drivers? Give them to me." Or if it's something where you have to kind of like figure out a way to program it in yourself. You got to get the chipset drivers from the manufacturer of the processor and then they integrate with the board. But there's a problem. The longer you use them, the more the performance just starts to go. And then eventually it locks up so bad you can't use it. How long is this like if you do it for a couple days or is this like 10 minutes? It's 15 minutes is about the time it starts to. That's not long enough. That's not long enough to do. I was like, "Yeah, you can just reset it every night." So I thought this would have resolved, I've done like biosupdates. I've done, you know, firmware updates. I've done every update that I can that is provided by the manufacturer of the server. The end result I think and I'm in communication to like do a big return is to do a different server platform. There's another company. And I'm not naming any of them because I don't know if one works or not. Okay, ASIS server. So I'm going to say are the ones that are not working right now in this specific use case. They have worked in other things that I've done before. A SUS or whatever it's called. And so Gigabyte makes an equivalent product that has also the hyperdense for unit thing. But I have to undo all the water cooling. I literally have to unbuild everything I've ever done. and I've built to pull it out of the commuters. Box those back up, return these giant hundred pound gigantic boxes and then get new ones in a rebuild all the bullshit all over again, which took me out days, weeks, woo! This sounds like the kind of server I wouldn't leave a tip and I always leave a tip. Give 'em a point, just take it out. - All right, all right, all right, Mark said so. - Have you thought about talking to the server's manager? - I have no manager. - That sucks though, that really sucks. - It's such a big pain in the ass because as far as I know, it should work, they read fine. When they're working, the performance is exactly where it should be, all is lining up. It's not overheating. I even thought that it might have been one of the peripheral components in the server overheating. At one point I had a graphics card in there, a very small compact one. I took that out because I thought it was overheating. Apparently it was not the problem either and no other peripheral in there because the fans are very loud. I do the water cooling also to try to quiet it down a bit. They're running it full-bore and still just crashing. I think it's just because the chip set is not updated to run with those processors. There's some kind of paging file system memories to an error that's building up over time that's causing like consecutive errors that run into each other and then eventually it freezes. - That one's a little, hey, sucks. - Nope, nope, nope. - I give myself one or whatever, that's worth. - But anyway, so that's the latest. It's not all such errors. But half of the other renaform is still working. It's just those hyper dense ones that are not working, which are the most recent addition and it's something I can return. It's just like, that's ours I don't have. Not devastating, but it's unfortunate. Just means I gotta undo a lot of work and redo it. - But it'll probably be fine at what you redo it. It'll probably work then. - Yeah, it probably will, maybe. But I do know that the gigabyte servers do have a late and more recent BIOS update than ASUS did. ASUS's last BIOS update is not only last year, but it's like mid last year, whereas gigabyte has a BIOS update that you can apply to the server as of earlier this year, which is after the processor is actually launched. So I know that the old BIOS is not current as of the processors release date and these ones on the other server are. So ideally that should work because the processors are fine. - Probably. - I never thought I'd be a truck guy. I got a Ford F-150 Lightning because I looked at the specs of what it could do in terms of power delivery. The size of the battery is way bigger than the Tesla's ever were. The range is not a lie. Those Tesla's all lie about the range. It just blew my mind. I would watch it and I would go like, I've traveled 10 miles. Well, surely the, oh, it's dropped 30 miles. All right, well, that's fine. It said I had like a 300 mile range on my Tesla. I got 180 at best. Even when I was driving like this, it's just lies. But this one I drive 10 miles and it's like the range estimate only dropped eight and I'm like, oh, thank you, oh, thank you. Now hold on, the Tesla wasn't lying. Someone took the road signs and spread them out further. That wasn't the Tesla's fault. It's crazy because this truck has a 240 volt plug in the back of it. I could run my render farm. Also this truck, this truck has more power than all of my backup batteries that I put in my render farm by three times. I want to see a Ford commercial with like a really nerdy dude driving his Ford Lightning with the render farm. It's like off-roading in the mud through the woods while like video game let's plays or like processing in the back. I could. I'm good. I have an update on my many hobbies. Which one is it 3D guns? 3D guns, ammo farm, where are we going? Look, it's about the render farm. It's been working delightfully, finally after four floor air conditioners, two wall air conditioners. That's some bitch exactly staying at a steady temperature. (laughing) But the harsh realities of operating a render farm. You built a freezer. It's not that cold man, even with all those. I woefully underestimated how much cooling was necessary. But I have another problem entirely. I got my power bill. Oh no. (laughing) Did the seven air conditioners raise it slightly? It's been up and down because we've been testing it but we haven't been running it. So the past month has been the first time it's been like running every day, chugging, you know, beautiful renders. My power bill was $3,000. (laughing) (laughing) Is your solar working? Or. - Yeah, it is. (laughing) - I bet you wish you were a man with five ovens instead of a man with seven air conditioners now, don't you? - I opened up that letter. I just, you know, they show a bar graph of your monthly usage. - Yeah. - And it's like, oh, and it's trending up as I'm testing. And then this past month, just. Astronomically. - Somehow it like comically goes off the paper. You just look at it, it's just like, you're like, what the fuck? - I don't know what the power company thinks is going on at this place. You know, this place being a bathroom. (laughing) You're gonna get rated by Swat for a render farm. - Dude, yeah, no. D.A. is gonna show up and be like, the power company thinks there's a grow farm here or something. Some crazy is happening. So it must be drugs. Oh, there's a farm. (laughing) He's like, you're gonna find seven 3D printers, a server farm, seven AC units and a bunch of 2D guns. - It's like a big wooden crate with like the rack and then just a bunch of pieces of paper with hand drawn AKs on it. - This man's planning to raid the Looney Tunes. - I got a gun safe and they're like, over there safe, I'm like, all right, I'll open it. Comically, and just a avalanche of papers, you know, mask style with all the dollars coming out. For some reason, they all have hand drawn serial numbers, but then their hand scratched out. (laughing) - Yeah, so that's my update and it's only going to, it's not gonna go down much in the next few months. - So they didn't say anything about that though? 'Cause like, when we lived out there, there was one summer where the really, one of the really bad fire summers happened and like we had all of our shit closed and sealed and we were running the AC just to try and keep positive pressure on our house because it was like toxic outside and our bill jumped by like 250% and the power company sent the bill and then they called and we're like, are you guys okay? Are you good? And I was like, yeah, there's fires and shit. Like, I don't know, it's been awful, but they, no one, they just saw your bill and we're like, all right. - Well, if they didn't call, I didn't answer, but yeah, I feel like they're probably more into that. - They're gonna think you're a render dealer and call you L-Servo. - I had this idea a long time ago before I started building it, a long time ago, it feels like a long time ago, like a year ago 'cause with the Mac Studios, Apple, Silicon, it's very power efficient. And I did some paper math that was like the power efficiency of this cost over a period of time, the cost of this computer going on eBay and finding random stuff. And the math showed me a year ago that the power cost of the actual server grid stuff would be astronomical, Lehigh. And I'm like, I must be doing my math wrong. That can't be right. And now here I am, my math was right. - Congratulations. - You're 36,000 a year on power, you're saving so much. - My yearly power bill is gonna be 36,000 dollars. - Good math, I can tell ya. - It's okay, you're getting that back 'cause the server farm is very profitable, I assume. - Oh, well, technically it's saving me, buddy. - Technically, I think I'm deep. I think I'm deep in the red. I'm like, I don't wanna go and do the math. I don't wanna get, it would have been so much better if I just tired of the company got them. I, oh, man. Oh, man. - You fact you're in maintenance cost, I'm sure you're gonna be well into the green. - Well, maintaining it, I'm not hoping to do that. I'm hoping that when it dies, it dies and I will just, 'cause there's no company I can send it back to. I've got it from eBay. So, and it's not like, I don't think I can sell it back again. - So you spend a year working on this, getting it to work, you installed 70 air conditioning units, you're paying 3,000 a month, and your idea is, - When it dies, it dies. (laughing) - I, even now, I'm like buying a whole, a room full of Mac studios might have been the better option. - How much global salt do you need for it to die in a tragic, global salt accident? - Very little if I go by that one guy on Reddit, it's a thought of what I was gonna do, which was shove global salt inside the computers and immerse them in his, so very little. Anyway, sorry, I got a weeb a little bit here. - I understand I would do about a 3,000 dollar power bill. - Yeah, I actually, in the past two days since we last talk, I did a full redo of my whole render farm. If I take a picture of it afterwards, because I didn't take a picture of it during it, I'll send it to the editors. If I remember, a lot of conditions this is never gonna happen. No one's ever gonna see this. - Editors, invent a render farm, make it look awesome. - It still has some wires across the floor just because of where the electricians put in plugs and the fact that it is a bathroom. But it's a lot cleaner now. It works a lot better. It turns out if you don't shove something up against your air conditioners that you've put in, It's able to breathe and circulate air better. So give those room to breathe. Everything's going, I got the leaning tower of Mac Studios. It's great. What's that mean? Well, I picture your render farm. Don't take offense to this. I picture C3PO after he got ripped apart and is like in the backpack. Or all the pieces are there. But there's wires and he's like heads on backwards. That's how it was before. Did I not show you a picture of the before? I'm pretty sure I did. It's a dystopian. It was a dystopian cyberpunk kind of nonsense thing. If I type server into my phones, like pictures, it doesn't come up. And I'm like, why wouldn't that come up? And I'm like, oh, yeah, because it doesn't look anything like what they would think a server looks like. What do you want the search term to find that picture? Most apocalyptic nightmare rats nest of wires. And oh, there it is. We look up apocalypse. Oh, no results. OK, nest of wires. Goddamn, rat king. Let me look up rat king. Nope. Yeah, man, I can't find this mystery. Apple intelligence, my booty. Well, it's working great after the rebuild, right? No issues. Yeah, so you rebuilt it and then nothing has gone wrong. And it worked perfectly the whole time. Kind of. It's it's not my fault. Of course, not. No, I can't find the big. Oh, no. No, come on. Anyway, it's it's good. Except one of the things that I dislike is these small like arbitrary details about computers. So, you know, you have to really read the manuals to know or read in depth about reviews and anything. I'm sure a bunch of people know that if you need to populate all of the RAM slots for server grade, hardware to run at full performance, it needs all the channels filled. It's expecting that. So for a while, these have been running with almost all of them full, but not all of them for the ones that have RAM slots. You can't skip any slots and or it. No, yeah, it's it's probably as common knowledge in the server world, but I didn't know this because I'm used to like running a PC for desktop reasons. And sometimes for PCs, if you have only half of them filled, it runs faster. So it's very opposite thinking, but whatever. Doesn't matter. Not important. Goddamn IT will be the death of me because for some reason, those management ports just decided to die. They they're plugged in just the same they were plugged in before to from one switch, put them into another switch, reset everything, reboot everything, and then just never picks up an IP address. Just cannot do anything. I don't get it. I don't understand how this stuff is supposed to be smart, redundant, fail safe. It just oh no, another plug. I can't do it on a different server. I think it's probably because I didn't let go of the fixed IP on the previous device before I moved it over, but I've done that before where devices just pick up anyway. It's a nightmare. I know that you know that you could pay someone to do that, but there's a part of my brain that would love to see you like like find an IT consultant or something and be like, I just need you to come. I have the setup and they walk into your bathroom and they're just like oh god oh no oh and like they like pull out their laptop and plug in and they're like looking through the software and they're just like oh god. I'm sure it's fine, but I'm sure a professional would that there's even people who build PC like just towers. There's so much shit that if they looked at the way I did it and I think I do it. I mean it works. I've never had a PC that didn't work. Got all those specs I wanted out of it, they would still probably just vomit out their entire lungs at what I did. I can only imagine for server shit what kind of things there are that you would have no idea about. There actually is a guy with Burning Tractor which is the VFX company I work with. They have this guy Rory there. Who is their IT guy? And so he has worked miracles for never actually stepping foot inside the server room. But every time I do something I swear to god he's I see on discord that Rory is typing stop. Rory is typing stop. Rory is saying and I know it's like stop fucking it up. You fucking delete. I know that's what's going on. I know he's doing that, but you know what could need to do. Well I think that's very relatable. We've all been there with our server farms. I think we've all been there with fucking up networking. I mean I missed the days when it was just like you have your modem from the the internet company and I still managed to fuck that up. So what was I think and having a whole farm? Yeah well it doesn't make me feel good that you're doing all this shit and it's this is pretty high tech stuff and it's a complicated system with lots of different. And I have felt exactly the way that you feel. But it's been me on my home Wi-Fi with a modem and a router and like one smart device being like, oh no I connected you to the you were on the Wi-Fi and now you forgot. But it's just me with like one light bulb being like, why won't you be blue fucking light bulb. But it doesn't make me feel good that I've been there but with you know kind of a different scale of technology kind of different. But it's all just like that man. It's weirdly applicable because the things I've learned are actually like in the IT world they're like common knowledge for, oh no there he goes, you had it with our tech talk. Wade speaks for the subreddit. Uh but it's common knowledge apparently in IT land but you know if you want your home network to work better, virtual networks are the way you create a sub network just for your devices and internet of things and then you create specific rules that those connect to each other and then to your other virtual network where you operate your normal internet connection. So it's separated and you don't have any cross traffic from that one and your network. What the fuck are you talking about? Yeah don't don't do it. I tried to do a VLAN. I took down my whole network for two days. Oh man, don't touch the VLAN button. It's not I'm not going to. Oh there's a part of me that's like, oh that would be so sick though. Because then there wouldn't, no it would not be sick. It would be horrible, it would be awful. It would be like taking another full time 40 hour a week job at a thing I don't know how to do and I don't need that in my life. You know. Thank you. What what was the noise Wade? Well shit is on the move out there but thankfully it's just Molly leading up after the cat. Okay, I would have phrased that differently. Shit is on the move out there. Thankfully it's just Molly. But you know so many things you've done so much cool stuff. I bear apparently I don't even know the beginnings of it because my my my render from is very powerful but it still isn't woefully under like you know in the Nvidia CEO what he's talking about. The more you buy the more you save with his $10,000 graphics cards while he's he's hawking a server rack that's as big as the bigger than the shelf behind me that probably cumulatively in the whole rack costs like $10 million and that's just to go really fast. You need a render cooperative or a render collective farm. What what is that a group of farms. Those are types of farms. You don't like a ground plow and kind of farm and the co-op kind of deal. What a name that would be though. The render collective. This is my render cooperative. I have 16 garages each stacked with every bathroom and every building I have access to. Stuffed full of stuff full of servers. I am a man with 10 garages. Oh, a render farm. I have some tech complaints but you know those are always under missed. I love when Mark moans about technology and so does everyone on the separate universally. All right. So I do have a few complaints. Okay, okay. I had another fiber line pulled to my render for it, right? You know how basic do I need to work this time? Line of five. Damn, are you asking me your way because if anyone has an objective because they saw one internet comes in on and goes out on. For a lot of people, we're actually not as many as there should be. Most people have a cable which is still complicated. Well, everyone's internet touches fiber opt to get some point. It just usually there's a junction at some point where it switches from your hard line to your fiber opt. Yeah, it's the internet. It's the fast kind. So you can get your own fiber going between your stuff. You could set up your whole house networking with fibers. You really wanted to. You could do that and it would be very fast, efficient networking. It's just that they're very fragile. And you hated having money in your bank account. Yeah, yeah. And if any rat even touches a wire to crack it in half or they'll nibble on it and show it or it's gone. Oh, an actual rat. I thought it meant like a really shitty person. And you fucking rat. It was touching my wires. It's too fast for you. It's super fast feeling. I'm a rule you would fear or I'm a rule you would love. And you're not going to like that anyway. So anyway, I started shooting my ceiling fiber. I had it too. Because I wanted a redundancy because there's a single connection going to my render farm right now. And if that breaks, then no one can use it or I have to go in there. But there's no computer setup in there. It's all remote action. Right. So if that breaks, even I can't use it because I tell my computers are, you know, I access it remotely. Yep. So I had another fiber line pulled and this one was like, I want hundred gigabit lines and a hundred gigabit because I was like, okay, that was cheap to install. Well, it's actually not that much more the most cost is in laying the conduit and pulling it through. I guess that's fair. The cable may cost like twice as much per foot or something like that, but still most of the work is in the labor of getting it from A to B. So they did it. I had them dig it really nice and the whole pipe and then, you know, it's all said and done and I go to the server clauses and I look at the cable and I'm like, pick up the other cable that they put in this 10 gigabit one and I hold them up next to it and I'm like, fuck it's, they pulled two cables through this fucking enormous thing and they're not hundred. Anyway. So that's the only complaint I have about that and this is more of a complaint with a particular comedy than anything. But it's like, I'll say this thing. I'll say these things and like the people that you'll come to the house because it's like a house and they're like, and I'll say I want a hundred gigabit and they're like, yeah, no one, he needs 100. That's crazy. I want a Julian gigabits. This guy, I want a ball bill. You know, okay, yeah, all right. And so they're like, I'll just pull this fiber, which is perfect. At this point, you have like three government satellites aimed at your house at all. Probably, probably, I have, I have an unbelievable amount of random bullshit going on here and they said the bear. Yeah, yeah, the bear, the American fuckers up. Anyway, I was just like, God, I, I just wish people would take me seriously sometimes because I'm saying what I mean. I am, I don't know everything about everything I'm doing. But when I say I want a hundred gigabit and they'll say like, well, you, you need to get really expensive infrastructure to even handle 100 gigabit. And then I go, I know, go ahead of you, dude. Anyway, I have a render farm update. It's not going to be exciting. I'll take a nap at the media immediately. So in my experience, I have had many ups and downs with the render farm, especially in terms of like how to get the computers in there. What format? And I'm circling back to this crazy thing that's going to make a lot of people mad. Glow ourselves. No, no, no. Okay. Don't even need it. I've, I've very much become like no water cooling at all. Unless it's like an all in one. And even then I would rather have an air cooler because it's just going to be consistent. At least I guess that the fan fails, but usually you can have two on them. I was so proud of my ability to go on eBay and find these create. Wait. No, that's fine. I'm so proud of my ability to go on eBay, you know, much like the Cincinnati bangles. I was just going to write a sign that said, wait, loses a point for every minute he sleeps, while Mark talks, so anyway, so I did not know this, but you know, the Intel chip that everyone seemed to not review very well that I've been like, it's actually very good. They make a workstation grade motherboard for that that takes error correcting RAM. And it's the same fucking price as a normal, well, in a high end, but a high end normal desktop motherboard. And why this is huge is error correcting RAM is very important on a server, especially if we're running a long time, because when it's doing calculations that are very precise and needs to not have wrong numbers in the memory, that leads to bad issues. And there's a lot of reasons why it would happen, why it doesn't have them, but error correcting RAM cracks it. And so it makes it really, really uncommon for those seven. But it's also stupendously cheaper than the even the discounted server hardware I was getting online. It is stupendous and it's more power efficient and it's like, and I look at it. I'm like, what was I fucking thinking with this stupid render farm? So yeah, I'm going to be overhauling it, because now that Iron Long is wrapping up, I'm gonna get it out of my bathroom. I'm gonna get it out of the bathroom. And then I'm gonna move it so that it could be, because it was always a stop cap. It was always like, this is urgent. I got to do this right here in this bathroom. There's no other place to do it, apparently. And then, you know, then one thing like another. But I'm gonna be moving it on the selling stuff and then I'm gonna convert it to a much more optimized stream wise. And this is once again, where I'm like, people like again, you just don't be loyal to AMD or Intel. Buy what is useful in the moment. And holy shit, I can like, they're so discounted because no one wants to get them, because the two frames per second, it gets less than the other ones. Who cares? It is such a good productivity chip. It's so good. I'm doing overhaul to that. So I got some computer parts coming in. Well, that's fun. It's very fun. I've already sold a couple things back. So technically, I'm up, except I paid for them before. So I'm there. But, you know, but you got that money back full price. I'm sure. Oh, yeah. Oh, yeah. Man, people are tearing apart my render farm on, on like the server in a home lab. People are really not people. And I shared the picture because I know it's bad, guys. I know. Oh, wait, that was real. That was real picture. Oh, yeah, you know, yeah. No, that's it. It's just like, look, guys, it was a pre I learned as I went, it's not pretty. But fuck it works. Look, guys, it works. Yeah, that shit fucking flies. And people need to understand, it's not just for like rendering video or encoding video. That's not what it's for. It's not even just for rendering the CG scenes, which can do very well. It's purpose built for distributed simulations, simulations need a lot of the compute power. And to do it efficiently, this is the best way that I've gotten so far to do it. It's not the best way ever. It's the best way I could manage myself. And let me tell you, all those people out there were like, why didn't you hire a professional to do this? I tried. They came in and I said, yeah, I want to put a server in my bathroom. And then they go, fuck you. And they leave. They don't do it. They don't do it. So yeah, that's it. So my my childhood thing, actually, that I do still to this day is something that I I I cultivated again because I was building the render farm. And I know that sounds weird. But my dad always had computers every ever since I was a kid. And so for a very long time, I was the type that was like, he got computer magazines, he got like the micro center mall like advertisement thing. And so I would after he was done with that, go through and look like, oh, yes, I'll take a thousand dollar hard drive. 30, you know, this five year old me just like, a 30 megabyte hard drive. Oh, 10 pack of floppies for only $29999. Well, take two. You know, so now that I'm I'm back in building render farm. And I'm done with the movie now. So I I now I'm just like, what computer can I upgrade? What one needs an upgrade? Like I caught old with that. They were new intel profit saysers coming out and it's like, oh, yeah, all right. I probably need that. I probably I probably could do with an upgrade. I mean, I can build another one. I can build another one. So it's just like I just desperate, desperately looking around for anything that I can build and anything that I can think or because it's just like, it's just tinkering with Legos, but with expensive computer equipment. You are a great candidate for a hobby. I've been trying to teach myself about and I'm too stupid apparently to do very well. Have you looked into home lab stuff? I was on the home lab subreddit. They were making fun of my render. Oh, they were for your. Well, some of them were just like, no, wait, actually, this is what this subreddit is all about. Really shitty setups. This guy, you can't make fun of him. And meanwhile, he's saying that I'm like, you don't look. I'm still getting hit in your man. Seriously, though, the people on the on the home lab subreddit who are like scoffing at people who scrawl things together and have janky it. What is a home lab? If it's not supposed to be some janky shit where you're like, I have these weird random components. Let me see what I can make with them. It's great. And the people who are chill about it are cool. The people who are on reddit about it are not that janky shit held the number one movie for 71 and a half out of 72 hours. What I will say is about that. Some people were looking at all the Mac studios and was like, oh, clearly, that's not a deal. I'll tell you something about Mac, even if you don't like it, that shit holds its resale value like crazy. And you better believe I'm a cell those things now that I'm done with the movie. So I only got them for this for this thing. So I cannot fucking believe how expensive like an M1 Mac mini still is like you think like, oh, that's really old. But it's an Apple Silicon chip. And but it's like, oh, that's like a five year old compact computer, like 300, 400, 500 bucks, depending on the specs and what the fuck you guys will just buy new stuff. Codam. That's the crazy thing is because I bought this. I mean, it was still the AI craze. But then people discovered that, hey, these Macs are actually really good for AI bullshit. And I'm not using it for that. I'm using it for efficient rendering. But the resale value. I might make money on these. Gonna make money on some Mac studios. It's crazy. So the render farm is no more. Whoa. That's a little underwhelming. It's gone. I mean, it is big news, but it's good. - Okay, it's gone. I feel like it's been, that's just a part of life. - Wow! - Fucking hell, though. - Why are you acting like you're not listening, man? I know you hear what we're talking about. - I am listening, I'm reacting. - Oh, okay. So yes, the render farm has left the bathroom. - Woo! - I know, that is a big deal. - And you burned it in a bonfire in your backyard? - I covered it in Glauber salt, which as we know, very flammable, extremely flammable. One of the big problems that everyone is trying to warn me about when I stuffed it inside my servers. It's hard to get up when you're old. It's moving to an actual, not a bathroom. Like, wait, you alright? - Oh yeah, this is about me, it's about you, go on. - Anyway, it's moving to a place that actually should have a render farm, a server room, a real, real deal server room. Purpose built specifically for that. - Oh, fuck. - What is happening? - We got a mole men are tunneling up from his basement floor. - What the fuck is happening? - I just squirted water all over her. - Well, jeez, man, don't do that. - I'm really trying to stick with the story of this one, Mark's render farm is going to a real room like a big boy. - And I'm selling some of it because with the current computer prices and RAM prices, don't give me that face way to carry high. - Oh, man, you're not lying. There's some extra stuff that I think I'll try to sell. Might even make some money off of what I got a fee bait for that. - I don't understand what's happening with Wade. I gotta be awesome, super distracted right now. - Okay, so my, what I do right now is like, okay, let's say making movies or whatever. It's like making entertainment. That's a lot of different disciplines involved in it. That's why I constantly come out and like, oh, I bought a graphics card, boo-hoo. You know, I built a render farm, boo-hoo. You know, glauvers, salts, boo-hoo, you know, lenses, boo-hoo. Does that count of like having a job that requires a bunch of skills? - I mean, technically it's your job, so I can't argue it, but you can never stop. This is easy. I don't want to stop. I want to slide into my grave. - I don't think he's calling to stop. - Yeah, slide into my grave, wishing I could do more. - The only time you're ever going to get Tunis honest is when Mark finds out he has almost exactly one year to live and at the end of the year, Mark dies. - All right. - That's the channel. - That's not bad.

Podcast Summary

Key Points:

  1. Mark is building a render farm using refurbished and discounted server hardware from eBay during a period of massive tech industry hardware offloading.
  2. He discovered that five of his Dell computers simultaneously suffered boot drive corruption, raising suspicions of a coordinated malware or security vulnerability.
  3. Dell customer service was unhelpful, avoiding forensic investigation and instead suggesting random software corruption as a solution.
  4. Mark successfully built an 8-GPU server by modifying cooling blocks and reducing GPU power to prevent overheating and fuse blowouts.
  5. The render farm is powered by a DIY setup, including solar energy and custom cooling, with plans to expand using affordable CPUs for physics simulations.
  6. A phase-change material like Glauber salt is being explored as a cost-effective, room-temperature cooling solution for the farm.
  7. Mark emphasizes the risks of buying discounted electronics from unknown sources, including poor quality, lack of support, and catastrophic failures.
  8. The project highlights a blend of technical innovation, DIY experimentation, and personal passion, with aspirations to create high-fidelity 3D animations and films.

Summary:

Mark is building a home-based render farm to reduce reliance on expensive online rendering services, using cost-effective, refurbished computer hardware sourced from eBay during a tech industry hardware clearance period. He discovered a suspicious pattern where five of his Dell computers simultaneously suffered boot drive corruption, leading to speculation of a targeted malware attack or security flaw, and he criticized Dell's lack of support and forensic investigation. To build the render farm, he modified cooling systems, reduced GPU power consumption to avoid overheating and fuse failures, and successfully installed eight 4090 GPUs into a server.

The setup includes innovative cooling concepts like phase-change materials such as Glauber salt, though these remain experimental. Mark acknowledges the risks of buying discounted hardware, emphasizing poor quality and lack of support, and points to the need for better reliability in DIY systems. He is also collaborating with Houdini and Stagger Dynamics to optimize the farm’s network and processing power, with a focus on CPU performance for physics simulations.

The project demonstrates a deep technical dive into computer hardware, cooling, and rendering, driven by personal passion and a desire to create high-fidelity 3D content, while remaining grounded in practicality and cost-efficiency. Despite setbacks and technical challenges, Mark has achieved functional results and continues to refine his setup.

FAQs

A render farm is a network of computers used to generate high-quality 3D graphics or animations. Mark is building one to reduce costs on expensive online rendering services and to create his own 3D content, like movies or visual effects.

Mark suggests looking for discounted server-grade hardware on eBay, especially during periods when companies are clearing out old equipment to make way for new AI-based systems, often at steep price reductions.

Refurbished computers offer significant savings over new ones. Mark has used them successfully, noting that they are reliable and cost-effective, especially when combined with bulk deals from auctions or online marketplaces.

Render farms require substantial power and advanced cooling. Mark discovered that even one computer can draw as much power as a microwave, and he explored solutions like water cooling, immersion cooling, and phase-change materials to manage heat effectively.

Immersion cooling involves placing computers in a non-conductive dielectric fluid to cool them efficiently. While effective for data centers, it's risky for home use due to potential leaks, toxicity, and difficulty in maintenance or part replacement.

All five of Mark’s Dell computers lost power and had corrupted boot drives simultaneously, which is highly unusual. He suspects a security vulnerability or malware, and suspects Dell’s lack of investigation or transparency may have covered up the issue.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.