Go back

86: The LinkedIn Incident

55m 1s

86: The LinkedIn Incident

The text describes the complex cybercrime supply chain, illustrated by the 2012 LinkedIn breach. Instead of a solo hacker, the ecosystem involves multiple roles: hackers steal data, brokers negotiate sales, and escrow agents facilitate trust. The narrative focuses on a hacker who targeted LinkedIn by first identifying a remote engineer on LinkedIn's own platform. The hacker exploited the engineer's personal website, hosted on his home iMac, to gain access to a virtual machine and then the host computer. There, he found a private key and VPN profile, allowing him to connect from Moscow to LinkedIn's corporate network. He stole a database of 6.5 million user records containing email addresses, usernames, and unsalted password hashes. LinkedIn discovered the breach three months later when the data appeared for sale on an underground forum. The company initiated a "code red" response, involving 40–60 staff in a war room and the FBI. Investigators traced the hacker via VPN logs, a unique user agent ("Sputnek") from his Russian IP, and evidence of him logging into 30 user accounts after cracking passwords. The incident forced LinkedIn to reset passwords, rebuild servers, and publicly announce the breach, highlighting the vulnerability of remote access and the importance of security practices like salting hashes.

Transcription

9752 Words, 54516 Characters

English
If you get fascinated with the cybercrime supply chain, it's never a solo hacker doing the whole thing. There's a lot of layers to this onion. So, let's say a hacker breaks into a place and stills a bunch of information from some company. Well, next he'll typically want to sell that data to make some money and do it again. So, now you've got to find a buyer. But before we even get to the buyer of stolen data, there's sometimes brokers involved. People who have negotiated deals between hackers and buyers. So, you might go to one of these brokers, offer a percentage for selling the database to someone. Now it's on them to find someone. But when the broker finds a buyer, sometimes one side doesn't trust the other. So they bring in a trusted third party, an underground escrow agent, if you will. Who will wait for both the cash and the database and then make the trade. Okay, but then what does the buyer do with this database dump? Well, if it's full of email addresses, they might use it to send spam to people. And of course, the spammer isn't selling anything themselves. They're typically promoting someone else's business, a porn website, or a pharmacy. And it's just fascinating for me to think about that sometimes. It's never about the data breach itself. But what happens to that data after it's stolen? These are true stories from the dark side of the internet. I'm Jack Recyder. This is Dark Net Diaries. This episode is sponsored by Privacy.com. This is a website I personally use every single day. Here's why. When you purchase things online, you need a credit card, right? I bought things from hundreds of sites online, and that's hundreds of ways from my details to get leaked in a data breach. But not since I found Privacy.com. What Privacy does is I connect my bank to it, and then they give me different virtual cards to use to buy whatever I want. And since it's a virtual card, I can use a made-up name. And now I'm so much happier knowing that I didn't give any of my personal details to a company when I'm just buying something online. The other way your data gets out there hasn't got anything to do with security, because major banks and credit card companies just sell your data to advertisers. But Privacy doesn't, so I highly recommend Privacy.com. But use the URL privacy.com/darknet to get your $5 sign up bonus today, or you can try out their free plan with no transaction fees for domestic purchases. Just go to Privacy, which is spelled P-R-I-V-A-C-Y. Privacy.com/darknet to make privacy part of your daily routine. I'm sure you all know what LinkedIn is, right? It's the social network for professionals. You pretty much start your account by posting your resume of where you worked and what you did there. And you can use the site to look for jobs and connect with other professionals in your field. It's pretty popular in the US. In 2012, a person wanted to hack in to LinkedIn and get as much user data as they could. But how are you going to get into the network of LinkedIn? This is a major Silicon Valley company made by some really skilled engineers and administrators. They would certainly be following all the latest best practices for securing a network by doing things like securing the front door to the network by putting a big firewall up to block all non-critical traffic from coming in and inspecting it from malicious activity. Even though conduct security audits on all the internet-facing systems to make sure there's no security holes, and of course they'll be running state-of-the-art monitoring tools and an avirus tools to watch for any intrusions. And they did all that. The front doors of LinkedIn's network was airtight. So the hacker would have to find another way in. He knew that engineers at LinkedIn had access to the corporate network when they were remote. I mean, today it's obvious that a lot of companies have remote employees, but back in 2012 there were LinkedIn employees who had remote access into the network. That is, they didn't have to be physically in the office in order to access the database or other critical systems. So the hacker set out to figure how exactly do some engineers get remote access into the network. And he concluded they must be getting in through a VPN. A VPN is a way to securely connect into a remote network. The traffic is encrypted from the edge of the corporate network all the way to the user's computer wherever they are in the world. And that's just it. If there's a back door entrance for employees only, it would also mean that the hacker could try to get in through that. So the hacker starts looking on LinkedIn's website for people who worked there. Engineers, system administrators, anyone who might have access into that VPN. And so he looked around for a victim. Which by the way, this is the reason why I don't like posting my information on LinkedIn, because you can easily search for all the people who work in a specific company and then figure out who the admins are there, who are probably posting things like, oh, I'm good at Cisco firewalls and Oracle databases. And they might even be posting what versions of Oracle they're good at, which is a clue to any hacker to know what to expect once they get in. But what this means is that it's pretty easy to find a target and narrow your sites on them by just looking at who's on LinkedIn. And this hacker found a LinkedIn engineer who probably had remote VPN access as well as access to the database inside. And the hacker zeroed in on this guy. The hacker saw this engineer's LinkedIn profile and on his profile, there was a URL to this engineer's personal website. Basically, it was like this engineer's name.com. The hacker went to the website to check it out. It was just a basic about me type blog. It said, hello, I'm a site reliability engineer at LinkedIn. And here are my hobbies and things. The hacker poked around here for a bit, but couldn't find anything to exploit. But he looked at where the site was hosted. And it was hosted on a residential IP address. It seemed like this LinkedIn site reliability engineer was running a web server out of his house. This means there are open ports from the internet into his computers. The hacker thought, well, if I can get into this engineer's home computer, this might give me away into LinkedIn. So he looked to see if there are any other websites also hosted at this IP address. And he found one called cacad.com. He browsed around here and this is a much bigger blog type website. cacad.com was a site ran by this engineer's friend. He just hosted it for him. And there's videos of pranks and pictures. It's basically a blog. But this site was built using PHP as the backend technology. And the hacker started looking for ways to exploit this site. He found a way to upload files to the site. And he uploaded a couple malicious PHP files. One was specifically called madness.php. Now if a hacker can upload their own PHP program to your website and get that program to execute, then the hacker can take over that computer that's hosting the website. Because when you go to view the file through a browser, it's going to execute whatever code is in that PHP program. And you can configure it to give you remote access to that computer. And that's just what this file did. The hacker got shell access to the website cacad.com, which was being hosted in the same place this LinkedIn engineers personal blog was hosted. Once he got into the web server, he started scanning for other IPs on the network and found one an iMac computer. And he found that this iMac had an open SSH port, which allows people to connect to that iMac. So he started trying to brute force login to that iMac. So the username, he used a first initial and the last name of the LinkedIn engineer. And he just started hammering away trying thousands and thousands of passwords looking for one networked. And this was all happening in February of 2012. And for days, this web server was attacking the iMac all within this engineer's house without the engineer knowing it. And after a few days of trying thousands of passwords, one worked. He got a hit for a valid username and password. So the hacker logged into that iMac with this and looked around. First he realized this is a person's personal iMac. It's not a LinkedIn computer or even an official work computer. Then he discovered that this web server that he got into was running on this iMac. Yeah, it was actually running on a virtual machine in the iMac and I find this fascinating because in essence, the virtual machine was the only thing exposed to the internet. The hacker got into the virtual machine and then got into the host computer from there. And it's fascinating to me because this shouldn't happen, but the way he did it was through the virtual IP interface. Because I always wonder how it's possible to escape out of a virtual machine and onto the host computer. And here's an example when a hacker did that. After looking around the iMac for a while, the hacker stumbled upon the keys to the kingdom. Literally, he found a private key to LinkedIn. This was the key that the engineer could use to log in to LinkedIn with. See when you log into certain systems, you could use a user name and a password. But another option is to use public and private keys. Where the public key is put on the server you need to access and your private key is on your own computer. So when you connect to the server, you authenticate using the keys. This is all done automatically and saves you from having to type passwords. But when the hacker saw the private key, he snagged it right away. But where does this private key connect to? Well, the hacker had to look around a little bit more to find out. And that's when he found a set of VPN profiles that allowed this person's iMac to connect to LinkedIn. And the profile contained everything they needed to connect to the server name. IP address, the username, the only thing missing was the private key which the hacker just got. The hacker then took this VPN profile and credentials and connected directly into LinkedIn's VPN server. Now here's where the hacker made a major mistake. He made this connection into LinkedIn from his home, which was in Moscow, Russia. And LinkedIn is in California. Well, there was nothing to stop this connection coming in from Moscow though. So he just got right in. And from here, he was able to find his way around the network, looking for the user database, and he found it. And he was able to log into that and grab the username, password hash, and email addresses of as many LinkedIn users as he could. And after that, he logged out and disappeared. LinkedIn had just been breached, but they didn't know it yet. And they wouldn't find out for another three months. The first moment when LinkedIn learned about this was through a forum called InsidePro.com. This was an underground criminal forum where you could buy and sell stolen data from hacks. Someone was offering LinkedIn user data for sale there. The team at LinkedIn saw this and immediately sprang into action. First they needed to verify that the data being sold online was real LinkedIn user data. They compared what was in that sample database with their own database. And sure enough, the hash is matched. The whore and the fear you get when confirming that you've just been breached, it's indescribable. Now LinkedIn's response for something like this is a four step process. First, you confirm, contain, remediate, and do post mortem. They just confirm that they were breached. This is for them to contain the problem. Is the hacker still in the network? What did they steal? How did they get in? Can we block them from getting in again? All these questions needed immediate answers. LinkedIn engineers and security team took over a conference room and called it a war room. Something like 40 to 60 people from LinkedIn were all working on this incident. They were flying in from foreign countries to help. They had the security team involved hunting through events and logs looking for evidence. They had SREs or site reliability engineers in their combing through their systems looking for traces of unauthorized activity. There were lawyers present. Their chief internet security officer was present and active doing all kinds of triage. Other executives were in the room too because this was the most important thing going on at LinkedIn at the time. So the atmosphere was heavy and intense. The first clue they got was from the VPN logs. The LinkedIn security team saw that one of their California based engineers had logged into the VPN from Moscow numerous times. So they called him in for an interview. "Hey, have you been to Russia lately?" "No." "Did you use any kind of Russian proxy lately?" "No." "Did you give anyone in Russia your login details?" "No." The security team was on the trail. This was a major clue. They found out that this engineer had been connecting to the corporate network from his home iMac. And I'm going to guess that probably wasn't allowed. But the security team asked him to bring that iMac in for an examination. So let's back up a month actually. So a month before LinkedIn even knows this happened, the hacker was looking through the database study stall. It contained email addresses, user names, and password hashes. This isn't the password itself. It's a representation of the password after it goes through an algorithm. So in other words, you couldn't see anyone's password in this dump. But the hacker wanted to find a way to crack these passwords. So he posted a few of the hashes to a forum asking for help on how to crack them. When LinkedIn was investigating this, they saw that old post which matched the hashes that were in their database. The situation was getting worse for LinkedIn. They're now saying that the hacker is actively trying to crack users' hashes. Unfortunately for LinkedIn, they weren't yet salting their password hashes. Salting password hashes is an extra step that you do to make cracking hashes even harder. They were in the process of doing this, but when you have hundreds of millions of users, it's not easy to get it done. At LinkedIn, they have different designations for how serious an incident is. Code yellow is something that is some kind of technical risk like a server running over capacity or they're not sure how to scale it properly or a degradation of service that isn't causing a whole outage but could at any moment. Code yellows happen every few months or so there. But as the LinkedIn team investigated this, they determined this incident was a code red, meaning it was business impacting. And because they were already seeing user data leaked to the internet, it meant this threat was certain. And it could be at any moment that the LinkedIn database dump was revealed to the world. I believe at the time it was only available to whoever would buy it and it wasn't freely available for anyone to look at. And this creates a tense and scary moment for any security team. Not knowing what or how much got stolen and not knowing what the thieves plan on doing with it. There's a level of anxiety here, especially because this was already hitting the news media who was announcing this hack to the world. Lots of different teams were pulling logs and saving them for incident responders to comb through. It's best to have logging turned on so you can sort of go back in time and see what happened where. One problem though is that there's now a lot of logs to go through. And if you think about the millions of users who are on the site every day trying to find a needle in a haystack is tricky. I think the day they discovered this or the day after is when LinkedIn called the FBI to inform them of the speech. And the FBI was very responsive. They asked for logs and started interviewing people right away. LinkedIn saw that IPs were connecting in from Moscow. And so they took that IP and started tracing it through the network. Where did it go? What did it access? They were looking through SSH logs, Wiki logs, server access logs, and they saw connections from that IP, which told them what user agent the hackers computer had. The user agent can tell you things like operating system, browser type and version. The hackers user agent was unique. It actually had the word Sputnek in the end, which isn't normal. Sputnek is the name of the first satellite to be put in orbit by the Russians. And this user agent just wasn't seen by anyone before that. Which makes me wonder if the hacker put it there as sort of a signature. With this extra information, engineers can now search logs for that particular user agent to see if that has any hits for it. Because maybe the hacker wasn't using the same IP every time. Maybe they had come in from different IPs and different channels and different VPNs or something. But if there was a matching user agent, then you could know that that's probably the same person. The books they looked at the public website to link to. See if anyone with a matching IP or a matching user agent was logged into any user's account on LinkedIn.com. And sure enough, there was some activity. The same IP and user agent was seen logging into 30 different LinkedIn accounts through the public website. This meant that the hacker had cracked some passwords that was in the data and was logging in to LinkedIn with those users. This certainly elevated the concern for LinkedIn. And back in Moscow, the hacker did in fact have a rather beefy GPU farm. See, cracking password hashes is process intensive. You have to cycle through millions of passwords, hash them, and see if those hashes match the hash in the database. And if so, you found a password. And graphics cards, GPUs, are particularly good at doing a lot of these little calculations like this. They can do a lot of simultaneous things at once. So the hacker was running the database dump through this password cracking station he had. And when he'd get a match on someone that he found interesting, he'd try logging into LinkedIn to verify it. And it did in fact work. He was logging in to LinkedIn.com as different users. Back at LinkedIn, engineers started checking the database servers. They took the information they discovered and searched the logs to see if anyone had logged into the database servers with these IPs, username, or user agent. The database LinkedIn used at the time was Oracle, which is on a Unix machine. So they looked to see if anyone connected to it using SSH. And sure enough, they did. There were logs of the hacker logging into the database server and then accessing the database and running queries in that database. To get this far into the investigation, took LinkedIn six weeks. And I'm talking in active war room. Code red situation for six weeks solid with multiple teams, dozens of people looking through thousands of servers coming through millions of logs, it just takes a lot of time. During this time, they forced LinkedIn employees to change their passwords. They created a whole new account for the engineer who initially got hacked with that iMac computer and they rebuilt servers to make sure there were no traces left on the system. Also, it appears that LinkedIn announced this breach as soon as they could to the public to inform their users that something bad has happened. "Din users beware." The business social network says some of its users' passwords have been stolen and leaked onto the internet. A hacking group rocked online network LinkedIn this week by publishing almost six and a half million user passwords to the site. Six point five million LinkedIn user accounts were claimed to be on some Hecker forums. However, not all six point five million passwords were visible. And the dump seemed to only be in the hands of a few people at the time, with just a sample of it posted publicly. Now the posting and the investigation all happened in June 2000. 2012, but my research shows the hacker got into the network back in March of that year, a whole three months earlier. Which means LinkedIn had no idea of this breach until it showed up on this public forum. So the hacker did this hack back in March 2012, but in May of 2012, before LinkedIn even knew they were up-raged, he hacked into another website too. And I'm not exactly sure what steps he did here. But I've got high confidence of how it went. So by May of 2012, the hacker had already cracked quite a few hashes in the database that he stole from LinkedIn. It was testing some of these logins by logging into LinkedIn with those usernames and it was working. So my theory is that he went through the cracked passwords, looking for anyone that worked for a big IT company as an engineer or admin. Because this hacker was in the business of selling massive databases to make money. So this was his thing. So looking through his cracked passwords, he found a quality assurance engineer who worked at Dropbox. And he tested that the password worked by logging into this Dropbox engineer's LinkedIn account. And yep, he got in no problem. Now this Dropbox engineer has stated on the record that yes, in fact, he was reusing passwords at the time. His Twitter, Facebook, LinkedIn and Google accounts all had the same password. So this hacker actually gained access to a ton of this engineer's accounts. But did this engineer use the same password where he worked to at Dropbox? I don't know for sure, but the hacker was able to log in as the engineer at Dropbox. And I mean, he was able to get into the corporate network with this login, either through a VPN or an admin web portal. I'm not sure. And obviously I shouldn't need to tell you this, but it's not a good idea to reuse passwords for this exact reason. Now this quality assurance engineer didn't have access to files that users stored on their Dropbox accounts, but he did have access to users metadata information about their accounts and stuff, because he would sometimes need to look into issues that users were facing. And the thing is, if a hacker has this guy's login information, then the hacker can access anything this engineer can. So the hacker got in and grabbed all the user data he could that this engineer had access to, which consisted of usernames, email addresses, hashed and salted passwords. Then the hacker transferred this to himself and got out. A month later, the FBI was investigating this LinkedIn breach. One of the things they looked for was to see if that IP address or user agent was logging into any LinkedIn accounts as users of the site. And sure enough, there were logs indicating that the same person had logged into about 30 different LinkedIn user accounts, meaning the hacker had cracked the user names and passwords for these LinkedIn users and was testing it to verify it worked. One of those accounts belonged to a Dropbox employee. And I don't know if it was the same quality assurance engineer, but the FBI saw that connection and thought maybe Dropbox might be the next target. So the FBI called Dropbox to tell them this information and even gave them IPs and user agents to look for. Dropbox looked through their logs and confirmed that, yeah, those IPs did connect in as a quality assurance engineer and got into the corporate network. Once Dropbox did confirm there was unauthorized access into the network, they immediately set up a war room to handle the incident. This is basically like command central, a place where all data can be combined and up to date information is relayed to. Now at the time in 2012, Dropbox had just under 150 employees working there and just like in LinkedIn, this was the biggest thing going on at Dropbox at the time. So it was practically an all hands on deck response. Dropbox had over 20 people working on this incident, but they knew they needed even more help. So they started hiring more security incident responders to just come on in and help. They first discovered that someone had unauthorized access into the corporate network of Dropbox. This seemed contained though, as the connections didn't seem to make it into their production portion where Dropbox.com was ran out of. This was in the corporate side. Well, this is still a big deal to have someone lurking around in your corporate network. We're able to see the crown jewels of like what data users were storing in their Dropbox accounts. Now specifically, they were seeing that a Dropbox engineer was connecting into the Dropbox network from Russia. Then once they connected, they went to the internal Dropbox Wiki, which has information on how to troubleshoot certain things and other technical details about Dropbox's network. The Dropbox team kept examining the logs. They saw which Dropbox engineer had his username and password stolen and went through the logs to see if there was any other suspicious activity around that. This engineer had an account at Dropbox.com, so they looked at his recent activity and it shows that he invited another Dropbox user to join his Dropbox team. The thing is, the Dropbox engineer did not invite that user. So this means the hacker got into the engineer's Dropbox account and then invited himself to see that engineer's files. And after they had their accounts linked up, the Dropbox engineer's account transferred some large files to the hacker's Dropbox. When Dropbox looked into what these files were, it was a list of 20 million Dropbox user details. Email, username, and salted password hashes. This made Dropbox aware that not only were they breached, but the hacker stole at least 20 million user details from their customers. They still weren't sure how the hacker got these or even if it was from Dropbox at all. The next victim here is a company called Formspring. This is a social networking site which is focused on asking questions. Think of it like a place that's dedicated to ask me anything type interviews. In 2012, they had 30 million registered users. And in June of 2012, the hacker got one of the admin user names and passwords from Formspring server and logged in using SSH. My guess is that he got this login from the LinkedIn data too, but I'm not sure on that. He also logged in to one of the web admin panels using the same username. He was able to install a malicious program called madness.php on the server so that he could get back in anytime he wanted. It's the same madness.php that was found on the iMac. And he found their internal wiki and did a search there for hashed passwords. I guess what he was looking for was information about them or whether stored or something. Using the web admin panel, he was able to run SQL commands on the database and grabbed a large amount of user info, specifically emails, user names, and salted password hashes. Then he logged out and that was in June. On July 9th, someone posted a database dump of Formspring users on some underground forum. It contained 420,000 accounts. Someone saw this and contacted a journalist. The email addresses in the dump contained the word Formspring in them a lot like user plus Formspring at gmail.com, that kind of stuff. So the journalist called up Formspring to get some answers. Formspring had no idea they were breached and had no idea how this database got on the forum. But this turned into an all hands on deck situation for them too. They only had a few dozen people working there at the time but everyone who was technical got involved with this investigation. And when one journalist posted about it, many more journalists were calling so the marketing team had to get involved too, trying to handle PR. This the Formspring security team needed to confirm the data. They took the 420,000 users and compared it to their database and it was a match. It certainly was their data. Next they started looking for anomalous activity. And that's when they found someone had SSH into the server from a Russian IP. They took the IP and looked at more logs which indicated that this user logged into the web admin portal. And from there they were able to see what the user agent was for the person who accessed it. They also saw this hacker accessed the wiki and placed madness.php on the web server and ran some SQL queries from the admin control panel. They discovered all this in about one day. I guess their environment was just a lot smaller than LinkedIn to be able to get through it all quicker. Once Formspring confirmed that they had an intrusion, they needed to contain it. They changed the username that was used to log in, deleted madness.php, put more rules in place to change passwords more frequently and set up monitoring rules to look for log ins that weren't from where the admin lives. And then totally destroyed and rebuilt certain servers that they knew the hacker had been on. On top of that, they notified all their users that a breach had occurred. They told users what data had been stolen and had them change their passwords immediately. The day after they discovered this breach, the FBI called them and said, "Hey, you had a break in. Can we see what happened?" Formspring sent the FBI all the logs they could to assist in the investigation. And a few weeks later, Formspring had everything back to normal and things were working just fine again. But this story isn't over. One guy hacked into three major websites and the FBI is now on the trail. You've got to hear what happens next. Stay with us. This episode is sponsored by DeleteMe. DeleteMe makes it easy, quick, and safe to remove your personal data online at a time when surveillance and data breaches are common enough to make everyone vulnerable. DeleteMe knows your privacy is worth protecting. Sign up and provide DeleteMe with exactly what information you want deleted and their experts will take it from there. So since my privacy is super important to me, a few years ago, I was able to get a new Years ago I signed up and delete me immediately got busy scouring the internet from my name and gave me reports on what they found. And then they got busy deleting things. It's great to have someone on my team when it comes to my privacy. Take control of your data and keep your private life private by signing up for DeleteMe. Now at a special discount for my listeners, get 20% off your DeleteMe plan when you go to joindeleteme.com/darknetdirees and use promo code DD20 at checkout. The only way to get 20% off is to go to joindeleteme.com/darknetdirees and enter code DD20 at checkout. That's joindeleteme.com/darknetdirees and use code DD20. So at this point the FBI was aware of all three of these cases. Someone had breached LinkedIn, Dropbox and Formspring all from the same IPs and same user agents with a trail connecting it all. Pretty much the day LinkedIn found out that they'd been breached. They called the FBI and the FBI began interviewing people at LinkedIn and collecting logs from them. They saw that the hacker had connected from multiple IPs in Russia. They also saw the user agent with the word Sputnik in it. LinkedIn was sending them hard drives full of logs to examine and feeding them all the information they were finding. They even supplied an image of the engineer's iMac that got hacked to the FBI. While reviewing all this data, the FBI found something crucial in the logs. They knew what IP and user agent the hacker had. So they looked at all the people who logged into LinkedIn.com, the public website, in the last few years. And they found a person named Jamiro Quattro. This person had registered for a LinkedIn account way before this hack and had the same IP and user agent as the hacker. This could be cold. Like I said, users of LinkedIn often post their full resume there. So if this user had information about himself, posted on his account, it could wrap up this whole story really quick. But Jamiro had a blank LinkedIn account. He wasn't associated to any company and he didn't have a single connection or friend on LinkedIn. But to create a LinkedIn account, you need an email address. So the FBI looked to see what email address registered this account and it was China Big [email protected]. The FBI thought this might be the email address of the hacker. No, the FBI had quite a few IP addresses that they were considering suspicious with this. But they narrowed down their interest to five that may be owned by the hacker and all of them were in Russia. If this had been in the US, the FBI could issue a subpoena to an ISP and get information on who pays the bill for that connection and get answers almost immediately. But things work differently when the FBI wants information from an ISP in Russia. There is a thing called the Mutual Legal Assistance Treaty or M-LAT. M-LAT was set up to allow foreign nations to cooperate in helping criminal investigations by supplying law enforcement with internet service subscriber info. So the FBI requested from Russia, subscriber records, through M-LAT to see whose IPs those were. But this is not a fast process. It takes eight months to five years to get subscriber info through M-LAT. So the FBI had to wait for a while on this. In the meantime, they started cross-referencing the LinkedIn data with Dropbox and Form Spring data. And all three attacks they found similar IOCs or indicators of compromise. The same IPs, the same user agents, the same browser and OS. And once again, they looked for users on those sites from those IPs who registered for an account before this hack took place. Dropbox also had a user registered with China Big 01 at Gmail.com. That person was named Jamis Guru's. A bit different from the Jamiro Quattro from LinkedIn. And Form Spring also had a user registered as ChinaBig01 at Gmail.com too. Because there were so many similar indicators on all three breaches, the FBI was starting to believe that this China Big 01 email address might have been owned by the hacker. So the next step is the FBI contacted Google. The owners of Gmail, an issue to search warrant to get any information on that user. The Google is a US company, so it's fairly easy for the FBI to get information from a US based company. Actually, I think they have to comply with law enforcement in this kind of way. And Google loves collecting logs on its users, so they had plenty to share. First the FBI saw that whoever was connecting to the server had the same IPs and user agents as the intruder who got into the other companies. Next the FBI agent was able to see what search terms this person Googled while logged into their account. And here are some of those search terms, WordPress vulnerabilities, TrueCript hack, Oracle Export Utility, EMS Data Export for Oracle. The user's Google activity also showed them visiting a few sites like Inside or Pro.com, which was the forum that these database dumps were getting posted to. These are also visited articles which talked about the LinkedIn hack. Then the FBI took a look in their inbox and looked at what emails they had and saw a welcome email to Vimeo, a video file sharing website. So he requested information from Vimeo, which also came back with matching user agents and IP addresses. The FBI also saw evidence that this person was logging into some LinkedIn accounts which were employees of Automatic.com. Now Automatic is the parent company to WordPress. And the FBI contacted Automatic and requested to see any login activity from these Russian IPs. And sure enough, there were some login activities. And from Russia was logging in with different Automatic employee usernames and passwords. It's unclear what exactly the hacker stole out of Automatic site if anything, but it is clear he got in multiple times with different Automatic Engineers usernames. The FBI agent then saw a welcome email to afraid.org. This is a website which offers dynamic DNS services. It's a US-based company. So the FBI agent issued a search warrant to get data on who owned the account related to China Big O1. And the name on this account came back as Zopakwe1. And afraid.org also showed that whoever was accessing the site had the user agent with Sputnik in it too. The FBI did a Google search on this Zopakwe1, which is a strange and unique word. And found a user registered on an online gaming site called congregate.com. The FBI requested user details here and confirmed the user agent and IPs matched. But also discovered the user had registered a credit card on that site and had purchased some game credits. The FBI tried to trace the bank details of that card, but it led them to a bank in Russia, which they could not get extra information on. However, they did look to see what email address was registered with this Zopakwe1 congregate account. And the email for this user was root [email protected]. Now since mail.ru is hosted in Russia, the FBI couldn't issue a subpoena for that either. But the FBI Googled the beginning of that email address root talka and found a Gmail user with the same name root [email protected]. So the FBI issued a search warrant with Google to get information on that Google user and Google responded with more information. The first thing they saw was what Google searches that user had searched for. And they were searching for things like LinkedIn hack, MySQL count fields, Change Mac address, Wi-Fi Windows 7. There were also some Google map searches that this user did. They saw the user search for a dentist in Moscow and some other map searches in Russia. Next the FBI agent was able to look at emails for this root talka@gmail account. He saw this person had registered an account at VKontakday, which is like the Russian version of Facebook. The site would email you anytime someone messaged you there. And there were people messaging him, asking about hacking email accounts and different relationship type stuff that was going on. But here everyone is referring to him as Zenya. And everyone only spoke Russian to him. By this point the subscriber records for that IP address came back from the M-LAT request to Russia. And it showed IPs were owned by two people and it gave their physical address. And the FBI of Jenny Neculin had one IP and someone else had the other. You've Jenny lived on the same street that this person was doing Google map searches on. So the FBI started looking up information about you of Jenny and found photos of what he looks like. They compared those photos with the person on VKontakday's account and they looked like the same person. And the VK account was for a person who went by the name Zenya, which is actually a common nickname for people named you of Jenny. I'm not sure how, but the FBI investigation led them to a Russian guy named Kislitsyn. Kislitsyn has been known in the past to broker deals between hackers and people buying database dumps. The FBI found Kislitsyn's email address, which was a hotmail address owned by Microsoft. So they issued a search warrant with Microsoft to see what was in his inbox. There the FBI saw emails going back and forth with the buyer of the form spring data. The buyer ultimately decided to buy the dump for an equivalent of $7,100 US dollars. I'm not sure how much data was in there though. Somewhere between 400,000 and 30 million user records. Supposedly the person who bought this was half Belgian and half Turkish. And what's really strange is they used a middleman for this cash deal and he was also involved with the E-Trade and Scott trade hacks, which I talked about in episode 76, naves out. The FBI indicted Kislitsyn as a co-conspiratory to this, but ultimately was unable to capture him. However, The FBI was able to arrange a meeting with him in the Russian Embassy in Moscow. So they visited with this Kislytsin guy and he gave them a lot of information. Not only information about this case, but information on a few other cases the FBI was investigating too. While meeting with Kislytsin, he told them that you have Jenny Niculin was the person who broke into Dropbox and still had access to it. Then had the form spring data, all with the goal to sell the database dumps on the black market for money. This was enough information for the FBI to issue an indictment for you of Jenny. Multiple trails led right to him, but would they be able to catch him? Stay with us through the break to find out. This episode is sponsored by Vanta. No, it's not your imagination. Risks and regulations are ramping up and customers now expect proof of security just to do business. That's why Vanta is a game changer. Vanta automates your compliance process and brings compliance, risk and customer trust together in one AI-powered platform. Companies like Ramp and Writer spend 82% less time on audits with Vanta. That's not just faster compliance, that's more time for growth. So whether you're preparing for a SOC 2 or running an enterprise GRC program, Vanta keeps you secure and keeps your deals moving. It's where I'd look, I need to surface like this. Get Vanta at Vanta.com/Darknet. That's Vanta spelled a V-A-N-T-A. Vanta.com/Darknet. So who is Evgeny Nickylin? Well he's Russian from Moscow, born in 1987, so that made him 25 years old in 2012. Evgeny loves cars. So much that he started a business in Moscow, buying luxury cars and renting them out to people. And it was from that where he was often seen driving Mazorodis, Lamborghinis and Bentley's around town in Russia. I don't know where all this hacking started for him. Details of his past are foggy. I imagine he got into IT and computers like anyone else, probably started with playing video games and then wanting to hack the video games or cheat or maybe wanting to change his grades in school or wanting to just mess around with his friends by hacking into them like it was a game. Who knows why he got started in hacking, but by 2012 he was pretty familiar with how computers worked and how to exploit them. From reading about him, I also feel like his online life overlapped with some of these pretty notorious Russian cyber criminals. He knew some of the other big hackers. Maybe they taught him. Maybe they were hanging out in the same forums or something. And maybe Evgeny wanted to be part of that hacking world that they were part of. Because after all he liked expensive things and was seeing some of the Russian hackers making gravy from their digital exploits. The FBI issued an indictment for Evgeny Niculin, but the problem was it was in Russia and Russia was not going to arrest you for the FBI and even if Russia did arrest him, he's not going to get extra-dited to the US for trial. So the FBI had to wait. They knew the exact address of where Evgeny lived but had no way to go into Russia to get him. So up until this point, the world had thought the LinkedIn data breach was for 6.5 million users. Because after all that's what was posted on InsiderPro.com. And what's more is that LinkedIn never clarified how many accounts got stolen. But in May 2016, someone posted that they had even more LinkedIn credentials for sale. They claimed to have 117 million user details from LinkedIn and was selling it for just over $2,000 in Bitcoin. This triggered a whole new news cycle. Morning Topping America's Money, a security breach that LinkedIn turns out to be much bigger than first thought. That's right, the social network for business now says a hack or stole 117 million user passwords in the 2012 breach far more than the original estimate of about 6.5 million. I think about all the users of LinkedIn. Yes, of course, professionals looking to network, but also many top executives have accounts there. After all, if your business is listed there, shouldn't the leader of that business be on there too? But on top of that, you have government officials on there, lawmakers are there, members of Congress, FBI agents, NSA agents, senators. And yes, even the president of the United States, Barack Obama made his account in 2007 when he was running for president and was president in 2012 when this happened. This news swept through lots of circles and impacted a lot of people. This more is this new dump contained a lot of cracked passwords that anyone can see in plain text. It wasn't that LinkedIn stored passwords in plain text, but the hackers were able to find ways to crack a lot of the passwords that were in there. Oh, and in fact, we got to see what the most common passwords that got cracked were. I'll read you the top six most common passwords that LinkedIn users were using in 2012. The most common was simply 1-2-3-4-5-6. Over 700,000 users had used this password, because yes, LinkedIn's minimum password length was six characters at the time. The next most popular password was LinkedIn, then the password password, then 1524-5-6-7-89, then 1234-5-678, then 111111. People use bad passwords, and you're telling me some of these users are using the same passwords on multiple sites, and on top of that, they're using the same password at work. Ah, it's outrageous. A few months after that, in October 2016, FBI got the breaks they were waiting for. Yves Jenny, Nikulin, was spotted in Prague in the Czech Republic. With the indictment all processed and things ready to go, the Czech police tracked him to a restaurant where he was eating with his girlfriend. There's body cam footage of the police arresting him. Here, have a listen. There's actually not much to listen to, so I'll describe what's going on. Yves Jenny and his girlfriend are sitting on a restaurant. From the video, I see about three police officers coming in, and they tell him to stay calm. They put his hands where they can see them. Yves Jenny puts his hands on the table, and they ask him to stand up and walk backwards towards the officer. Yves Jenny does exactly that. Then they pat him down, take some things out of his pockets, and handcuff him. All while the other officer is making sure the girlfriend isn't getting up, it's all done very quietly, without any fuss. And Yves Jenny was taken to a Prague jail. For the next two years, Yves Jenny's lawyers fought to keep him from getting extra-dited to the US. I wasn't able to get a second confirmation on this, but his lawyer said the FBI was trying to pin Yves Jenny for hacking Hillary Clinton's emails at the time. It was trying to get him to confess to that. Eventually, two years after his arrest in 2018, the Czech Republic did extra-dite him to the US for a trial. And yeah, I went through the court records, and I never saw one reference to Hillary Clinton in there. The US had nine charges on him. Computer intrusion, aggravated identity theft, conspiracy, and an international transmission of information causing damage to a protected computer. And the victims of this case were listed as LinkedIn, Dropbox, and Formspring. But here's why I love this story so much. Yves Jenny pleaded innocent on all these charges. He claimed he didn't do any of this, and why is that my favorite part? Because it means this case had to go to trial, which means witnesses, evidence, FBI testimony, and so much more becomes public record. To research the story, I got the pleasure of reading hundreds of pages of court transcripts. It was glorious to hear all the details from victims and law enforcement. We rarely hear these things. Like there were three people from LinkedIn who all testified explaining how the hacker got in and what their incident response plan was. There were three people from Dropbox giving testimony, and the CEO from Formspring explained everything he saw. On top of that, there were three FBI agents and a secret service agent who all gave testimony on how they were able to link all these pieces together and track them down. And it's only from all this that we know anything about this story other than what you've seen in the headlines. I mean, I reached out to LinkedIn multiple times and the FBI multiple times to get someone to tell me about this story. Nobody wanted to talk because I get it. What company wants to come on this show and tell me about the worst thing that's ever happened to their company? No one. So it's just really rare for us to see all the details of what happened written out so wonderfully. This trial started in early 2020, but then the pandemic hit, and the trial was delayed like three months. And during that time, the secret service arrested a hacker suspected for breaking into the SEC, Oleksandr Iramenko. When they arrested him, they got access to his laptop and on it was all kinds of evidence on you of Jenny. Pictures of him, videos of him, chat messages with him, emails, you know, tons more evidence. That's weird though, because by 2020, if Jenny had been in jail for four years, what should you think about it? That's a 13% of his life that he's been in prison without anyone deciding on whether he's guilty or not. This took a mental toll on him for sure. He barely spoke any English. He would sometimes shove guards or medical examiners or just try to walk out of the place sometimes. He made a mess in a sale by getting toilet paper wet and throwing it up on the ceiling. And he kept asking the judge for permission to have a Game Boy or a PSP to play. You have Jenny didn't testify himself. He had an interpreter in the courtroom saying everything to him. The whole time he kept saying he had nothing to do with this. But the trial started up and there was quite a lot of evidence connecting him to the incidents. Just to recap the trail here, IPs that access to victims networks were registered to his name specifically. The IP used to steal data from LinkedIn led to a certain browser user agent. And that was associated to a LinkedIn account with a Gmail address. And that email was registered at afraid.org, which had a username that was on congregate.com. And that person bought things with a bank card and that bank card matched other things. That you have Jenny bought on top of that. There was kisslitsin who said that you've Jenny did this and earamenco's computer, which had more evidence on July 10th, 2020, the trial concluded the jury found him guilty on all nine counts. The judge then sentenced him to 88 months in prison, which is just over seven years. And they also ordered him to pay $1.7 million in restitution for the damage he caused to the companies he hacked. Oh, and after that he has to do three years of supervised release, which I'm not quite sure how that works if you're not a citizen of the US. So what do we learn from this story? What sounds like in 2012, these victim companies weren't doing user behavior anomaly detection. Like if a user VPNs in from California and then an hour later that same user VPNs in from Russia, that should trigger an alert, right? Yeah, well, it didn't. The technology at the time didn't really do that kind of correlation. Now there's better tools for monitoring user behavior analytics. And I think tools like that have a lot of potential. Next, it's crazy to me that some people use the same password for their linked in account as their work accounts. Don't reuse passwords like that. Use a complex, unique password for every account you have. And the best way to do that is to use a password manager. They aren't hard to use. So go get one. I have a affiliate link to one in the show notes if you just want a good recommendation. It's also worth noting that these companies seem to have exceptional logging turned on. And when they learned about this breach, they were able to archive those logs and do system snapshots right away to preserve any data that can be used forensically. I've seen a lot of companies just not logged properly. And it just always really bothers me. Oh, in that LinkedIn engineer who was hosting those two websites on his home computer, he's moved those websites to host them on linux now, which is one of our sponsors. I think one of the lessons he learned from this was that opening ports from the internet into your home network can be dangerous. It exposes your computer to a world full of chaos, which can ultimately result in someone getting access to your home network. And I do think about that a lot in the story. If he wasn't hosting those little websites at home, he probably wouldn't have been the way in for this hacker. And it's also interesting to see that bad guys target employees at their house because that network is often not as strong as the corporate network. But here's the crazy part of all this. Remember that LinkedIn dump of 117 million user details that showed up in 2016. Later on that year, it just hit the public for anyone to see. So anyone can go look in the LinkedIn database to see what is in there. And there are still many people who did not change their passwords or changed this is something and then just change it right back to what it was before that. What about all those people who reused passwords on all the other sites like, yeah, I changed my LinkedIn password because I was told to you, but I didn't change all the other six things that use the same password. And that's where we pick up in the next episode. Someone finds a password in the LinkedIn database and has quite a story to tell about that. Hey, do you know about the dark night diary shop? Listen, I love coming up with new shirt designs every month. I throw a few more up in the shop. These shirts look great. One is of Medusa, but she's got ethernet cables coming out of her head instead of snakes. And there's one that looks like a bouquet of flowers, but flowers are actually made of computer cables. Another one is of an archer who's shooting an arrow, but the arrow looks like a USB symbol. You've got to see these shirts for yourself to understand what I'm saying. So visit shop dot dark night diaries dot com and buy some shirts. I'm an independent creator who loves bringing this show to you free of charge every two weeks. But what really helps me keep on that schedule are my patreon supporters. These are people who donate money to the show every month to help keep it going. If you want to show your support for the show, please visit patreon dot com slash dark net diaries and consider donating. Thank you. The show is made by me the chief biscuit dunker, Jack recider, sound designed by the dream alchemist Andrew Maryweather editing help this episode by the Wizard of Light bulb moments Damian and our theme music is by the phonic magician break master cylinder. And even though when you ask me stupid question, you get a stupid and see this is dark net diaries.

Podcast Summary

Key Points:

  1. Cybercrime is a multi-layered supply chain involving hackers, brokers, escrow agents, and buyers (e.g., spammers).
  2. The 2012 LinkedIn breach began with a hacker targeting a remote engineer via his personal blog and iMac, eventually stealing a VPN key to access LinkedIn.
  3. The hacker stole 6.5 million user records (emails, usernames, password hashes) and attempted to crack passwords using a GPU farm.
  4. LinkedIn discovered the breach three months later via an underground forum, triggering a six-week "code red" investigation with the FBI.
  5. The investigation traced the hacker through VPN logs, a unique user agent ("Sputnek"), and activity on 30 compromised accounts.

Summary:

The text describes the complex cybercrime supply chain, illustrated by the 2012 LinkedIn breach. Instead of a solo hacker, the ecosystem involves multiple roles: hackers steal data, brokers negotiate sales, and escrow agents facilitate trust. The narrative focuses on a hacker who targeted LinkedIn by first identifying a remote engineer on LinkedIn's own platform.

The hacker exploited the engineer's personal website, hosted on his home iMac, to gain access to a virtual machine and then the host computer. There, he found a private key and VPN profile, allowing him to connect from Moscow to LinkedIn's corporate network. 5 million user records containing email addresses, usernames, and unsalted password hashes.

LinkedIn discovered the breach three months later when the data appeared for sale on an underground forum. The company initiated a "code red" response, involving 40–60 staff in a war room and the FBI. Investigators traced the hacker via VPN logs, a unique user agent ("Sputnek") from his Russian IP, and evidence of him logging into 30 user accounts after cracking passwords.

The incident forced LinkedIn to reset passwords, rebuild servers, and publicly announce the breach, highlighting the vulnerability of remote access and the importance of security practices like salting hashes.

FAQs

It involves multiple layers like hackers, brokers, and escrow agents who facilitate the sale and use of stolen data, rather than a single hacker doing everything.

The hacker exploited a LinkedIn engineer's personal web server, escaped a virtual machine to access the engineer's iMac, found a VPN private key, and connected from Moscow to LinkedIn's network.

He connected to LinkedIn's VPN from his home in Moscow, Russia, which left IP logs that were later traced by LinkedIn's security team.

They found stolen data for sale on the underground forum InsidePro.com, verified it matched their database, and traced the hacker through VPN logs and a unique user agent containing the word 'Sputnek'.

A password hash is a scrambled representation of a password. Salting adds extra data to each hash, making it much harder for hackers to crack passwords, which LinkedIn hadn't fully implemented at the time.

He used a powerful GPU farm to cycle through millions of passwords, hash them, and match them against the stolen hashes, then logged into accounts to verify.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.