Frances Haugen (2022) | Unmasking Facebook

Welcome to the Festival Dangerous Ideas podcast. My name is Danielle Harvey and I'm the Festival Director. Over the coming months we look forward to sharing with you the audio recordings from the 2022 Festival. FODY started in 2009 and is the original Disruptive Ideas Festival. FODY exists to bring to light important conversations that push the boundaries of conventional thought. Former Facebook product manager, Frances Hagen did not set out to be a whistleblower. But when it became a question of saving lives, she knew it was time to tell the truth. On top of her concerns about mental health and hate speech, finding out that the Facebook platform was being used by human traffickers in Ethiopia tipped the balance. With almost half the people in the world using Metas platforms and leadership of the company vigorously denying her accusations that it values profits over people, Frances Hagen shows us Facebook from the inside, so we can see what is really encouraged and valued. Please enjoy her FODY 2022 talk on masking Facebook. Hello everyone. My name is Frances Hagen and some of you may have noticed last year that there was a large number of news stories over a very short period of time all involving Facebook. Those were the results of a large number of documents I took out of Facebook, so that 1,300 documents, 22,000 pages. Those documents led to multiple senate and congressional testimonies. I have testified before over 10 parliaments and congresses around the world, including the Australian Parliament. And the details of those documents outlined some pretty outrageous things, things like Facebook knew that their products were harming the mental health of children, that it was facilitating human trafficking in places around the world, that Facebook employees were worried about offending the buyers of humans in places in the Middle East, that it was causing ethnic violence in places like Myanmar, African countries, places in India, on and on and on. What I'm going to talk about today is not beat you over the head with the details of everything that was in the disclosures. There's much better higher quality journalism that can give you a lot of detail into what are the impacts of the choices that Facebook has made. The thing I want to talk to you about today is how do we go forward? Are there options for making the world better? We've been told by Facebook that we have only one fundamental choice. We have to decide between freedom of speech and safety. They get up there and they say, "I know, I know these are serious issues. They're really tragic, but you remember after remember, it's a fundamental choice. You have to decide, do you want your freedom or do you want your safety?" The thing we're going to talk about today is why that's just not true, that there are lots and lots of options involving product design around understanding how these systems work, around how we govern these systems, and that's the path forward. So because I like to talk to my critics, more than I like to talk to my fans, we're going to start out and I know there's some of you on the audience that think I'm a dark horse for censorship. So back in the fall, there was a huge media campaign which was later uncovered by the Wall Street Journal to be from Facebook. They said that the thing that I was pushing for was censorship. The reality is AI censorship does not work. It does not scale to those who need it the most. That's people who live in linguistically diverse places and fragile places around the world. It does not scale and there are a better path forward. The reality is that Facebook knows how to keep us safe without censorship. I'm going to go to my prompter because I am not used to the podium. I know how to keep a safe censorship, but it chooses not to because running the system as it is today makes substantially more money. Before I came to Facebook, I worked on three other social networks. I spent a great deal of time working on algorithmic products across some of the largest tech companies. I spent years working on search quality at Google. I developed the first computer vision team at Yelp and I ended up working on the second largest social network, at least in the United States or in most places around the world which is Pinterest. But how did I end up on this stage? Back in 2016, I was recovering from being very, very ill. When I was 29 years old, I got hospitalized for seven weeks because I had silly X disease and I did not know that I was actually starving enough. That I got so malnourished my protein synthesis stopped. I had gone from being able to bike 200 miles or excuse me 200 kilometers in a single day to be in a wheelchair. It took me a number of years to get better. One of the people who was most important in my life to making better was an assistant I hired. He was a friend of my younger brothers and he took me walking every single day. He took me swimming three days a week. He was in a weightlifting so he taught me about macros and eating protein and like how do you put on muscle? And in 2016, he fell down a rabbit hole in some of the darkest corners of the internet. At the time, I was working at Pinterest, working on the home feed, which is like the newsfeed, and on related pens. And I was dealing with the limitations of artificial intelligence. So a key difference between these systems that now direct most of our attention, things called recommender systems, and the field, something called information retrieval. So information retrieval is 30 years old, 40 years old. The field of recommender systems is very young. So the first recommender system, that's a system for suggesting like here's what you should pay attention to. The first commercial one went live in 1999. The major law in the United States that governs the internet has no mention of recommender systems. And that's because when it was written in 1996, there were none in the world except for maybe one or two research ones that just involved trying to focus your attention a little bit better on email. But by 2016, we had a number of products, Twitter had an algorithmic feed, Facebook had an algorithmic feed, and I was working on the algorithmic feed at Pinterest. The challenge with these systems about having a computer director attention instead of another human is that these systems are just hill climbers. So what does that mean? My mother is a life scientist. She's now in a physical priest, but she spent 30 years in academia studying cells. And she used to often say, be careful what you select for because you'll get it. In the case of AI, instead of making intentional design choices like being an editor in our newspaper and saying, we're going to have a lifestyle section and a sports section and a new section and a local new section. And this is approximately how we're going to allocate your time across it. Instead, we write a computer program where we feed in a whole bunch of past interactions and say, can you generate more clicks? Can you get people to stay there longer? Can you get people to repent more? But what we're seeing in 2016 was that when you went and pinned a daisy, we had gone enough pins in Pinterest. We'd gone enough data from enough users. We'd gone enough, you know, gone enough better at figuring out how to interpret all those things, pick the algorithms out, that if you went and pinned a daisy, suddenly your feed got full of daisies. And I went to a meet-up once and someone was like, oh, you work at Pinterest. I just love Pinterest. Pinterest is so magical, but it's just so boring now. And my heart dropped because in that moment, I realized, oh, I know what we've done. We've gone too good at our jobs. And so we ended up introducing something called variety. We had to have humans in the loop. We couldn't just have a blind march to making metrics go up. You know, the system we had was really good at getting the number of repents to go up because the thing you're going to repent next is the thing most similar to the last thing you repent. But that conversation I had with that woman, that required human intuition. The idea that humans get bored. The AI had no idea of the concept of boredom. It had no idea that if you give people the same thing over and over again, people will get bored. But when we added in the concept of variety, the idea that we have to show different things, we ended up getting people to consume 6% more content. And we increased the revenue company by 2.5%. So this is my frame of mind in 2016. You know, the idea that AI doesn't give us what we want, it gives us the most specific match to the way we described the problem. And what we found, what I saw was my friend, so as he's getting spiraling. So Bernie Sanders lost the primaries. He started going online to get comfort. He started talking to people about like what happened. And he kept having more and more conspiratorial thoughts. You know, he would tell me about how George Soros ran the world economy. He would, when I would try to argue with him about it, when I would send emails back and forth, he would say things like do you even read your own citations? Like these are all from the mainstream media. And so I'm in this frame of mind. I'm feeling so powerless. And I'm looking at Facebook. Right? So when most of us look at Facebook, we think of our family, our friends, pictures of cats, people's breakfasts. I'm sitting there looking at Facebook, thinking about what I'm doing every day at work. So this is idea that this is not an intelligence system. This is a system that's just climbing the hill. And I'm living with the context that my friend has become sucked in by really insane information. And I see this. So this is a post. And back in 2016, if you clicked like, or maybe you wrote a comment, maybe you clicked on a link because someone shared a link, a little carousel appeared underneath that post. This might seem pretty innocuous. It's like, oh good, like you like this post. Here's some other posts you might want to also read. The only problem was there was a bunch of very entrepreneurial people in Macedonia who realized that this was the route to millions of dollars. And since it sounds crazy, it's like really, like how could a single little feature like this lead to like actually a large-scale real conspiracy? So what happened was these people in Macedonia found a whole cottage industry. There were thousands of people who started putting out really, really ridiculous news. Things about, you know, Putin, fighting Satan, you know, all these different things. And the reason they did it was they realized that little carousel, the way you were able to actually get in those boxes, because let's imagine there's another 10,000 posts that go off that way. The way you got in position one was by being the most click baity, the most extreme, the most like shocking, because people would reach out and say, I got to look at that. And the result was really dramatic. Billions and billions of impressions were spread around the United States in exchange for not a huge sum of money. When I went and looked into it afterwards, the amount of money the Macedonian and Miss Info operations made was on the order of tens of millions of dollars in the order like 40 million dollars. But it ended up being a huge scandal. Facebook ended up founding this thing called the third party fact-checking program. They came out, they found it a whole team that I eventually joined called civic integrity. And they started saying we have to crack down, we have to make sure that this is safe so elections can't be manipulated because a few people somewhere halfway across the world have realized our systems can generate them a little bit of money. So why are we having this conversation then? If this problem was seen, this problem was solved, Facebook says they invest 100, I think it's only order of hundreds and millions of dollars on things like fact-checking and misinformation. The reality is that system, third party fact-checking, the classifiers used to find that content, it just doesn't work. Back in 2018, so I guess it says 2021 up there, but it's actually 2018, the date changed after I came out, it was very interesting. But Mark Zuckerberg came out and said, hey, we got it, we have a problem. You know I said the clickbait was the thing I got the clicks. Mark said in this white paper, the reality is this system that we're currently using, this thing called engagement based ranking, that means a thing is better if it can get more clicks, more reshairs, more likes. It's dangerous because people are drawn to engage with extreme content. You know, the more sensationalistic it is, the more provocative people click on it more. But I said, don't worry, we have this magical thing where we're going to find the bad things and we're going to get rid of them. People are drawn to this content even though when they click on it afterwards they say don't like it. It's okay, we're going to get rid of them. The only problem with that though was there's a fundamental trade off when we ask anyone, it could be a computer or a human to do this task, which is that language is nuanced, it's complicated, it encodes a lot of context, and it's very easy to be wrong. So let's take a moment. Let's imagine we're going to make a system that's going to take down content, take down people's speech. How often is it okay to be wrong? So all systems, be they humans, be they eyes, have something called a precision versus recall trade off. That means if you want to get all the bad stuff, you have to accept you're going to be wrong a good deal of time. When a human does this, it might be 10% of the time. When a computer does this to get all of the bad things, you may have to take down a lot of good speech, maybe half the good speech, 20% of the good speech. You're going to get comparatively a lot more errors than the bad things. Or you can choose to make very few mistakes, we're going to miss almost all the bad stuff. I always try to be relatively generous to Facebook, like if I can give them the benefit down something I try to do that. And I think Facebook made a good call on this one. They said, "We have a lot of power. We're doing something that really touches people in a personal way." People invested time, they invested effort to write those posts. We don't want to take it down. We want to set that, when we pick out that curve, we're going to pick a plan that curve where we're not going to be wrong more than 10% of the time. And I think that's a reasonable assumption. The only problem was, they didn't tell anyone that that meant they only took down 3 to 5% of the hate speech. They did things like put up graphs like this on their transparency center that sure looked like they take down all the bad stuff. If you saw a chart and said 97% found and flagged by us, you'd be like, "Oh, good job, Facebook. I'm so glad you care." The only problem was, and this is a recurrent problem. This is from their transparency center. There's a lot of euphemisms at Facebook. The team responsible for Genocide is called the social cohesion team because communal violence happens when social cohesion breaks down. Here in the transparency center, I really love it because I view it like a work of art. If we were doing art criticism lecture up here, I'd get there and be like, "What is this painting trying to say to you? What are they doing to make?" What is this technique evoking you? Facebook's transparency center is just like that. This number is 97%. I love the number. It's almost 100%. What it actually means is the amount of stuff taken down by Facebook's robots divided by the stuff taken by robots plus stuff reported by humans. It's not how much of the bad stuff got taken down. The reason why Facebook didn't tell us that is if you write, you know, it's like a 3,000 page white paper, if 3,000 were a 3,000-word white paper about how your system is dangerous and you're admitting it. But don't worry, we've got AI to keep us safe. You need the graph to say this. You need to say we got rid of it all. If you say we got rid of 3% of it, people are going to start asking questions. The second is these solutions just don't scale. When we look at where the places at most risk at violence for, they're places that are linguistically diverse. When I joined Facebook, there had been one genocide so far that the UN had attributed to Facebook. That was in Myanmar. Hundreds of thousands of people died because the Myanmar military sent its own officers to Russia to learn how to run information operations. They came home and they used those techniques against their own people. You literally had Buddhist monks posting things, calling for people to go kill their neighbors. This is not stuff by making up. There's 100, it's like a 300 or 400 page long. You underport the details at length, what went wrong. Since then, there's been a second incident. So in Ethiopia, there was a very, very similar event that took place over the course of 2020 and 2021. We can sit in this room and say, "Facebook, I've been guilt-shipped by my friends for a while about getting off of Facebook." I guess it's time. That actually is in a solution. Most of the world, many of the users on Facebook, at least a billion, maybe a billion and a half of the three billion users, live in places where Facebook is the internet. Those kind of places I mentioned, places where they're the most fragile in the world, the most linguistically diverse. Those are the same places where more people say, "I use Facebook than even say, I use the internet." Often by 10 or 15% margins. They don't even know they're using the internet. That's not by accident. Facebook went into these countries and said, "If you use our products, we'll pay for your data. If you want to use anything else, you're going to have to pay on the open web, you're going to have to pay for yourself." These are places where people used to, back when they were buying SMS minutes, might spend as much as a quarter of their cash income for that connectivity. That's why it drove towards being the internet. From majority of languages in the world today, 80 or 90% of all the content on the internet is only available on Facebook. If we sign off of Facebook, off of Instagram, it doesn't change that fact. Because we have the English web, we have alternatives. But people who live in the most fragile places don't. And the problem of focusing on content instead of focusing on design, focusing on how we make the systems, how we govern these systems, is the AIs have to be rewritten language by language by language, even dialect by dialect. Even things like the difference between slang and Australia versus slang in the United States, I guarantee you the hate speech detector doesn't work as well here as it does in the United States. Because the words that are harmful don't get translated. The last reason that it's so important for us to have this conversation is that when we talk about censorship, it distracts us from talking about what does all these problems. And an understanding how we got here so we can make different choices both with Facebook and with future products. Facebook has done research that if they take your content down, It drives you crazy, right? And if you actually did something that was was was violence. It makes you even more mad when your content gets taken down. They know that if we sit and have to argue over what content should or shouldn't be allowed online, we're never going to agree. But if we were to actually list out all everything that could be done, and this is a big project that I found in a new nonprofit, and next week we're announcing our first funding. And we're doing a big project that's around that question of like, if we were to list out what can be done across a whole bunch of problems? There's probably an area that we'd actually be willing to agree and say, yeah, that's a reasonable set of rules of the road. So why can they do that? So one of the things that people often take for granted is that Facebook is very different than other major industries that are similarly powerful. When we look at things like seatbelts in our cars, we have seatbelts because we were able to buy cars. We were able to take those cars apart. We were able to crash those cars and understand that some cars were safer than other cars. We were able to put sensors in those cars and understand do they actually do what they do? We can have accountability for cars because we can control those things. We have actual access to the important decisions that were being made. But that isn't actually obvious. We got to that level of transparency. We got to that level of understanding because we've worked out for 100 years. When it comes to Facebook, the reason we can have Facebook say to us the only thing to talk about is content moderation. It's because all of the interesting choices on these products are behind our screens. We can't go buy a Facebook data center. We can't go and run the product different ways. We can't even see what representative data is on how these things work. We're starting like it's back in 1920 and there's like a bunch of Model T's roaming around. We assume Facebook is still about our family and friends. When in reality, Facebook gives the most distribution to the most extreme ideas. And it's not by accident. They didn't set out to do it, but they made a whole bunch of individual choices for business reasons that let us survey our today. So if we were to roll back in time to 2008 and part of what I really like about this audience instead of say when I go talk at college campuses is I can say do you remember Facebook of 2008? The first time I went with a college student and I was like getting all riled up and I was like talking about how we can change these things and how like do you remember like how good Facebook of 2008 was? She looked at me and she's like, I don't remember. So back in 2008, Facebook was about our family and friends. It actually was about our family and friends. But our family and friends let Facebook down because every quarter, Facebook needs your family and friends to make a little bit more content. And your family and friends did not comply. So instead of making a little bit more content, a little bit more content, a little more content so that their stock price could keep going up. Because remember, these products are not free. We are paying for them with our attention. We're paying for them with our ad clicks. So instead of making more content so we can consume more and more content, Facebook had to figure out other strategies. So let's say back then, or maybe let's say even six years ago, you know, part of our attention was our family, part of our attention was our friends, our citizen was on our pages, parts on groups. Not enough people were joining these new kinds of things that could put more content in our feeds. You didn't have people joining enough groups. You didn't have people joining enough pages. So Facebook started getting pushy. They started trying harder and harder, pushing people towards suggested groups. And it's had some very unfortunate consequences. Remember, we talked about that carousel. Now the carousel prioritized the things most likely get clicks. That same system, most extreme content gets the most distribution. Same thing happened for groups. In 2016, 65% of people who joined neo-Nazi groups in Germany joined because Facebook suggested those groups to them. Then they had this other problem, which was it didn't matter how much they were pushing people to join groups. People weren't joining enough of them. They needed people to keep feeding the attention engine. So they started saying, hey, you know, what does someone invited you to a group? Maybe we should just put you in that group for 30 days so you can get taste of it. Like maybe you would have liked that group had you'd seen it. And so they started doing this thing where they stopped really treating invites like invites. And if you engaged with any of that content, you put a comment on something you'd like. Even if it was to say like, this is misinformation, now you're part of that group. And over time, this is what it looked like. It's actually more skewed than even that. Back in 2019, when I ran this analysis, 60% of all the content people saw on Facebook was now from their groups. I would guess today if we were to look at what Facebook actually is, it's probably more like 80%, 85% is from groups and paiders. And this is a problem, which is Facebook prioritizes this distribution of content so much that it adds features that turn people into, basically facilitate information operations. So instead of having to go and reshare into group after group, they'll put up a feature that lets you spam into 10 groups at the same time. Or they'll go in there and say, "Hey, we need to incentivize people to produce more content." We know that if you get more likes and comments and reshures, you produce more content. And so they'll do things like changing ranking. Only to find that when they do that, it gives a bias towards all the content that anger's people that provokes those extreme emotions. And the consequence of things like this is that when they made that change back in 2018, back when Mark was having to write papers about how dangerous these things were, as they went into Europe and talked to political parties, the political parties on the right and on the left would say, "We know you changed the algorithm, because even six months ago, if we had shared a white paper on our agricultural policy, we get it. It's not the most interesting thing in the world. Like it didn't get a fight in the comments. People weren't resharing it everywhere, but we could see from the statistics that people were reading it. They were spending time with it. When they switched over to saying, "We need you to be an engine of engagement. We need you to be scattering likes and comments on your friend's content, so they'll make more of it." Content like that, the bread and butter of democracy no longer had a place in our news feeds. And this isn't a political issue. The right and the left both experience this. The fact that Facebook got to vote on what we got to learn from our political parties before we did, like they got to influence our political parties positions before we put our votes in the ballot box, that's unacceptable. And when we combine all these things together, it gets really, really dangerous. So those groups, when they push you towards a mega group with a million members, that group might make a thousand pieces of content a day, five thousand pieces of content a day. But you only get to see maybe five or maybe at most ten pieces of content from that group, depending on how much you use Facebook. And the way they pick that content is with that same algorithm, the one that went let us look at the bread and butter of democracy. And so we end up in this engine where we give more and more distribution to the most extreme ideas. And lastly, I want to touch on real quick, is the third party fact-checking and these systems of censorship aren't free. One of the big scandals that came out of my disclosures was that Facebook was running a two-tier enforcement system. Where if you were a politician, if you were a celebrity, you could say anything you wanted. Because they were afraid that when they made one of those inevitable mistakes, you would make them look bad. So the idea that when we focus on content instead of focusing on design, we actually have to do these kinds of risk mitigation systems, or otherwise the air is piled up. Or we're seeing more and more scandals involving the people who look at the appeals. So the AI takes something down, you complain. The people who look at those appeals, they get PTSD. There was a multi-tens of millions of dollars lawsuit in the United States because the people who look at our content, the people who are keeping us safe when the AI has hiccups, those people were committing suicide. They were going on disability usage by PTSD. And then there's just the impacts on actual freedom of expression. There's a bunch of these different initiatives that are all around the idea of who gets to decide why ideas we get to consume. You know, in these cases, it's more humorous. You know, it's people who are photoshopping nipples onto things to prove that we should have freedom of the brust. But at the same time, do we really want a private corporation to decide what speech is good or bad? So what's the alternative? The thing I keep emphasizing, and it's only going to get more important as we move towards end-to-end encrypted systems, is we need to always be thinking about how do we keep humans in the loop. One of the most impactful things that you can do, you're just simple chief, is requiring people to click on things before they reassure them. The public radio in the United States, for April Fool's Day, a couple years ago, I did an article that said the headline was, "People don't read anymore." And if you have been to click into the article, it was God very popular, it was all over Facebook. If you clicked on the article, the very first sentence said, "If you receive this in your newsfeed, please do not reshare it. It's about how people don't read articles. They just read the headlines." And by the fact you can see it everywhere, you can people don't read anymore. If we require people to click on a link before you reshare it, it's like 10% or 15% less misinformation. No one was censored. What we said there was a valued human judgment, a valued people actually being that filtered directly in our attention. There's also little things around design of these systems. How should virality work in these systems? So let's imagine Alice writes something. Her friend Bob reshires it. Her friend Carol reshires it again. And it lands in Dan's newsfeed. Dan doesn't know Alice. Alice doesn't know Dan. He doesn't know if she's very credible or not. Imagine if we said, hey, we're going for a little wall of. Dan, you can still say anything you want. But you have to copy and paste it. You have to make a choice. You can't just be doing a reaction. This little change has the same impact on misinformation as the entire third-party fact-checking system. And it works in every one of those vulnerable places that I talked about. It doesn't require you to translate it over and over again. It just requires to be thoughtful about how you design these systems. The fundamental question I want to leave you guys with today is how do we want our attention to be directed? When the internet came into being in the '90s, when we passed the last major legislation in the United States around the rules of the internet, in '96, we did not have any recommender systems. When you received a comment, when you received a blog post, when you received a newspaper article online, you did because a human wanted you to read it. Humans directed our attention. These are systems about humans communicating with humans. The fundamental question is, do we want humans to focus our attention, or do we want computers to focus our attention? Because we can design social media to be about our family, our friends, to be about our actual communities. And if you design it, there's always humans in the loop being that buffer. So the most extreme ideas don't get the most distribution. Thank you so much. [APPLAUSE] This session was recorded live at Carriage Work Sydney. The Festival of Dangerous Ideas is presented by the Ethics Center. The Ethics Center delivers programs and experiences designed to bring ethics to the center of our life. We offer counseling for individuals, leadership and development services, events, courses, and educational resources. As a nonprofit charity, we rely heavily on donations and memberships. Please visit the Ethics Center website to find out more. Thank you. (upbeat music)