S8E12: #ExploratoryAnalysisOfSentimentTowardABAonTwitter with Albert Malkin and Priscilla Riosa

Hello and welcome back to Behavior Analysis and Practice the podcast. I'm your host Cody Morse, Assistant Professor of Behavior Analysis at Salve Regini University and today I'm speaking with Albert Malkin and Priscilla Burnham Riosa about their paper #ExploratoryAnalysis of Sentiment toward ABA on Twitter which is a fascinating, fascinating topic. I'm really excited to get to share the interview I conducted with Albert and Priscilla with you all. So without further ado, here it is. Hi Albert and Priscilla, welcome to Behavior Analysis and Practice the podcast. Hi, thanks for having us. Yes, thanks so much. I'm really excited to talk about your paper titled #ExploratoryAnalysis of Sentiment toward ABA on Twitter which when I saw it, I think I saw it in free, free issue when it first got online. It was instant like, oh my god, this is such a cool paper really excited to talk to you about it. I've been pretty sure I've messaged you on research gate or something. I'm like, when this comes out, I want to bring you guys on the show which I don't normally do, but it's really cool topic and so I'm excited to get to talk about it today. Before we jump into it, we always like hearing a little bit about the guests, their background, what led them to behavior analysis and what they're currently up to. So could we start with Albert? Would you mind sharing a little bit about yourself? Sure. I'm an assistant professor at Western University that's in London, Ontario and I teach in two online programs. I teach in a master's of a professional education and behavior analysis and are newly launched doctorate in education and behavior analysis. So if you're looking for online doctorate work and putting in a plug now and yeah, I'm just interested in studying social issues for the most part and I've been a behavior analyst for a couple of decades now. So study things that I think people will find interesting. So I'm really excited that you found our paper interesting. Yeah, what got you hooked into behavior analysis in the first place? Oh, it's a common story like everybody else. I was looking for a job as an undergrad and so an ad to work with a child with autism and I didn't know what one of those was until I got the job and then it was a really fun job and just kind of kept kept at it, wanted more education and kept going with the field and then just kind of saw the potential that we could have with the field and just kind of wanted to study more. Nice. Did your undergrad college, did they have behavior analysis now? So you had to kind of wait till masters probably to get into it? Yeah, exactly. It was a bunch of different psychology courses that I took and I was really frustrated that none of them had a coherent world view of any kind. So now you're doing this and now you're doing that. I mean, none of them were coinciding with each other. So it was really happy to find something pragmatic. Yeah. Nice. Cool. Perseaud, do you mind telling us a little bit about your background? What led you to behavior analysis and what you currently up to? Sure. So I'm an associate professor and I'm also the department chair in the Department of Applied Disability Studies at Brock University in St. Catherine's, Ontario, Canada. I did my PhD in human development. I've been a doctoral level board certified behavior analyst for for a while now. And I'm also similar to Albert, a registered behavior analyst in Ontario. And at Brock University, we have a really interesting department in applied disability studies in that we have two specializations. So we have an ABA specialization. So I primarily teach courses on supervision, training, assessment, intervention implementation. And then we also have this other specialization in leadership community diversity and culture or LDCC and a lot of the work that students in that specialization do are focused on advocacy, leadership, community engagement, and really advocating for social change in both the disability sector as well as beyond. So I'm very fortunate to teach across both specializations, which I feel really dovetails really nicely with with the paper that we're talking about today. For sure. And I love the idea of sort of tying in those two components. I always talk to my students about for better or worse, you've got to be a clinician, but you also have to be an advocate. In the world we live in, you know, with the populations that most of us end up working with, they're very vulnerable populations. Most of them are living in contacts where they are not being, you know, seen and heard in the way that they need to be to get, you know, the best treatment possible. So it really puts us in a position of advocating a lot. And sometimes I think I'm more of an advocate than a clinician and I'm sure that's somewhat unique to my role, but it's such an important skill set. So it's cool that you guys are targeting it specifically. Yeah, I agree. You know, looking sort of broadly now, turning our attention to the paper, which I think connects well with that topic. Again, your paper focused on this sentiment toward ABA online, on Twitter specifically. So, you know, I like to sort of start by saying or asking, you know, why should behavior analysts care what is being said about ABA on Twitter? Like, why is that an important topic for us? Yeah, so we, you know, we wrote about it a little bit in the paper and social media for better or worse has a lot of influence in this fear that we live. So, you know, we wrote the paper a few years ago before all the changes that that happened to Twitter. So it's an important that maybe a side in that our data is a little bit dated by a couple of years and, you know, things have changed on Twitter. But at the same time, even now, you do see Twitter posts by politicians, by influential people in the headlines. You know, and we think that's really important. And, you know, if you're able to, anybody is able to write something on a social media platform and have that take off, have lots of, lots of eyeballs on a particular post, that's going to influence other people, you know, really talking about potentially very influential verbal behavior. And that can be by anybody by design on that platform. Yeah, I think that's such an important idea for whoever you're going to add to that. Yeah, I was just going to say that you, as practitioners, as researchers, we can't ignore that critical social context. And so if we keep ourselves in that box and don't start to explore what is being said about our science, about our practice, then we're really missing the mark in many important ways for the evolution of the field. Yeah, I think that is extremely well said. I think the idea that behavior analysts frankly cannot afford to not attend to what's being said online. It would be, I think, very irresponsible to bury our heads in the sand on this one. I think probably now we live in a world where this is probably a pretty widely accepted fact that online conversations impact offline behavior as you put it. But it is just so absolutely critical. And I like one of the things you said in the paper when sort of talking about this idea and the importance of understanding and attending to some of the sort of online conversations about behavior analysis, which is when you talked about Bayer at all 1987's warning that is the field grows social validity becomes necessary for survival. And I think that's just so true. And I think this sort of online platform social media is an interesting way of thinking about social validity on a wide scale. Yeah, that's exactly how we were looking at it. And you know, I don't want to put words in a dawn bears mouth or or or not wolf or any of those folks. But you know, I would hope that they be quite happy to see different kinds of evaluations of what social validity looks like these days. And they didn't look at it in a static way in terms of, you know, it's obviously a subjective opinion and an individual one that people hold. But it's not within a vacuum. And if lots of people have a similar opinion of what we do in practice, we need to be able to take stock of that. Yeah, you know, I think about how when we think about like the idea of social validity, we talk about, you know, measuring at a few different levels, you might have the privilege. personal level where you're getting a sense of the sort of individual client and research participants, feedback, maybe they're immediate, you know, support network family, whatever that might be. And then we also say society at large. And I, you know, I agree society at large is an important component of that evaluation, but how do you typically assess that I think is largely left unspoken on sort of prescribed? Like how do I know what larger society thinks? Well, looking at the discussion on social media is actually, I think, a very interesting way and frankly, practical way. We'll talk about, you know, it wasn't an easy process that you guys follow necessarily to go out of work on your part. But in some ways, much easier than, okay, well, if I want to get sort of large scale opinion, I need to like create focus groups. And like I need to somehow, you know, get like a large like we actually have that day. The day that already exists in many ways, it's about mining and analyzing and understanding it. So I think your paper is really a very cool example of how we can start doing this more and understanding it more. So in some ways, you know, I don't want to embarrass you guys by overconfirmating you, but in many ways, I kind of revolutionary to how we're kind of thinking about this stuff. We'll take the compliment. Yeah, I think that we framed it as a naturalistic observation of the behavior. And, you know, I'll stand behind that. I think that's exactly what we're doing. Yeah, I like that idea of naturalistic, right? Because even if I did go through the trouble of creating focus groups, it's contrived to an extreme extent. If I'm doing a focus group or survey or anything else, your, the participants are asking questions that I have designed. And that way, not to say that does not useful like, of course, that's useful. But looking at it and the light that you've sort of sort of provided, which is like, this is naturalistic. You're not probably you're not on Twitter going, Hey, if you guys put in mind, structuring your tweets like this so we can code it easier. That'd be great. That's not really what we're doing. We're looking at these. And, you know, we're in conversation. It's happening in Twitter. You're respective of, you know, anything that we as behavior analysts are doing in your analyzing. It's really cool. So to kind of tie into the paper and kind of get more specific to it for those who might be less familiar with social media. And we'll get into the details of exactly what you did in a moment. But, before that, just provide a general orientation. Explain how analyzing or looking into online conversation is even possible like for people that, you know, aren't familiar with that process. So we had the benefit of working with our data librarian at Western University, who is a huge help with us and just kind of encourage other researchers to get in touch with their librarians who have all kinds of skills that we don't necessarily get trained in. So, you know, we have a lot of data analysis programs. What is what is more readily available from Twitter, and I'll call it that because that's what it was at the time. Researchers were able to apply for a researcher account. And that would give you access to a key, which was a code that would allow you to log in and then download your research terms of interest. And what you're able to do is pull lots and lots of data. And what we were able to do is pull data on 11 years worth of tweets that contained the hashtags that we were interested in, which were relevant to behavior analysis. So a lot of it is some back end work that I honestly don't have the skills to do, but thankfully our librarian that has a background in computer science was able to pull the data for us. So usually, yeah, usually, and this goes across different platforms. So now there's a fee that you have to pay. And lots of other platforms like Facebook is in this transparent reddit allows you to do that. And I'm not entirely sure about some of the new platforms to be honest, but it's very worth looking into. Nice. I love the example of the collaboration. Did you just approach the librarian so like, hey, I want to like, I'm curious as to what online discourse looks like. And can you figure this out? And they're like, yeah. Really well timed. We were starting to have some of this these conversations about the different kinds of posts that were a lot of people observing online. And I happen to go to a workshop that the librarian our librarian, Christy Thompson was putting on for us. And she was talking about the benefits of the open data sources that social media provide. I immediately send her an email after that and let her know what we were up to. Awesome. Yeah, librarians, university librarians have to be one of the most unrecognized like heroes in this whole sort of academic adventure that we're all doing. They're so vital. The operation of everything. So, you know, with that orientation really helps. So, you know, you're basically, you know, you're interested in this discourse. You have an ability to analyze it. And so you sort of set out to do this research. What was the specific purpose or what exactly were you hoping to accomplish with this particular study? And so, we had a little bit more background. We both, Priscilla and I were part of a working group through the Ontario Association for Behavior Analysis. And they call together practitioners. Autistic people, researchers, just people from the community. There was a particular concern at the time with the social media posts that a lot of people observing. And there were lots of comments being made about how, you know, the perception was the majority of posts about behavior analysis were negative. You know, every time you log on, you'll see something about a being harmful and abusive and what have you. And that was the perception. And I walked away from that meeting with the question of, is that true? And that really set the stage for getting in touch with Priscilla and other other colleagues to answer those questions. And I just wanted to add to that, I was thinking about that, Albert, as you were talking about the process of connecting with Christie. And I think it, everything just coincided in our worlds of really thinking alongside other behavior analysts, people from the community, autistic self advocates around what is going on in our field and what's what's the perception of the field and what can we do about it? What can we do about it to to make things better to listen? And it's sort of led into this research adventure that we have been on and that we continue to be on with learning through online discourse and that that aspect of society where people are really talking about our field in ways that we need to pay attention to. Yeah, that's such helpful context to have. And I, you know, I experienced that myself, I have mostly managed to stay away from too much of the online stuff in my algorithm for, you know, my own personal sanity. My wife on the other hand is very much in that algorithm. She's not a behavior analyst to behaviorally oriented clinical psychologist, but you know, she's obviously invested in supportive of my career. So she's like going down the rabbit holes and stuff. And so I, and of course she reports back. And so I get that like it can feel like this is all skewing very negative online. And so I think that's what I think is the most important factor is that maybe just like the algorithm that's, you know, my wife and many other people reporting sort of getting in on. And so to sort of take a step back. And to go, okay, objectively, like, let's see what this landscape is because that's the complication with social media, right? It's like you can without necessarily knowing it. And of the discourse of the, you know, the collective in not know is this actually representative or not. It's, it's really fascinating. Absolutely. And that's the benefit of being able to pull all the data that we were able to get. So it wasn't that little corner of the algorithm that we were being fed. context that we both faced at the time. So I was hearing a lot of questions from my students as well. And they were on social media and wondering, what am I getting myself into? You know, what is this field that I'm studying and is it okay? And were you getting these questions in class? I don't know if that you had the same experience, Priscilla. Yeah, I would agree students were coming in. I think in my experiences, they were already motivated in it, but wondering about how to respond to or address the online discourse and the discourse surrounding them about ABA and the pushback in many ways that they were receiving across the different networks that they were associated with. So it wasn't for me, it wasn't as much about students saying like, I'm not sure about this field. They were already in and highly motivated and really interested and passionate about making a difference for people through the science, but it was more about how do I respond to what's being shared with me or things that are making me uncomfortable? Yeah, I certainly had that with my students as well. And, you know, as I was reading your paper, I'm thinking about the topic. I was thinking, you know, online discourse, I could see how it could impact families potentially interested in receiving services. Now, maybe they're seeing negative stuff or positive stuff that can impact them. You know, legislators who impact tremendously what we do. I honestly, even though I've experienced that to an extent with my students, I hadn't really even thought about what about potential future BCBAs and maybe you've got an 18, 19 year old who's on, you know, whatever social media and is seeing negative ABA stuff in terms of long term health of our field. Like, we can't underestimate the impact that can have on sort of the youth and going like, I don't want anything to do with this field of what they're seeing as negative. Like, well, then we won't have future practitioners, right? That's dangerous. And I would add to especially if that, if the online world is so much of their world, yeah, which is different from my generation, for sure. Now, it's true. They seem very much more sort of susceptible or even more susceptible, I should say to some of the influences of sort of online discourse. So yeah, hugely, hugely important topic. And I love that the two of you and your team ultimately, you know, put on your data, you know, cap and we're like, how, because yeah, I think there's a lot of ways you could begin the conversation around this, the fact that you actually got to the data in a way that I would have never occurred to me personally to begin looking at it. Like, I think it's really cool, right? This isn't resistant. An opinion about the discourse on Twitter, this you have data sort of breaking this down. So it's really, really cool step. So does that kind of generally capture the purpose of what you're helping to do in the papers or anything to add to that piece of it? Yeah, I think that's it. That's what we were trying to get is this grand picture of what does the discourse look like? To kind of get a little more specific, although, you know, to the points we were making earlier, I think the the the nuts and bolts of the thing is probably beyond my comprehension, but it would take kind of be a little bit more specific in terms of the process you took or the like the terms you were searching for, like, can you kind of break that down a little bit more for for people who are interested in what you did? Yeah, absolutely. So in terms of the process, you know, really reaching out to Priscilla and Laura Mullins, one of our co-authors, the reason that I got in touch with them is one, I really like working with them too. They're both experts in qualitative methods of research and I'm not. I'm kind of learning as I go and it's been a great learning experience and what it required was really getting together and thinking about what are the most common terms that are used in labeling posts about behavior and else's online. So is this preliminary kind of process and then seeing if that checks out, trying out those terms and hashtags and seeing if we're getting posts that to us look like they were relevant. And yes, we picked out hashtags that we noticed were being used quite often. So it was #ABA, #BethaverNalysis, spelled in two ways because we're Canadian. And yeah, applied behavior analysis or the terms that we were looking for. And then there would be other hashtags that coincide with that that we analyzed later as well, but those were the main ones that we looked for. Did the hashtags that you analyzed later those came up because you saw them associated with your initial search? Exactly. So you'd kind of you put in these search terms, right, you get all these tweets back or all this content back that requires coding, right? And to your point about sort of this qualitative process of coding, can you talk about what that process generally looked like? So we generally, we tried to come up with a coding scheme that we were happy with. So now that we have all those tweets that require coding in the computer science term, we needed to code the data to make sense of it from a qualitative research perspective. And then what that required was for Priscilla and I to come up with definitions that would one identify that the tweets were relevant to behavior analysis. So there were lots of examples that were clearly relevant to behavior analysis. So they would and there would be other hashtags and other indicators that would imply that a post is about behavior analysis. So #Altism, #BCBA, special education, autism speaks, the association for like for behavior analysis, those kinds of things. And then unbeknownst to us, hashtag ABA also is used in lots and lots of other ways. So we needed to figure out a way to identify posts that were not relevant. So the American basketball association, American birding association, American beauty association, and it's a city in Nigeria as well. So there were lots of posts that we noticed that were not relevant to behavior analysis as well. So that was the first set of criteria that we needed to identify. And just to add to that too, when I reflect back on the process of developing the coding scheme along with Albert, it was quite iterative in nature. So a lot of back and forths, we had a few, at least I don't remember exactly how many meetings we had Albert, but we had we had a few where we were like, yeah, this is what we think is going to work. And then we sort of looked at some preliminary data and thought, okay, there's something missing here. We need to adjust it a bit. And I think that that is something that is it's certainly more pronounced in qualitative inquiry and those type of those types of approaches. Arguably, it happens in quantitatively oriented approaches and refining, you know, how you're doing training or some instruction that you're delivering or feedback provision or whatever the case is. But it was quite nuanced. And I think we ended up having it at a really good place. I think a couple weeks after we started Albert and then we were like, okay, we feel like we're ready to test it out with the students. You know, that's a great point. You make about the iterative nature of some of the research. I mean, even my own research with lit reviews or, you know, experimental studies. What people don't always understand is like a lot of times that if I'm publishing like a lit review, I might have done that lit review like once or twice or gone through the coding like a couple of times and like you, then you kind of catch an error and you kind of redo everything or the experimental stuff. I might have done that same procedure with previous clients and like, you know, an aspect of it got messed up. And so like a year and I don't get to report that. Those outcomes. And so a lot of times I think people see this research and go like, oh, this is like on their first pass. A lot. We're like, we're learning as we're going, we're making adjustments. You're seeing a finished product. You're probably not seeing the, the, you know, in my case, the 20 failed projects that kind of led to the one project that did work, kind of thing. Absolutely. And to add to that as well, it's, you know, I've had projects [BLANK_AUDIO] where we think we have everything really conceptualized and we're ready to go and then the first participant ends up being your pilot because you messed up something. And that's something that I think is really important for researchers who are newer to the field or students who are getting into the field and that what they read is very linear and clean and that's not necessarily how it goes in research and arguably that messiness is what makes research interesting and fun. - Absolutely. I also spoke like a true qualitative researcher. (laughing) - Yeah, I think one thing that I point out too is that I think that it also feels a lot like just practice and behavior analysis. You know, we're coming up with definitions and instead of observing different kind of behavior, we're just observing verbal behavior and we're trying to see if the verbal behavior is well described by our definition and whether we can categorize it as such. So, you know, whether it's somebody asking for something or whether it's somebody, you know, talking about behavior analysis, it's really just observation of behavior. So I felt quite at home doing that and that iterative process seemed to just make perfect sense. - Yeah, you know, that's kind of the thing I was thinking about is although this is like a completely new way of looking at it, the process of like narrowing into usable, informative data is not a dissimilar process necessarily, it's different because it's a different source of data, but it's very similar in many ways to other things we do. Looking at the sort of the data analysis process where you're starting by looking at relevance, it looks like you guys started with about 120,000 tweets there about. And so as you're going through in your code or you're evaluating it for relevance, is there a way to do some of that through automated or did you have to comb through all of these by hand? Like what did that look like? - So we needed to come up with a way to make the data set more manageable for us. And we picked a random sample from that larger sample of 120,000 tweets because that would have taken too long. But taking a random sample of posts per year seemed reasonable. - And so what were those samples? - Yeah, so we decided to go with a thousand, I believe, per year. And we picked that one because it's a round number. To, we needed to estimate how long it would take to go through the process and just something that seems sufficient. We spent some time looking for some precedents for how many tweets to analyze. And we looked at other relevant research. We found some articles analyzing tweets about a sugar tax in some country, I want to say New Zealand. And there was something like that, correct me for rolling for some time. And you know, there were studies out there at the time that were sentiment analyses that were somewhat similar, but there was no good number to pick. So we just needed to do something that would work, that would meet the aim of what we're looking for. - Sure. - And I think the fact that we took a random sample a crop stratified across each of the years that we were looking at helped to build the credibility of the sample that was selected. 'Cause yeah, I do remember Albert when we were, especially when we were developing the coding framework for relevance and for sentiment, we were kind of thinking about, okay, how long is this taking us to do? And so what is realistic for research assistants to engage and to commit to? And so we were thinking about the practicalities of doing this kind of work and doing good work and doing good quality work at the same time. - Yeah, I think that's very wise. And I think getting random sample, a large sample, I mean, 1000s and I mean, you think about it essentially 1000 data points per year. Like that's quite a bit of information. And you know, make sure that it's something that can be managed is a wise way of going about it. So you sort of narrow into about a thousand per year. What was it, was it 10 years of data, right? - Yeah, I didn't think of being 11. So it's kind of a race in time to finish a study before it becomes irrelevant. So as we were writing it up, we got into the next year and had an extra year available of data. So it just seemed like the thing to do to add an extra year. So we ended up with 11 years of data. So you've got 11,000 tweets. You first go through and evaluate them for relevancy. And again, is that by hand that you and your research assistants are doing that? - Yep. - Whoo. (laughs) - There's a training process as you would expect. So we would, you know, through multiple exemplary training, here's a, you know, here's a relevant tweet. Here's an irrelevant tweet. They went off and scored a thousand, came back the next week. We did see if we had agreement. And then we just kept going until we had a higher agreement above, I believe we wanted above 90 or 80 or 90. - Yeah, I think in the paper says 90. And so then how many tweets were you left with after that relevancy coding process? - About 5,000. - Okay. Yeah, so, oh yeah, 5,400. - Wow, that is a mental data. Once you got to that process, you began coding for sentiment. Or did you code for sentiment while you were coding for relevancy? Was it a simultaneous process or sequential process? - I believe we were doing it simultaneously where the, our research assistants were trained that if something was relevant, then go on and do these sentiment analysis. - Gotcha. That makes sense. That's much more efficient. Sometimes I feel like I want to go sequential when I'm doing things like looking at literature reviews or something, but it is going to be much more efficient to do it simultaneously. So as you're doing that and you're identifying papers that are relevant, you code them for sentiment. Can you talk about the parameters you use for that process? - Yeah, so what we were doing, and so sentiment, I gotta say, was a lot more difficult to nail down as something that we had agreement on. You know, it's obviously very subjective or can be subjective. You know, some, some extreme ends are very, very clear and then some get difficult. So you can have sarcasm, you can have memes and all kinds of different posts in between. So the way that we conceptualized it was a bias toward ABA, a bias against ABA or a neutral kind of sentiment. - And what was some of the, 'cause I wasn't even thinking about like sarcasm, how do you like ultimately train the coders to identify those variables that you're looking for? - Essentially as multiple exemplar training. It was kind of the short answer. And it's a matter of just trying to observe what is happening in the tweet. And yeah, yeah. And I would add that it really was a balancing act of focusing on the content of the tweet while also recognizing some of those other elements without getting pulled in too far into those other elements where it gets a little bit more interpretive. And that was I think I recall Albert the challenge that we had in some of the disagreements during training where the research assistants when they were sort of going off course with coding it was around, oh, well this was the intention. It's like, well no, we can't read into it. We have to see it for what it was for what the person is saying and always refer back to the description that we have outlined it in the paper. We did provide the definitions of sentiment. And I would say that the training process and the materials that were shared with those who were coding We were a lot more extensive. And as Albert said, we use multiple exemplar training as well to help facilitate that process. But yeah, it really was a balance of reading the content for what it is, looking into elements like sarcasm if you can identify those within the post. But what's really observable here without going too far off the edge? Yeah, that's helpful. And I've experienced something similar with coding papers or statements in papers about certain things where you really have to take what is said literally. You don't want to make assumptions about it. If there's context within the text, of course you want to consider that, but you can't pull in. Well, I know this researcher said this thing at a conference. It's not relevant to the thing you're actually coding. You have to hold to that. That is a unit of information in and of itself. Because even if you look at sarcasm, it makes sense. They did that in sarcasm. Yeah, but if you take that tweet and you look at it, is that obvious or is that potentially saying something negative or positive or neutral about ABA? It doesn't like, you have to look at it as a unit of information. One positive that I think supports the process that we took, even though we're sort of talking about it right now, a little looser is our I/OA was great. Our reliability was excellent across the coders. So that gives a little bit more confidence in the findings and the way that the tweets were coded. Absolutely. Now, to you separate based on relevance, you code for sentiment. You also analyzed engagement with consideration of sentiment. Can you talk about what that process looked at? Sure. What we did was total the kinds of engagement that tweets can have. So whether they were liked, so somebody hit the heart button or they retweeted it or so posting that post again or posting it with a comment, so retweeting the comment. And all those were totaled as an engagement score. And then what we did was break those scores up by the number of, so we totaled the number of tweets by different kinds of engagement. And then we looked at the mean engagement, so the average engagement or different kinds of sentiment. And to cut to the kind of exciting finding, which is not good for a view of behavior analysis or maybe just shed the light of the kinds of things that get more engagement on social media, the negative posts got 7.68 engagement scores, that was the average. And then the neutral ones got 1.91 and the positive ones got 2.43. So basically, three times more engagement went to the negative tweets. So if you think about the virality that I was talking about earlier, those kinds of tweets with a negative sentiment or bias against a BA could have more influence because more people are clicking on those and sharing them. And therefore, they get pushed more into people's speeds. Yeah. And I want to dive into that even more, but I want to first, I guess, circle back to cover the rest of the results and then we'll kind of jump into it. So looking at just total results for the negative neutral positive outside of the engagement, what did you find in the study? So there were more positive, thankfully, there were more positive and neutral tweets. So there was a 2300 positive, 2700 neutral and then only 259 negative tweets in our sample. Wow. That's, I mean, almost 10 times more positive or neutral, almost than negative. But to the tie into that other piece, the results that you just shared, but negative gets more play. Yeah. And you said, what was it? What was it? Four times more likely, three times more likely. About three, I think. So yeah, that is sort of a difficult position, right? Where you, we are seeing much, much more positive overall. But if the negative is getting more attention that complicates things, especially, I mean, I'm the next one in this area whatsoever. But my understanding is, you know, to your point about the virality that like the content that gets more attention is more likely to get shared with more people. Yeah, as far as we know, we, as far as we understand. And you know, the algorithms change over time and the kinds of posts that get into people's feeds kind of change over time as well. And you know, whether something has an image or a link, an image might get shared more. Whereas a link might get shared less. Those kinds of rules are there and have been documented somewhere. But, you know, it's not, it's not exactly an open kind of algorithm that we can have access to to see what ends up in people's feeds. Right. I imagine algorithms are probably proprietary in nature. Exactly. Yeah. And, you know, looking at these two results and, and, you know, you also have a table of the hash tags, most popular hash tags, which is interesting and people can check out that table if they're interested. But looking sort of at a zoomed out view of your results, what are some of the big sort of takeaways that you have from your data here? I think the big one, there are more positive and neutral tweets. You know, there are a lot of people promoting themselves and, you know, talking about, hey, I'm doing this thing here, come check it out or where, you know, we provide great services, come work for us. You know, anecdotally, those are, those are the really, really quite common kinds of posts or lots of posts about, you know, like, here's this great finding or here's this great outcome, those kinds of things. And lots of memes about behavior analysis, those kinds of things are there. And then the negative hash tags and kinds of posts do equate, seem to equate behavior analysis with arm with the medical model of disability and some, yeah, and I'm just kind of abusive and a form of convergent therapy. Those are the, those are kind of the takeaways that we noticed. And those are the interestingly with just thinking about the results more globally, as Albert was saying, those were the kinds of posts that were turning heads more. Right. And so it was, yeah, I would say even just thinking about the paper now and the results now that it was fascinating to see that, oh, yeah, you know, first of all, like, oh, fuf, only a small proportion of the posts are negative. Oh, what were we thinking? There's nothing terrible going on here. And then when we got into the whole sentiment analysis, it was like, wait a second, these are the tweets that are picking up traction. These are the tweets that are turning heads. And so we really need to pay attention to this piece. It's not just, oh, yeah, you know, just a few hundred, just a couple hundred tweets within a random sample over an 11 year period, not a big deal, actually a big deal. Yeah. Actually something worth paying attention to. Yeah, you know, when I first saw your paper come out, I was super excited, I downloaded it. And then I immediately, in this I typically do what I'm excited about a paper, I immediately go to the graphs, you know, like, skip everything. I'll figure out the rest later. Let me see these data. Yeah. And I saw the, you know, the positive tweets and neutral treats way outnumbered the negative tweets to your point. I was like, who, whoa, actually first, I don't know if I said who I think I said, whoa, like I was not expecting that. I was surprised. But then to the engagement, I'll come back. Okay, that's kind of the other shoe dropping a little bit right where it's like, okay, we got, and so it's funny because I think there's like, things to be excited about in these results, right? That it's like, okay, cool, we're seeing a lot of positive sentiment. That's exciting, that's encouraging. But we're seeing more engagement on the negative. So it's like, you know, you kind of have the two ways of looking at this, I guess. And to build off of that as well, I think one of the conversations that Albert, myself, Laura and the rest of the team had about engagement in particular was who's engaging, right? So even engagement with respect to the positive and neutral tweets, is it other people who are proponents of the field? Is it other behavior analysts? And who's engaging with the negative tweets? Is it people who are, you know, set in stone with thinking about ABA as inhumane and cruel or to what we were speaking about earlier? Is that being engaged, is that type of content being engaged by people who are considering intervention options for maybe they're a family member, they're young child? So I think who's actually engaging is another really important question to think about. - Yeah, that is fascinating. I mean, and I don't know that like, for example, behavior analysts who are kind of doom scrolling through a bunch of the negative stuff, I don't think they would be liking it. So I guess your engagement wouldn't quite show this, but, you know, I think in terms of traffic, I think about, you know, that criticizing people who follow these feeds, 'cause I think it's important to follow them. But I think it can create a lot of traffic on those streams that are just behavior analysts just watching, you know, and tracking what's happening there. Again, I completely understand why they're doing that, but I do, but it would also push their traffic up. - Absolutely. Yeah, because eyeballs are very important, measure in the algorithm, right? So I would imagine that would be the case, but it is important to see what's being said. So I guess it's a double-edged story. - Yeah, damned if you do damned if you don't, right? You can't win there. Yeah, it's interesting and prosciutto, to your point about who is contacting it. You know, I was also thinking about, you kind of gave the sort of anecdote about the positive, seems like maybe it's driven by a lot of, maybe clinics promoting certain things, whatever. I also wonder, I can imagine clinics are probably not tweeting negative things. So like, are we sit like where the sources of negative tweets, like where is that coming from? I think is interesting. Is it people who have experienced it? Is it people, you know, a lot of the anti-ABA stuff seems to me to largely be perpetuated by people who I don't believe have ever experienced ABA services. And so like, is it just kind of entering one of those, like, pools of people that are just sort of self perpetuating, getting maybe one negative information and then playing telephone with it and spinning it every which way, like, where is that coming from? - So it's an important point. I think one of the things that we don't know in our data and won't be able to know is that you can use whatever username you want on this platform and you can't identify who people are based on their username or, you know, we might be able to get some hints at it but we're not ever really gonna know and what people's experiences are. And your points will take in absolutely that, you know, maybe they are people that haven't experienced, you know, behavior and analytic services but at the same time, they're still talking about it and there's a perception about the field that they're perpetuating that lots of people can see that would, you know, just to circle back to our earlier conversation, you know, potential students, potential service recipients, policy makers, they're all going to see that discussion and it's still important. - Yeah, doesn't matter if it's something they experience, like it's misinformation, right? If someone's spreading misinformation, it's the reality is it's spreading to more people. No one's going, well, actually, you know, for the time of vaccine stuff, well, actually the Wakefield study was, was Holly problematic? Like, no, it's about people saying, oh, you know, there's a study in the '90s that linked vaccine to autism, you know, whatever it is. And yeah, you know, it's a dangerous game in terms of the spreading of some of that information. Where do you see this type of research going next or where do you see some of the next? I know you guys were working on a paper, I don't wanna put you on the spot to talk about something, that's not quite published yet, but I'm just curious, like what are you guys working on? What are some of the questions that are out there? - So what are colleague Laura Mullins, through this iterative process of doing this research and looking at our findings had a really, kind of profound realization that there's gotta be something in these posts that are even positive about behavior analysis that are perceived in a way that perpetuates some of the things that the autistic community has a problem with. And our posts, I'll just say a horror, because I'm a behavior analyst and I think a lot of listeners are, but you know, posts that talk about treatment are and be perceived as problematic because it is a form of using the medical model that you know, some that perpetuates the idea that you need to be changed and treated rather than supported. - Right. - So yeah, we have a follow up study to this one that's in the review process that's a content analysis to see whether the content did perpetuate what the themes were, I should say, in the content of our tweets that were in our sample and we did identify stuff like that, that there was one of the themes that I recall was a hegemonic normalcy. And that was on the negative side of things that the people took issue with. - And also another project that is sort of in the works behind the one that Albert had just talked about is where we used multiple methods so some quantitative analyses as well as more emphasized qualitative analyses to look at autistic people's experiences and perceptions of KBA. So that's something that is on its way. - Awesome. Both of those studies send amazing. And you know, to the first one, to the point about maybe some of the things that would be considered positive to us, the behavior analyst, maybe being seen as negative to other folks, is a really interesting point. And, you know, I don't know the validity of any of this, but there's a podcast revisionist history by Malcolm Gladwell that I really enjoyed. And one of the episodes he talked about the show Cole Bear Report being viewed by people over considered themselves left or right, seeing it and both seeing a different experience. The right people thought that he was making front of people on the left and the left people thought he was making front of people on the right. And, you know, I don't want to get into the politics of it, but to me it blows my mind because I'm watching it with a certain bias. And I'm like, what, I can't see how the other person would have seen the other angle of that. But it's fascinating. And I think it's a point that aligns with what you're saying about we could be seen as behavior analysts. We could see like, oh, there's amazing treatment outcome data. Like that's great. They're helping those kids. It's possible that some of the anti-ABA communities would go like, that's horrible. They're, you know, they're saying they changed behavior. And if fundamentally they disagree with that notion, that's a complication. - 100%. Yeah, that's, it was a really profound insight that Lorette and thankful that she did because it's important to see if we can gather some data to see what those points of disagreement are. - Yeah. - So, yeah, can we identify what people are saying that's being perceived in that light? - Yeah, absolutely. And I've had, I've had little things, and I don't think that viral, but like, you know, certain journals will track Thanks for related to paper. to get tweeted and I've had papers that I've published. I had a paper on a satin ABA and one of the sentences is a non-example of like a behavior analyst wouldn't do x, y, and z. That x, y, and z got picked up and retweeted a bunch by anti-ABA groups and with me cited. It was like behavior analysts do x, y, and z. Literally it is an explicit non-example. It is like over the top non-example to just drive a point home and they grab that and I had that actually happened on the second paper as well. It's what anyone can grab from anyone's content and choose to highlight or interpret it. It's complicated. Yeah, I should point out a source of inspiration for the paper. It was also this blog post by Tom Critchfield and he actually, he just posted it coincided with us already starting this work and then really hammered home that we really need to study this. He posted an article that he wrote on research gait. It was just like a one or two page paper on his experience with heward and colleagues and Tom Critchfield was one of the authors of behavior analysis from A to Z. So you know, lots of great examples of look at all these things that behavior analysis can do. It's super cool. It's meant as an inspiration for students and researchers about all the different areas that have been that are supported by practices and behavior analysis. And for the first bit, he shared his experience online on Twitter exactly. And for the first week or two, there were positive posts like, look how awesome this paper is. It's so great. And then about a week or two later, there are other communities picked up that paper in exactly the same way that you mentioned and those tweets about that paper just out way out numbered the tweets that were positive about it. And you know, there was no mention about anything abusive obviously in in their paper, but it was perceived as such. Yeah. And that I think is kind of a bigger topic. A bigger problem is the unpacking of the negative sentiment, right? Like I think there are different sources with different levels of validity, right? If you've got a family or an individual who have experienced poor quality services, and maybe even abusive services, right? Like that can happen and not only can it happen, objectively, it has happened. I've watched videos that were shown on the news of a kid being abused in an ABA services, right? That obviously doesn't represent the field. It's not what we do. That was an ethical violation and a violation of all sorts, right? That's not what behavior analysis is. That what we stand for. But nevertheless, a horrible experience that a person had, right? And that is a very, very valid and important source of this content that we need to understand and prevent from happening in the future. There's other sources of criticism that might be coming from a place of there should be no services for children with autism whatsoever. And to me anyway, that's a, that's, I'm very, very concerned about that. And I think often they're coming from from places of people who, you know, may or may not sort of fit the ASD diagnostic criteria. And they're, and maybe, but maybe they've, you know, they're communicative and they've had certain life experiences that don't reflect those of all individuals that would meet the ASD criteria. And I think that sometimes frankly, they're not aware of the people who, you know, maybe are non-communicative or have some of those intense struggles. And so that I think is like if, if our disagreement is based on you fundamentally do not believe any services or appropriate whatsoever, regardless of ABA, that's hard. That's, that's a hard conversation to have. I don't know, I don't know how to resolve that problem. I think what we came away with with a, with going down this road for a line of research and just kind of dialogue was something that we said in the discussion too, it's just we just need to have more dialogue and talk to the, you know, people that hold those opinions as much as possible and see if there is common ground. So we held a panel of autistic self advocates to expand on this project and I don't think that at least with that panel, this is not the end of the conversation at all, but there was no, I think there were points of agreement, I guess, you know, the, there's agreement that people will need some support. And I, I, it just makes me think this whole part of our conversation today, makes me think about being open and staying open, open to information. And for people who are practicing in the field, how are you keeping your head to the ground and really listening to the people who you are supporting and serving? And, and if there are certain kinds of communication challenges, what are the innovative and creative ways that you're going about to ensure that they're still cooperative with the procedures that you've designed and, and you know, how are you attending to social validity throughout? So not just at the very beginning and then it's a closed book, but how is this a regular continual process? Yeah, I love that. And it ties into other conversations I've had with folks this season about the importance of needing to really these different ways of referring to this, but like design our services to be client-centered, right? And to make sure the client every aspect of their well-being, yes, their clinical progress, but you know, their satisfaction with services, everything that that be front and center at all times. And frankly, I know that's like an out like duh, like it's obvious in some ways, but I think it can get lost. We can lose the foreshure that treats very easily in the clinical process. And I think it can be really hard to do, especially with contingencies on practitioners saying we got to get more people in, more people to service this and that. And so there's a lot of pressure to move things at a certain pace, where truly being client-centered is about slowing things down. And we need to slow things down to provide appropriate services. Absolutely. Yeah, and I've had I think a very similar conversation already this season of saying like, you know, I get insurance companies or other funders having, you know, quality standards and expecting progress like that makes sense. However, we can't have uniform progress requirements across our clients. They're going to learn at different paces. They need to go at a different speed, you know. And so the pressure of, you know, get these things done in this time, I think is where a lot of the like compliance focused attitude comes from where it's like, look, we need your button a seat. So we can run through the stuff so that we can tell insurers that we're doing this stuff instead of taking a moment to stop back going, okay, what would be best for this client to make clinical progress? Like what is going to be the thing that helps them develop the most skills that's going to help them gain independence and autonomy? And again, I think at the end of the day, I think that every behavior analyst knows that and believes that, but can get a little distracted. In fact, I tell my students this all the time, but almost every time I consult clinically or organizationally, I start by saying like, what is the goal here? It's the goal to help the client gain independence and autonomy because that's kind of the way that I think about the way that I think about clinical services is that. And if the answer is yes, which I hope it is, okay, like how do we do that? So all these things about, well, we have all these sacred cows that we kind of were kind of holding and we have to do exercise like no, no, no, no, we have to help the client gain independence and autonomy. And I recognize we also have to keep the lights on and the process. So I understand we've got to navigate this world of stakeholders, but we got to help the client away that it's meaningful to the client at the end of the day. And so I kind of went on a bit of a sidetrack there, but I think it's an important topic. I'm very much related to, you know, the great work you guys have done here. And again, I'm super excited that you've started this. This is I think really a step forward in beginning to understand and how all this fits together. And again, to tie it back into the stuff we were talking about with a Bayer and colleagues about understanding social validity on a larger scale. We're operating at a larger scale now. Our field has grown tremendously. We need to, I can get better at understanding how we fit into everything now. What people are thinking about us. Yeah, I can agree more. Hey. Well, this has been a really fun conversation. And I imagine, you know, exciting for the listeners learning about this research. For people who are interested in this, again, it's kind of like a very niche area. So maybe not a lot. Are there other resources? Things they should check out, like anything. I know they should keep an eye out for your future paper or anything else that they should be aware of. I would say for listeners who are interested in learning more about qualitative research methods and how to think about integrating them into behavior analytic work, they might want to check out some of Bernie and colleagues' papers. So there was one paper published in 2023 in perspectives on behavior science with looking at how to integrate qualitative methods to enhance social validity in our field and to explore some uncharted territory and topics of interest. And then more recently, they did a paper on qualitative methods and how they can be applied to, to, to, to ABA and provided some really, I think, poignant questions around here are some considerations that you need to think about if you are considering using qualitative methods in your practice and in your, in your research. So they have a really nice set of guiding questions for people who are even just curious about it. And that was published in behavior analysis and practice. In fact, we've, we've covered that paper on the show this season. So for listeners interested in that, you can check out that, that other episode that's out now. - Oh, great. - Yeah. - I think I can add to that a little bit if you're okay with it. - Of course. - Yeah. So there's a, there's a couple of things that we point out in the paper. And one, that we probably need to get better at science communication. You know, so if we're going, you know, we've kind of started a discussion about how we're perceived and, you know, can only improve things by being getting better at communicating what we do. Just a pretty neat book called "Husten." We have a narrative by Randy Olson. And I think that it's just a great read about science communication and just telling stories about science. So, you know, that it's probably a good start. And we can go from there. I think lots of initiatives on those kinds of things would be helpful for both researchers and practitioners to communicate what we do. And then just one other place to point people. This is kind of in line with a research agenda that I have in a way that is, that makes use of open data sources. And there's a call for papers and behavior and social issues. Sorry to bring up a different journal of that. - Oh, yeah. - I think that folks could give that a read. Yeah, there's a call for papers that I'm a guest editor for that on natural experiments using open data. - Awesome. - Yeah, so I think that, you know, if there's a topic out there that interests you, there might be data that already exists. And it's a matter of searching for it and making sense of it. - I love that. And I think why invent the wheel, right? Re-invent the wheel if you've got the data out there, especially in these really socially meaningful contacts like this, your paper being a prime example about. Like you, you can't get any closer to the heart of the matter than like looking at this narrative and this sort of open forum. And so really cool. I wrote down the name of that book. I'm gonna check it out myself. But, you know, the listeners should check out all the recommendations. I really appreciate the two of you coming on this show today to have the conversations really fun and I'm really, really appreciative of this work. Well, thanks so much for having us. It was a pleasure. - Absolutely, thank you. (upbeat music) - Thank you so much for listening to this show today. Please remember to subscribe and like us on whatever podcast player you use. Before you take off, I just wanna thank a few people for helping make this podcast a reality. I would like to thank ABAI for sponsoring the podcast. I would like to thank my production assistant Megan Ellsworth and Jesse Perrin for the work they do to make this happen. And I would like to thank Jim Carr and his band New Latitude for letting us sample their song Cruising Altitude throughout the podcast. Thank you. (upbeat music) (upbeat music) (upbeat music)