Students ‘cheating’ with Generative AI

Welcome to education, technology, society, a podcast about education in the digital age. Welcome to education, technology, society, a podcast about education in the digital age. Hello and welcome to education, technology, society. This is a podcast that looks at education, technology, digital education, but from a slightly more critical perspective than is often the case. Today we're talking about what is undoubtedly one of the most boring current ed tech topics. Panics around students cheating by getting generous of AI to write their assessments. But don't fear, this is a much more sensible discussion than you might be used to hearing. Today I'm joined by Philip Dawson from Deakin University Center for Research in Assessment and Digital Learning and felt as an expert in all things to do with digital assessment, digital feedback and is an ideal guide through the Genitive AI hype. Phil and I had a great conversation, we touched on some practical tips, some theoretical insights and hopefully some new angles on what is a really important but very overhyped area of discussion. Enjoy! So to start the conversation off, I asked Phil to reflect a little bit on the past two years since chat GTP came to prominence and universities collectively lost their minds over generative AI. Looking back, how does he make sense of the past two years? I was reminded by a journalist recently when I was talking about this AI staff that I spoke with her and she published some stuff I believe was in early mid 2022 and I'm saying students are going to be using AI to write their assignments. It's going to be all anyone's going to talk about and we have an opportunity to make some changes now in advance of that. No one did, we claimed we were caught completely by surprise from this but we weren't. We've known this was on the horizon. In 2019 I was playing around with sort of AI writing tools, getting them to write essays and all of that. I think it's just one of those classic ones. The real innovation is in a user interface that's easy for the everyday person to do and it just took that innovation which was chat GTP versus some language model you download and install on your computer. Yeah, it was a moment when people suddenly, as you say, suddenly clicked and there was that kind of uncanny moment where people suddenly kind of got peaked by the idea that this thing was writing things that they could have done themselves. I mean a lot of things were being hyped around Gen AI in 2023 and very, very quickly the concern of students cheating and doing their homework and writing essays became the thing that people were talking about. Did this surprise you? I mean, looking back, we could imagine an alternate scenario where people suddenly became really enthusiastic about the creative things students could have done with Gen AI but no, no, it focused on cheating. I mean, was that a surprise? It wasn't a surprise to me. I've been living the sort of world of cheating for a while and yeah, people love a good sort of moral panic about technology. And people, you go back to sort of Socrates supposedly saying that with the advent of writing, people are going to appear to know things that they don't actually know. And this is a really big problem and technology after technology, we have had similar moral panics about our ability to know what people are capable of. And yeah, I do think it is to a degree an important thing. We do need to know what someone's capable of. We want to graduate competent people, we want to graduate safe people. It matters if they've used AI instead of doing something themselves. It also matters if we graduate them and able to use AI into a world with AI. As an expert, I mean, as you said, there are, there are, there's not completely no smoke without fire. What aspects around Gen AI and cheating do you think you're actually worth paying attention to? I have some substance, some merits in which ruttily kind of overhyped. Taking a step back and saying, why does this matter? And it matters for assessment validity. And going the simplest version of assessment validity, which is just, are we assessing what we mean to assess? And AI has thrown a thing into it that makes it more difficult for us to make that judgment. So when a student goes and gets AI to do the whole assignment, that means we can't judge what that student is capable of. Similarly though, when we set an assessment for students and say, here's a take home essay, don't use AI. That is a huge validity problem because we've got no way of knowing at all what they've used. So I'm of the view, if we're not supervising something, we need to assume complete free use of AI is out there. We need to stop trying to say, you know, here's a take home exam. You've got two hours, we're not watching you, but don't break the rules. You've got to sign a piece of paper to say you've not used AI. It's crazy, regardless of the provenance of those concerns, the universities have kind of acted in a very keen to be seen to be responded to this in a kind of quite punitive way. What do you make of how he has responded in general? So initially there were some bands. And I think they were stupid. Really stupid. And our federal ministers said they were going to ban AI from schools, which is just nuts. It's crazy. And I think, you know, also rhetoric around regulating it. I'm pro regulation. But I also think we need to be realistic about what regulation can achieve in a world where like there's one click models that you can just kind of download 50 different AI models to your local machine and do whatever you want. And it's all open source and can't be regulated. So I think our ability to stop people from using it is not really there. So HE went through that. There was also some let's totally embrace it. And I think we need to embrace it to a degree, but we also need to, you know, be critical, be ethical in how we do this. We need to have space for students to be conscientious objectors to this as well, I think. And that's really hard from, you know, an assessment sort of view. If I set the task and 95% of the students don't use AI, but 5% do, what does that look like while vice versa? I was going to say, I mean, what do we actually know about the realities of students using Gen AI for their assignment tasks or assessments? Is it, you know, everyone is cheating, as you say? It's just a small minority that are actually bothering to play around with these tools. So self-reports are of a small minority, but then the anecdata seems to be of a large amount. And it's one of those ones which really hard to square it. In the cheating research, we know that self-report of cheating is under what the actual rates of cheating are. People do all sorts of really interesting ways to incentivize truthful reporting, the rates of reported cheating go up. I think there's interesting stuff around cheating is a known morally wrong thing, whereas AI use, we're still not sure on the morality of it. We haven't kind of come to our judgement on that. So I think it's probably more than what's in the literature, but it's also different ways that people are using it. So, yeah, we do a lot of interviews with students in our research and some students are saying, "Where's the line here? Inacceptable and unacceptable use? We're actually working on a paper on that right now." Because one thing we found unprompted, students and educators just keep saying, "Where is the line?" But you are seeing universities adopt these strategies of traffic lights, for example. This is red, this is amber, this is green. So, I mean, I'm wondering about the institution responses you've seen. So, look, I have much love for the people out there trying to do these frameworks and traffic lights and all of that. But I think fundamentally, there's really only two states that we can actually feasibly do with people that don't use it, or use it in these tight ways that we are actually going to watch you do, or there's do whatever you want. And anything in between is working on another paper with a colleague, Tom Corbin. He has this idea of we can make structural changes, or we can make discursive changes. A lot of the things in between the red and the green light are just discursive. They don't actually have any real impact. And I'd much rather say, "Hey, you're grown up. We trust you." You can use AI on this task. That's cool. Or, in this moment, because of tie stakes, we need to know what you can do without it. For these reasons that we want you to understand, let you in on that conversation. So, that's why we're making sure you don't use it. But too often we've got to stuff in the middle, but it's just bullshit. Yeah, absolutely. As is always the case. I mean, an obvious question is, what's new here? We've had student cheating for a for decades. What continuities would you point to? And what is genuinely different and new about the Gen AI mode of cheating? So, I think firstly there's that morality thing. Cheating's totally wrong, but the morality of AI use. A lot of students are viewing that as, "I'm just being smart." So, talking to students who are saying, "My gosh, they make us write these weekly reflections for like 1% each week. I'll get nothing out of that educationally." So, I just use AI to do it. And in the next breath, they talk about how I'd never use AI for cheating. That's wrong. So, I think there's different people, got different views. I think cheating was viewed as this thing only those bad people did. And it required a sort of intentionality. If you're going to go to the most egregious forms of cheating, things like commercial contract cheating, where you pay someone else to do the work for you, it's a very intentional act. So, I'm, however, brainstorming with chatGPT on my assessment. Is that okay? I don't know. I haven't been given any guidance on it. And why is it okay for me to brainstorm, but not to produce all of the final written format? And I have a real concern in that space as well. We privileged the final written product so much. So much of the guidance to students is, "Hey, use chatGPT for brainstorming and for coming up with the ideas and for doing the research. Just don't use it to write the words." Yeah, yeah. When did the words become the most important thing? So, this brings me to a really important point. A lot of people have said that this actually tells us a lot about what's wrong with higher education and assessment. Rather than a generative AI, it's not really about the tech, it's about universities, the stress tests for the traditional model of what we do. I mean, you seem to have some sympathy with that argument. Yeah, yeah, I do. I do. We've had some great practices in higher ed for a long time, and we've had some not-so-great practices that take home unsupervised first-year essay as a high-stakes assessment of learning moment. It's been busted for, well, since its inception, people have been getting other people to do their work for them forever. We've had some great practices where we talk with people about their work, much better assessment security, validity, properties there, hard to scale, hard to resource. A lot of this is showing to us, hey, if you actually really want to do assessment properly, you've got to resource it properly, and you also probably have to stop pretending that all these things that we've really all known were total crap, that they actually work because they haven't worked. Which is easy for you to say because you're not in charge of university. Absolutely. If you were in charge of university, what would you want to be doing? Thank you for that. So firstly, I'm going to say, let's look at the degree programs. Let's identify three, four, five, six moments that really matter for our judgment of what someone's capable of. And let's make those more robust. So let's actually put some resourcing into it. I think it's crazy that when I talk about something like interactive oral assessment, people say now we don't have the resources to do that ever. I think it's so sad someone can pay tens of thousands of dollars for their degree and never sit one or one with someone and have a chat about what they know. That's so sad. So I'd be saying, hey, where are those really key moments? And then I'd look at, well, how do we scaffold people towards being able to achieve those moments? And let's stop trying to lock those all down. Let's focus on those being educationally meaningful things. Now, a challenge with all of this is, what if someone just uses AI for all of these initial ones? And then they can't do the high-stack stuff later. And I think that's a really big problem. Working on a piece again with Tom about chat GPT and the generative AI models, not having a concept of the zone of proximal development, which is sort of classic educator thing. I think it's probably the one concept that unites us as educators, more than anything else is at one stage someone told us about. You've got what the students capable of on their own, you've got what they're capable of with the assistance of a more capable other. And then you've got this zone of proximal development, which is the gulf between them and how we get someone through that. These AI models don't intend to help you get through that zone of proximal development if you use them on your assessed work. They'll just do it for you. So I think as a scaffold, they're quite harmful. So we need to develop educationally driven forms of generative AI that can, as you say, can have that more kind of zone of proximal development role. Yeah, I'd like to do that. I mean, I think there's also a sort of thing that I got from your work ages ago. I think I first read your work in like 2011 or something. Is we in education obsessed so much about the state of the art rather than the state of the actual we focus on this kind of teacherly pedagogical practice with technology rather than students actual personal practices with technology. And I think we have to accept that a lot probably most of students AI practices are going to be their own personal stuff they do for themselves. So we might set up a great pedagogical AI, but we focus so much on this pedagogical use of AI and ignore students actual AI practices. They're the things that really matter. Yeah, that's true. So I'm going to finish with I mean, you've had your idealistic view of what a university, but why do you think this is all actually going to end up? Is it going to end up in a massive binfire university is just going to kind of dwindle away or why do you see actually kind of resulting? Yeah, I think we may have a real crisis of assessment and the competence of graduates. I think in three years from now, there's going to be articles in the daily mail or the equivalent things about chat GPT graduates can't do XYZ. It's going to really cause lots of confidence in us as a sector. And I think there's probably going to be a regulator coming at us with a big stick and this will probably happen internationally. It has a history of big stick regulation. The US does not at all have that, but it might lead to regulation of education sectors to say, hey, do your job in graduating people who can do the thing. We might realize that's very expensive or something that's infeasible. And I think that might really change things. That assessment could cease to be such a big important thing. We might say, well, we give up. We're not going to try and assure that. We're going to hand that over to employers who are going to do that in the workplace or professional regulatory bodies. You know, engineering accreditation, accreditors are going to say we decide whether you become one of us through our means. That's a scary prospect. Anyway, it's an interesting space to be working in. It's quite scary space to be working in. But thanks ever so much for talking about it. Thanks ever so much for doing the work that you're doing. I look forward to reading those papers as well. Thanks so much, Neil.