Learning analytics – what do students think?

Welcome to Education, Technology, Society, a podcast about education and the digital age. Hello, and welcome back. My name is Neil Sowin, and today we're talking about learning analytics, something that's now an established part of educational technology and something that's creeping into many universities learning management platforms. I caught up with Hannah's house from the University of Innsbruck to talk about his research on students' mixed reactions to learning analytics. This is a technology that can sometimes encourage superficial forms of learning, exacerbate existing inequalities, but can also motivate some students to boost their performance. So learning analytics is a really interesting part of Ed Tech that brings together some fundamental tensions around how educators and technologists have different ideas about how to approach university learning in university students. It's always good to see how these sorts of technology are playing out in real life context. So Hannah's study is really welcome. I hope you enjoy the conversation. Yet its core learning analytics is about using data to better understand how people learn and to help them to increase learning success, to optimize learning. So this means in practice collecting and analyzing data about what students do in digital learning environments through algorithmically driven technologies, I would say. So for example, how often students lock in, how long they spend on tasks, or how they perform in quizzes, for example. And the aim is to turn this into insights about learning and learning analytics can be descriptive. So understanding the past of student behavior, it can be predictive. It's also about forecasting future outcomes based on on this data and it could be prescriptive. It's about also recommending actions based on this prescription. And in universities, for example, in Austria, this typically appears as dashboard, as applications integrated into learning management platforms like model or canvas, who can track and analyze behavior already. The dashboards, for example, show progress, they compare performance to peers, they provide real time feedback. And this could be useful for teachers for instructors because they can highlight which students might need to board or which parts of the course are causing difficulties. And the key promise is that learning becomes more visible and more actionable. So instead of only receiving feedback at the end, students can adjust their learning while it's happening. And also teachers can adjust their content and their teaching style. And more broadly, many researchers see this as part of a larger transformation in education, I would say, where data plays a much bigger role in supporting learning and teaching. And some even talk about an emerging era of learning analytics, at least in higher education. Yeah, I mean, there's certainly a lot of it about what do you think the underlying logics are to these technologies? What do these learning analytics developers and learning analytics community see itself as doing? And what logic and ideologies and theories kind of underpin what they're doing? Yeah, we also did a literature review and in our opinion, a key feature is a technosentric logic. So the field often starts from what can be measured, what can be built rather than from better logical questions about, for example, what good learning actually is. So that's why learning analytics to this community has been heavily focused on developing tools and systems and less on asking deeper, better logical questions. But at the same time, the community sees itself as doing something good, something positive because they want to support teachers, they want to support students. I think the main goals are improving success, learning success, improving motivation, reflection, to give better feedback, to enhance self-regulated learning faces. So, for example, if a student can see their own learning patterns, the idea is that they can adjust their behavior and take more control over their own learning process. I think this idea of self-regulated learning has come up a lot. And you see learning analytics people talk about blooms, too sigma and the whole idea of one to one learning. Now, I mean, you talk about these things in your paper, but one of the things you really look at is learning analytics through this lens of algorithmic governance, which is something that a lot of listeners will be familiar with. But you draw on something actually I hadn't come across Antonette, Rive and Thomas Burns. I mean, what what the burns and the Rive talk about when they talk about algorithmic governance. Yes, their idea builds on the work of Michelle for call on governmentality. So, for call was interested in how people's behavior is guided and shaped by a specific form of power, which aims to direct actions of individuals and groups in more subtle and indirect ways. And certain behavior becomes more likely than others and the raw war and burns extent this idea to the specific form of governance that is based on data and algorithms. So, instead of explicitly telling people what to do, the systems analyze patterns and then try to predict and influence future behavior. And the key point they make is that this can look very neutral and objective because it's based on data and statistical models. And it often appears as if decisions are simply derived from what the data shows. But the argument is that the systems don't just represent reality, they actively produce reality because they make certain things visible and actionable while ignoring other things. So, the important aspect of this form of governance is that it works in a very subtle and anti-Sipatory way. For example, through recommendations, through notches and for risk predictions. So, instead of saying you must do this, it says people like you usually do this. But that's good for you, you are at risk and that already steers certain behavior. And raw war describes this as a way of reducing uncertainty, basically learning what is possible into what is probable. In the end, it can narrow down the range of options, people even perceive as possible or available to them. But when you say the system is saying you are like this, you should do that. You make the point that in fact the system's knowledge of you is very much what you and other people have referred to as the data double. Yeah. So, I mean, I'm interested there about what the data double reduces perhaps. How would you describe the reliance on the data double? Yeah, I think it's a heavy reduction because learners are no longer primarily addressed as reflective individuals, but as data profiles. So, instead of seeing a student as a complex person, the system classifies students as, for example, at risk or as high performers. And based on this classification, it starts to guide behavior. So, it not just students to act in ways that align with those categories. So, they might become their own data tables. Exactly. Now, I think that's a really interesting way of putting in it. Lots of people have been critical of learning analytics. You're actually gone one step further and done some critical empirical research on learning analytics. In your article, you look at a particular learning analytics dashboard being brought into master's courses in an Austrian university. Can you just give us a sense of what this university was? Why were they keen on introducing dashboards? What are the university authorities think they were doing? And why were they doing this? So, the case study was a pilot project. And in this project, they developed a dashboard. It was a block in in Moodle. So, they were searching for possible courses where they could look at it and test it. And so, it was integrated into two master courses in the master of business education, where I was teaching and working. And the university from from the university's perspective, the motivation was quite typical for learning analytics initiatives. So, the idea was to use data to improve learning processes to increase student success and importantly to reduce drop our rates. So, dashboards were introduced as a way to give students more feedback to help them to stay on track. And especially also to support more self directed learning in these self learning phases. And across all our Austria universities are increasingly interested in learning analytics as part of the university. of their digitalization strategies. So learning analytics is explicitly linked to goals like improving teaching quality, increasing exam activity, which is really important here, and also reducing dropouts. And so learning analytics is just seen as a tool to make education more efficient, more transparent, and more data informed. Yeah, so that's the background of our study. But we were interested in maybe we could say, taking a step back and asking, what does this actually do to students? So how does it shape their actual behavior? Well, that was going to be my next question, ironically, although the institutional goals of efficiency and visibility and all the rest of it may have been met. But what did you find then in terms of how learning analytics was kind of shaping the students experience? What were they making of it? And was it helping them and empowering them? And improving their learning? What was actually going on? Yeah, we found three main ways in which learning analytics shapes how students see themselves and how they act. First of all, many students developed a strong trust in the objectivity of the system. Many of them, the older us, they feel the system is more objective and more fair than human judgment. Because the system just shows the facts in, like one student says, in black and white, you have just the facts without personal bias. And secondly, it encouraged student self optimization. So the field of action was shaped through rankings, through profiles, like their data doubles, and they define what means to be a good student. And it encouraged learners to optimize themselves according to these data driven norms, like ever-retches, rankings. So students constantly compare themselves to ever-retches, to certain benchmarks, and try to improve the scores. And they became very focused on measurable performance. What was really interesting, they prioritized tasks that count in the system, over less visible aspects of learning. They told us, for example, yeah, now we don't need to go in the library and do our own literature research, because the system shows us what is important. And a third aspect is that the system shaped behavior in a very immediate, almost reactive way by privileging fast and responsive behavior. That means through notifications, through alerts, through real-time feedback students felt pushed to act quickly, to keep up, to turn fall behind. So instead of reflecting on their learning, they often moved from one task to the next, reacting to the system. So overall, we found out that the learning analytics didn't just support students. It's structured how students think, act, than even what they see as meaningful learning. So it changed their view on education. Which is fascinating and also a bit surprising. I mean, did you find any pushback or resistance or subversive use, or did students just feel like they have to go along with where the system was prodding them, where the system was nudging them? Actually, it was quite interesting that the students' processes of subjectuation were highly ambivalent. So between students, but even within one student, for example, they felt like a set of system highly objective and better than human judgment. But on the other hand, they recognized that the system is kind of a black box. So they didn't really know how it works and which exact data it uses. So some students questioned, then, whether it really captures what matters. And that opened up spaces for critique. So some students criticized it because they didn't really know what was measured, how it was measured, which data was important for the system. And we saw also resistance driven by pressure and comparison. A lot of students like these rankings and this gamification system. But especially students who have been constantly below average felt demotivated and stressed. So instead of engaging more, these students started to distance themselves. In some rare cases, even ignoring the system altogether, because they said, yeah, I can't catch up at all. So why I should do anything. And a third form of resistance was about the pace and style of learning, the system encourages. So some students described a kind of task chasing. They felt pushed to react to notifications, to complete activities really quickly, to don't fall behind. So some of them explicitly criticized that this leads to a more superficial kind of learning. And they called form of freedom and space for deeper thinking. Now that makes perfect sense. And I'm listening to you describe all of this. On the one hand, it sounds really depressing and soul-destroying. And there's a hollowed out version of, as you say, a gamified data driven race to the bottom. On the other hand, you could, if you're being cynical, what's the problem here? The universities are highly standardized and depersonalized and soulless places anyway. Is the problem with learning analytics, or is it with contemporary education? Why are we seeing this as a problem? I think, yeah, that's a fair point. I mean, many students are under pressure. Many of them maybe just want to get through their studies, even without learning analytics. But I think learning analytics does more than just support this process, which is already there. I think it directs students attention to our data fight, trying to have statistical version of reality, which focuses on what can be measured and predicted, while often overlooking social and personal complexity of learning. That means these systems tend to reinforce existing inequalities, and rather than to have the full complexity of an individual, the system creates feedback loops, where expectations based on data start also do shape future performance. So for example, one student said to us, it's like a self-fulfilling prophecy. The system already in the beginning shows you what you are expected to be. And I think that's really dangerous. Yeah, and also there's no sense of becoming something other. You can only ever be what you've been previously, which is really dangerous. No, that makes it pretty well put. Oh, maybe you should become your data double. That's also something problematic. Yeah, that's as best you can hope for. Now, it's not enough for academics like you and me to just critique systems. We've got to come up with plausible alternatives. So if we don't like the forms of learning analytics that you've just described, what are you working for instead? Or do we need to get rid of learning analytics altogether? It's not that we should totally abandon learning analytics. Maybe we need to rethink what is learning analytics for and how it's designed. So we describe three possibilities to rethink learning analytics. First of all, we need a stronger focus on data literacy and critical engagement. I think it's really important that students not just use these systems, they should understand better how they work, what are their limits, and they should also be able to question them. A second point is learning analytics needs to move beyond narrow ideas of performance and competition. We think a more inclusive approach would be important, which takes into account diverse learning paths, different spaces, or the different ways of being a student. And a third aspect, we need to create spaces for reflection rather than constant reaction. So that means designing systems that don't just nudge and accelerate behavior, but maybe support meaningful engagement, also interruptions in learning, productive interruptions. So I think a different form of learning analytics would be less about control and prediction, and more about supporting diverse learners in reflection and in equitable ways. Well, really good paper, really interesting to read. Thanks, I have so much, Hannah, for the take in the time to explain it all. Thank you for having me. It was really nice to talk to you.