Episode 20: Learning Analytics and AI: Opportunities & Challenges

We have all been using for example spell checking and searching for years that it's just this chat bot being released last year has brought everybody into talking about AI. Having it as decision support tools that we can give back to our educators becomes really important and I see learning analytics research is playing a really crucial part of this process. Welcome to Solar Spotlight, the podcast from the society for learning analytics research solar. In this podcast series, we have conversations with guest speakers to engage the wider community with leading research, practice and key issues in learning analytics. I'm Shibani Antanet from the University of Technology Sydney and I have a co-host for this episode. Roger, would you like to introduce yourself? Thank you, Shibani. My name is Roger Scalissa and I'm a postdoc researcher at the University of Oslo, Norway, and a member of the Solar Executive, happy to co-host this podcast. With recent advances in generative AI and tools like chat GPT, we've seen increasing interest educators in using artificial intelligence, but how does it relate to work in learning analytics that aims to improve teaching and learning with data? Today, we have two special guests, Anisha and Barbara, who are going to talk to us about learning analytics and AI, what we've learned so far, what challenges are ahead and how we can look toward the future. Can you give us a brief background of your role and your work in learning analytics? Shane Dawson got me into learning analytics, so I worked with him on the SNAP tool, which was an early social network analysis of discussion forums that plugged into Blackboard and Moodle. I've been a learning analytics manager at the University of Queensland from 2019 up until August this year, and within that role, we've basically developed a lot of learning analytics initiatives, including a teacher-focused dashboard with the ability to filter and provide students with feedback. Great. Barbara. Hi, I am the director for the National Centre for Learning Analytics. We have been existing since 2016, but I have been involved in AI and education since 1985, so a long time ago. And I would say we were doing learning analytics before it actually was called learning analytics within the educational technology field. And I would say my first publication was in 1998, where a student visualized some data that we collected from a collaborative workspace for me so I could see how groups were collaborating together. Great. And I think that's a very good start to give a new experience in the area. But before we dive into AI, maybe Barbara, could you talk us about your experience in implementing learning analytics in practice and the barriers we have encountered so far? Yeah, so one of the projects that we have had has been about implementing learning analytics in the municipality of Oslo. We were the research centre together with the Department of Education, and they want to collect data from across different vendors and be able to visualize it together in a dashboard. So we were developing the dashboard and working on that. So this project has been very interesting in that it's highlighted a lot of the issues that are involved when you're actually implementing learning analytics. And maybe it's a bit different than Australia because we have the GDPR in Europe that really guides our data protection laws. The data protection laws together with the educational laws in our country decide what we can and can't do with data. But because we were a research project, we have a little bit more leeway. But there's many barriers to it. Some of them are related to the educational technology companies that have to put their data into a particular format for us. They have to share their data with us. We have to collect the data from them or they send it to us and we can't do this necessarily in real time. So issues of that kind come up. Are we allowed to collect the data that we need? That's another issue in the Norwegian system discussing whether we're actually allowed to collect data in schools. And in fact, some of the work we've been doing in this project has influenced this national report we've made for the government asking them to make our educational laws more clear as to what is possible and not possible because we sort of see a little bit of a freeze in the sector right now because people are afraid to do something. Other issues are the teachers ready for these kinds of dashboards that we give them. Do they have the time to use them? Those kinds of things came up. Who owns the data? That's also a big question. Is it the municipality that owns the data for the children and keeps it for the children? Or is the hidden data that is created when they use tools? Is that owned by the companies? These kinds of questions. Also, it brings up other many legal issues about parents and access to data. Parents are actually, by the GDPR, allowed to keep their children's data from being used for training algorithms. So you always have to have that way to allow them to take it out. We have questions about whether the privacy officers working in the different municipalities have the right knowledge about educational systems and the use of children's data to be able to make decisions. Right, so it seems like there are lots of questions before we can actually use these systems with students, particularly in schools. It's interesting to think about these in the context of how powerful AI tools are becoming more and more accessible to teachers and the potential opportunities and risks they bring along. So Anisha, maybe you could expand on the challenges you see for educators, institutions and learners with all the recent advancements in AI. I guess I'd like to say that I can see that there'd be some productivity savings for educators and that generative AI in the form of large language models and just the different types of chatbot tutors that it affords will really help learners. But I guess I'm a little bit weary that in the past we haven't actually seen productivity necessarily lead to workload reduction. So essentially, we just have new ways of doing things, but workload still increases. So those productivity gains would probably be fleeting, whether it's helping you generate content or helping you create your PowerPoint slides or design a lesson plan, particularly. Yeah, as educators, there's always something more to do, I guess. That's exactly right. But I do think there's an opportunity particularly to support lots of diverse learners with things like adaptive pathways, different types of chatbot tutors to actually encourage problem solving and critical thinking. And I think a lot of this would be because we can now make content a lot faster and generate different types of assessments. The thing to really think about in all of this is that it is very, very easy to make a chatbot tutor. And I keep thinking back to when Khan Academy released KhanMego, I think it took about six months to actually get the chatbot behaviors correct, you know, so there's not giving direct answers, it's actually supportive tutor and a number of other sort of behaviors. The moment is very, very easy to upload a PDF, enable the chatbot tutor that I think we're not really evaluating for learning particularly. And so therein lies the challenge of how do we get that right, how do we make sure the right guide rails are in place and, you know, can't be avoided or students can't bypass them so that we're not creating safe environments for students. And I think possibly related to that very question, maybe I'll start with Barbara. Can you provide all thinkables and specific examples around recent AI or adaptive learning tools that have positively all have potential to actually impact education outcomes? Yeah, and I think this comes to one of the issues that we've always had in the field of technology enhanced learning is actually what is it that we're measuring and what evidence do we have. And in one of the critical reports we wrote for the Council of Europe entitled, artificial intelligence and education, a critical view through the lens of human rights, democracy and the rule of law, that's just the thing we argue is that there's not really a lot of evidence that it really supports learning. And I think Anish was in that changes the way we do things often, and we know in collaborative learning, for example, that was, we went away from asking about effect to looking at how it changes processes, right, how knowledge processes change, etc. And I think it's going to be the same here. We did a large study of one of the tools that's being used in schools in Norway, it's an adaptive learning tools, and it's in mathematics where many of the tools are, in fact, for schools, where the children get an adaptive path through tasks. And you have to look at what the role of these particular tools are. And in this particular case, this tool was replacing the workbook where kids might work on, you know, they'd have a page in the end of the textbook where they would or in the workbook that was accompanying. The textbook where they might have like 20 exercises to work at. What we found was they instead of doing 20 exercises a week, they were doing 200 exercises a week. So the amount of exercises they were doing and not only that they were tailored to them. But then of course comes things like kids figure out the system and they start gaming it and they figure out if they answer wrong, they get easier questions, they get out of doing their work faster because they can answer easier questions, etc. So you always have to look at how the tool is actually implemented. But this tool was based on the Newton adaptive engine, but it also provided a learning analytics dashboard for the teacher. So the teachers were able to keep track of where everybody in the class was going on, which is also another challenge in schools where you've got kids working at different paces all over the place. How is the teacher going to be able to really follow up 30 or 30 plus students to know where they are? And that's where learning analytics comes nicely in and can give you a nice dashboard to give you, you know, analytics over how the whole class is doing, how individual students are doing if you want and gives you a tool that enables dialogue between you and say parents for parent teacher interviews or you and the children, you and your fellow teachers in other classes, etc. So it gives us a sort of boundary object for discussing, learning with and how things are going. You know, we've had many tools, those are adaptive tools, but there's also as a critiquing systems and we've got as a generation. And it's not always the big tools that we see in education, either, like lots of people are using like speech to text now or text to speech and generating, you know, animations and videos and being creative and those kinds of things. So we have to remember, those are also sort of applications have all been using, for example, spell checking and searching for years. It's just this chatbot being released last year has brought everybody into talking about AI and making it more, you know, prevalent for everybody to talk about. Yeah, absolutely. There's been decades of research in artificial intelligence, learning analytics, lots of tools tried and tested. But the new chatbot, it can take different roles, it can sometimes be a tutor, it can be a grammar checker. So it seems like there's a lot of power in bringing the capabilities of these together, but it obviously comes with a lot of challenges. So Anisha, maybe can you give us some examples of where you think the potential is in terms of specific examples, but also what we need to put in place to tackle some of those challenges that the tools will bring? So I think one of the ways artificial intelligence can actually enhance learning analytics is that learning analytics has traditionally always had the problem of we collect a lot of clicks. There's no way of mapping that to higher level learning constructs. It's been coined as the clicks to construct problem. So I do see some hope that these large language models do show some promise in terms of advanced classification schemes that we can make and make via what they call a few shot learning, which just means we can give it a few examples and then see how the classification does so we don't need to label large datasets. And that's also been another problem in learning analytics with making available the datasets that people use for, you know, ethics and data privacy issues. There are not a lot of learning analytics data sets that are public for people to run experiments on. So I see the advances that have happened, you know, particularly with chat GBT becoming available and having an API, particularly, that we can start to really look at some of the classification schemes, like community of inquiry or the dimensions of self-regulated learning to get to these higher level constructs like self-efficacy, intrinsic motivation, things like that. What else it will enable is just novel applications where it's not necessarily via a chatbot user interface where we'll actually be building custom user interfaces. So whether this is to help people actually write and I think this is right up your alley, Shabani. Yeah, for sure. New UIs that have, you know, instead of where we'd make things with a toolbar where you'd make things bold or italic, it might be where you've now got things to actually change the text, provide feedback on the text, rewrite it for you, give you new additional ideas and things like that. But there's also possibilities of helping very specific learners. So I have recently seen a publication where it was a very specific tool. I had a semantic UI, but it was really to help people writing emails that had dyslexia. The tool is called Lampost. So I see a lot more of tools like that becoming available. I can already see, you know, I'm really looking forward to what's happening at 2024's conferences now that researchers have been able to use these APIs just to see what sort of high level classifications they're using, just to see what, I guess, how is lag changing. You can already see that in the workshops that are going to be happening at the next lag as well, just that AI is being infused into different things. Yeah, what do you think about it, Barbara? Do you think it's a boon or a bane? Well, the thing is, I come from the old school where we had rule-based systems and we were in control of the knowledge that students were interacting with. So it's a new challenge to us when we don't know exactly what it is the data they're trained on. And maybe when we get more possibilities to train it on our own data, maybe on small datasets, yes, but on larger datasets, it's going to be too costly and different languages. But what worries me, for instance, there was a med student who was telling us that he was using it to generate patient cases. And the thing is, is that if you know the work from the last 20, 30 years, there's been a lot of work on generating patient cases for students to interact with and things like this. And it was controlled. Now you could have a chatbot that could generate a patient that could never really exist. And is that really something we want? So I don't think we know enough yet how it's going to impact the different disciplines because I think you would have to be careful to make sure that this collection of symptoms that maybe could be generated for a patient. Are really feasible in the real world and something that, you know, people could present themselves to a doctor with this, you know, this situation. So I don't think we know yet. We don't have enough evidence yet to know exactly how it's going to work. But as some of the examples Anish gave, there's, there is, it's very exciting to see what's going to be happening. But I still really wonder, like, who's going to own the knowledge now? Is it these big companies in the U.S. owning knowledge and defining what knowledge is? Or are we going to be able to maintain that as our own fields of research as, as, you know, defining what knowledge is for us? I don't know, it's going to be exciting to see. I think that's one of the big issues is how do these chatbot shooters get evaluated, particularly there's larger companies in control of it. Like get them on with open AI and their release of GPTs. It's really, really easy to just, you know, upload your own data, credit chatbot for it. But the actual behaviors, like, how do we validate the behaviors are positive for learning? At the moment, the technology, while it's really quite amazing, the breakthroughs that they've made to actually get you, they do hallucinate. And in circumstances, that's going to be really detrimental to learning if you put your trust in some of these things. But more importantly, I think, just depending on the age groups of the learners that need to interact with them, it's quite easy to break the guide rails. And these chatbots can really go off topic and start giving advice in inappropriate domains as well. And so I almost think, you know, the host is bolted, and this is already out there. But educational communities and researchers haven't been able to guide this to evaluate it in the correct manner. Yeah, it's too early for the evaluations. But I think one thing that's going to be interesting is that, you know, industry and workforces are taking this into work as well. So what's it, you know, the relationship between then the education and what's happening out there? For example, law, we've already seen cases in the states where people have taken precedence into the courtroom that don't really exist at all. Here in Norway, my colleagues tell me there is evidence that the students do worse when they use the chatGPT in the law discipline as well. But I'm sure that this is somehow going to work itself out over the years as we see how the tools are adopted in industry or in the profession as well as how we allow them to use it in the school. So at my university, they're looking really carefully in the different disciplines as to how it can be used and whether it should be allowed to be used for different things as well. And it's very, very discipline based. Yes, definitely. And that just reminds us, right, that opportunity comes responsibility. So what do you think about the current state of AI policy and guidelines? Particularly, I think Barbara, you spoke a little bit about the policy in the EU. How does it look different now that we have these tools out in the open, educators, probably students trying them out themselves? What are the implications for data and student privacy? Well, I think there's lots and lots of questions around this kind of aspect. And as I say, it's going to be very dependent on which part of the world you're living in and how it is working. For instance, I think even everywhere that it's not recommended that children under 13 use the chatbots that are existing now. In most countries, it's not allowed to take children's answers to essays and put them into the chatbot and have them marked and generated. But we know people are doing this all the time. And who is it who's actually responsible for if things go wrong? You know, how can we trust that the technology is being used in the right way? And we're in fact just starting a new project this month that is on trust in the educational system to these kinds of AI systems. And we have on the one hand the technical side where trustworthy AI systems are fair, they're responsible, they're explainable and they're accountable. And on the other side, we have societal acceptance of these kinds of tools and the trust in them. So the potential harms and risks that the UN has brought up and UNESCO brings up, UNICEF brings up about this, that we don't know enough about what happens when we're profiling children using other data, etc. We don't know all the harms in the long term for them. We have a lot of dilemmas and we can't be expecting the teacher in the classroom. If I talk about schools, the teacher in the classroom to have to make all these kinds of decisions. So how is it going to be regulated and where the policy is going to come? And as I said, in Europe, we've got GDPR, we've got the AI Act, which now is going to be about the use of AI and the regulating of the risks. Associated with that. And then I work with the Council of Europe and we're about to work on the next two years on a regulation for the use of AI. We're looking on policy guidelines and we're looking on how these things can be regulated because where's the accountability and who is it who's going to regulate these things. You know, and it's quite different when you're talking schools versus universities as well because you're talking about minors and who's going to have the right for them, you know, and who's deciding what tools they're allowed to use or not. We have activists in this part of the world at the moment, both in Sweden and Norway, wanting to get screens out of schools even. So we're back to, you know, very basic challenges. Yes, we've, in fact, seen the ban of chat GPT in schools in Australia and in the US and so forth. Anisha, what do you think about policy and regulations and challenges behind regulating AI for education? The challenges, as far as I see, just become more and more over time. So one of the things I'd like to see with policy is addressing equity. So at the moment, the most capable model is GPT4 and that's subscription based. So I can definitely say people that use that are able to do much more. It's got better reasoning. It's got better mathematical capabilities and better ability to adopt different personas. It's a great for agent building. It also incorporates tools. So you can use things like advanced data analysis tools, any Python programming tools into your workflows, as well as it can browse the Internet. So that's not freely available. Then also got the fact that OpenAI has stopped new subscriptions because they just don't have enough GPUs to handle the load, basically, of what this is. So how do we make this fair and equitable? So that's one way of thinking about it. The other side of it is, I know the US regulation or where they were headed with it or their document was really looking at trying to regulate these large language models that are over a certain size, particularly. I've tended to disagree with that. Just seeing what open source communities are doing and the smaller models with fine tuning are actually becoming quite capable. So this whole aspect of actually banning it, I think, is not going to work because these open source communities are going to make these smaller versions. They can actually run on a laptop and we'll get there in the next year or so, essentially. And that really complicates it. OK, so it looks like those are not going to work. So what else can policy or regulation do? I think where policy should really be headed is to try and set within the curriculum where we think people should be self-reliant. It comes back to this idea and notion of is, and I know Andrew NG has looked at machine learning. We called it like the new electricity. This is the problem, basically. So if electricity goes away, how do you adapt for that day? Do you just sit back and do nothing until it comes back on? It's the same goes for when there was an open AI outage last week. I was going, oh, no, it's down. It's gone. What am I going to do? Where should people be self-reliant? Where should they actually be able to write at what level is it appropriate for them to write an essay and they must do it on their own? So we make sure that they're capable of doing it on their own versus, OK, now you can get the supports of AI. Now you can collaborate with it inside of it. That's the way that I basically look at it. And I think Australia is getting this right with the Texas paper that's come out where it's really focusing on the sort of collaborative nature. People, once they graduate, will have to be using these technologies. And so we need to start incorporating it while still maintaining our level of assessment. Preston, and I think looking ahead, maybe going back to Barbara, what trends do you anticipate in the intersection of learning analytics and artificial intelligence? Do we see any potential disruptions, maybe for either fields? Will learning analytics possibly be for ways to improve in any way by AI? So where do you see this relationship going forward? Yeah, I guess it's what you see yourself as the relationship to between AI and learning analytics. And they're very tightly intertwined and they have synergies between them. In fact, in our Council of Europe report, we have learning analytics being part of AI. In particular, learning analytics, for me, always has to have a stakeholder. You're giving information or giving insights and possibilities to make decisions to a human in the way. Whereas if it's an automated decision for me, then it's the AI. Like if you're making the decisions about what tasks to give next in a tool, that's AI to me. Whereas giving a visualization or an insight into what concepts have been covered or the paths the student took, that would be the learning analytics there. So I think they feed each other and their synergies on it. And of course, if we're going to have predictive learning analytics, we're using traditional machine learning or deep learning algorithms. If we had enough data to predict dropouts or things like that, then we're definitely... The learning analytics is showing the dropouts and the prediction on the one end, but using the AI to generate the models, predicting that. So for me, they're very, very tightly intertwined. So I think that the learning analytics makes the AI systems relevant for people. You know, it gives them the possibilities to make decisions. And that's actually a legal issue as to if something that's coming out of a system is automatically used, there is a different legal thing than if it's a recommendation. And that becomes also something that has to be looked at. If they're generating marks for students, for instance, it really needs to be coming from a human, not from a machine. Although the machine information or data can be used to help make a decision. Nisha, do you like to expand on that? And something that came up in our earlier discussion that's probably worth resurfacing is also the potential of using learning analytics and data to inform research in AI and education. While there are shortcomings like the medical case that you mentioned, Barbara, in the use of synthetic data, it could also be useful for education where we cannot use student data because it's quite private. How do you think that comes up with the two fields? Yeah, that's very interesting. So one of the things that I'm seeing a lot at the moment with the way the open source community, and perhaps even open AI, I've done this, is in how they have trained and fine-tuned their models. And some of that data at the moment is actually being synthetically generated. So they want certain behaviors in a chatbot. They use another chatbot to generate the data that they need for those behaviors and then use that to fine-tune the smaller model. And that's how they're getting a lot of these smaller fine-tuned models to get good performance. So some of that might be really useful for making datasets quickly for learning analytics. But then I think the important part there is really to get humans in the loop to actually, you know, that was that whole reinforcement learning with human feedback aspect of it, where someone actually is, you know, rating what the synthetic data actually might be doing. So we can actually then get datasets that we can share and use for research purposes in that manner for things like writing analytics, particularly you could look at in last chatbots to, you know, what are the most common errors and generate sample writing with these common errors in them that we can use for doing those sorts of things. But I think the early papers that I saw coming out from the start of this year, were looking at taking a particular dataset. It might be something like reflective writing or grading of essays or something like that. And then just using API to try and get it automatically graded against a rubric and then comparing it to the human tutor that was involved or the human teacher, how they graded it. And so a lot of this was, it went into this aspect of auto classification. What I'd really like to see move learning analytics forward is having it more as decision support for educators, because we don't necessarily want everything to be 100% auto graded. We still want our teachers to know where students are making errors. We do perhaps also want to improve marking practices and fairness and have a way of obscuring tutors if they are marking things. So this is where I think that whole aspect of, you know, explainable AI, having it as decision support tools that we can give back to our educators becomes really important. And I see learning analytics research is playing a really crucial part of this process. The hard parts for us really is going to be decision support, particularly explainable AI in other domains like law and medicine have been going on for quite a while. And there's still trust issues. There's still aspects where people that are meant to use it don't trust it, or it doesn't improve the human accuracy in having it there as a support. So there's a lot of challenges in that area. But essentially that's where I'd like to see learning analytics head. Great. Thank you. So I just have one final question, which is about words of advice for educators and institutions that are looking to harness the potential of AI. So what would your best advice be for them? I think that we really need to have AI and data literacy among everybody, especially in the school system, not just for teachers. Everybody talks about the teachers and the students having to have the literacy, but I think it's all the way up in the system that the school leaders, the privacy officers working with the regulatory bodies that have to regulate in the countries, have maybe not dealt so much with education so far. So there's a real need for literacy to raise it so that we can have safe use. Yeah, that's a great point. What about you, Anisha? I think we need to not make the same mistakes of the past. And by that I mean, at the moment, all tools that are in the learning management system, ecosystem or add-ons that we buy for it, it is really hard to get the data out of the vendors. And it would really be a shame to be making the same error when we get chatbot tutors coming from these vendors. And there must be a way to actually track almost message for message what's happening inside these tutors so that it's all sent back to central repository back at the, whether it's education departments for high schools, primary schools, or the university if it's higher ed. Because that's where you'll get the centralized learning analytics running from. And I don't think learning analytics outside of a research study context is useful if it's only provided per tool. I think it's across the whole life cycle of tools that the student uses. We will get the full picture. And so if we're introducing these chatbot tutors and we can't get the data back from the vendors because it might be proprietary or they have intention to do their own learning analytics, I think we'll end up back where we are with a whole lot of tools that we cannot make any insights from. Absolutely critical to get the infrastructure ready and done at the start. And good reminder. Thanks Anisha. Thank you so much for this interesting conversation online analytics and AI. Great to have you on the podcast and thank you for sharing all the experiences. And we hope the community will really enjoy the experiences you have shared. Thank you so much. At the end of our podcast, we invite our special guests to play fun game called "Two Truths and a Lie". Our guests will share with us three statements about themselves. Two are true and one is a lie that we should find out. Here are the answers from our last podcast before we hear from Anisha and Barbara. Sambit. So the first statement was I love cooking so much that I share photos and videos of the dishes I cook. It is true. I also have an Instagram page and also I share videos on YouTube about my cooking. I love cooking. The second point, I was introduced and then started my research in the field of learning analytics by choice. So that's a lie. And the third point is also a truth which is I love painting and even have a junior diploma in finance. Anisha, would you like to give us your three statements for "Two Truths and a Lie". Three statements from me. My best TV series was from the 80s and it was MacGyver. As a child, I used to love to play with Barbie dolls and I'm a huge Madonna fan. I'm going to leave the audience to their guesses and go with you. Barbara, would you like to give us your three statements? Yeah, I have lived in 14 different cities. My favorite TV show is Masterchef Australia and I'm a figure skating judge. Interesting. Let's do that in the next episode to find out what the lies were. Thanks for listening to Solar Spotlight. Conversations on learning analytics. You can subscribe to our podcast and find all available episodes on SoundCloud, iTunes and Spotify. To remind you, registration for the learning analytics and knowledge conference which is being placed in Kyoto 2024 is now open. Take advantage of the early bird registration which closes 29th of January 2024. You can still submit your work as a poster, demo or a workshop paper until the 16th of December deadline. My name is Roger Scalissa and I'm Shibani Antoinette and we've been talking to Anisha and Barbara on learning analytics and AI to enhance learning. If you'd like to continue the conversation, please tweet us @SolarResearch using the #SolarSpotlight.