72. Generative AI Hub: How Stanford is Empowering K-12 Ed Leaders with AI Tools | Chris Agnew

in a world with artificial intelligence, how do we transform education? Thanks for tuning in. I'm Daniel Lopez, and this is the AI education conversation, where we explore the opportunities, risks, and impacts of AI across education. If you'd like to join the conversation, check out the AI education conversation.com. Let's jump in. - Chris Agnew, let me officially welcome you to the AI education conversation and I'm really excited to have you today. - I am so happy to be here, Daniel, thanks for having me. - Chris, I know we've spent a little bit of time just off my care being able to talk to each other and really excited for this conversation, but wouldn't bite you first and foremost of, you know, for the folks who were listening and tuning into this episode, why should they listen to our conversation that we're about to have and what would you hope that they would get out of it? - So there is so much noise in AI right now, AI broadly, and then certainly AI within education. And I would hope at the end of this that listeners could feel like they might have a couple tangible frameworks to start structuring some of their thinking around AI and its applications in education and specifically K12. I think, or I hope that they would also gain a little bit of awareness on where we're at with the research and what the research is saying about AI and education and not just the hype, but like what is actual studies telling us right now. And more importantly in that realm, where they can go to get some more information and educate themselves. And here's the secret in all of this that I wanna give everybody who's listening permission to do with this topic, is my hope is people are walking away on this topic with AI and education, with the feeling of permission to be a beginner on this topic, in that it can feel inaccessible, it can feel like there's a bunch of technology first people that are experts in this realm. And I am a high school principal, what do I have to add, how can I tackle this topic? And if you're to take away one thing from this is, there's a lot of noise in AI and education, there are no experts. We are learning this as we go. And so it is imperative on all of us to lean in and start experimenting and get curious and explore ideas you have. And being approaching it with a beginner's mind is actually a super power and people should feel ready and encouraged to do that. - Yeah, well said, I think especially the perspective there of saying with so much noise in the field, what is the research tell us and what can we learn from the research feels like a really important lens to explore as we really think about optimal, the optimal role that artificial intelligence really can play in our educational experience. I know that in your day role, in your day-to-day job now, you are the director for generative AI for education, the hub at Stanford. I'm curious like how'd you get there? - Yeah, so I do not have the standard background to come to and be thinking about technology, specifically as it applies to education. So a little bit around my 25 years in education. So I spent two decades working in non-traditional learning environments that were both adjacent to K-12 and posterity. And when I say non-traditional learning environments, what I mean by that is wilderness as a classroom, community as a classroom, a apprenticeship as a classroom, service learning, things like that. Learning environments that are immersive, experiential project-based hands-on that often teach many of the skills that as educators, we know are of increasing importance in success and work in life often referred to as 21st century skills in the categories of social-emotional learning or durable skills, people call them many things, but ways to communicate teamwork, things like that. In 2015, I took a role leading an organization called the Teton Science Schools. In Teton Science Schools, we had a K-12 school. We hosted school groups from around the country to learn next generation science standards in the national parks. And we had a graduate school for education in partnership with several universities and colleges. And for me, it was this amazing opportunity to do what I'd been aspiring to do much of my career in blending the more formal academic setting of learning often in the classroom with this more experiential learning and community learning outside of the classroom and in more relevant and hands-on ways. And I was there six years, and I left in early 2021, and I left for a couple of different reasons. One, just fatigue and keeping school doors open during COVID, of course, like many education educators and education leaders in that time. But second reason was actually more important. I'd worked these two decades in these immersive learning environments. And this organization I led was based in Jackson, Wyoming, one of the most philanthropic locations in the country. And what I learned is the business model for these immersive learning environments was broken. Even with massive philanthropy, there was no path to scale and access for students. There was no path forward. And I recognized even with these massive philanthropic gifts, if this was impossible here, how could it be possible for the millions of kids across our country that need these learning environments? And so recognized technology has to play a role in this. And I did not previously have an experience working in ed tech. And so I spent the last three plus years working for a couple of venture-backed tech companies, most recently working for a company called Multiverse. Multiverse founded in the UK, using apprenticeships as an alternative to university. They entered the US market. I was earlier on the US team, and I held US higher ed strategy and credentials. And while at Multiverse, I got my first kind of aha moments around AI. So, and this I think is a great application for what I think some of the real promised AI has in actually making education more human, ironically or paradoxically. So here was the early application. So at Multiverse, we had coaches, human coaches that worked with the apprentices. The coaches were the learner-facer like a teacher and instructor. And all apprentices' student work were based on real products and real deliverables that they created working on the job. So all of it was authentic assessment. And these outstanding coaches would spend hours every week greeting these spreadsheets or slide decks or whatever it was that the apprentices were creating on the job and the coaches would review this authentic student work against the student outcomes that the apprenticeship needed to hit. Important work, truly authentic assessment, really time intensive. And so we started thinking about and working on how to leverage AI to review student work and match it to the expected learning outcomes of this period of the apprenticeship. And it would still be human-reviewed at the end. But the goal was can we reallocate what previously these coaches were spending sitting in front of their computer, staring at spreadsheets and can we save them time so they can get more one-to-one time working directly with apprentices and the time that really matters in apprentice work. And in that moment I recognized there might be something here with AI in my whole career working in these environments where learners engaging with the real world, doing real work, project base. That type of learning is really hard to create is operationally complex and can be expensive to run. And those have been historically so many of the barriers that prevented it from accessing all the kids that truly need it. And could AI create an opportunity where this type of learning, this more hands-on, human first learning is actually more accessible to more learners broadly. And so that's what got me really excited about AI and ultimately, fast forward a few months, landed me taking off and leading our gender advanced or education hub that's part of the Stanford Accelerator. - Wow, I really appreciate that, that arc there. And so much of that is something I didn't even, you know, did not come up in some of our initial conversation. So I have so many reactions. I'm curious to make sure I'm understanding the initial learning though from the multiverse piece. It sounded like the AI use case that you already have. landed on that that was highly effective in that context is similar to what some of our schools have found right now, which is AI was you were really able to help leverage AI to lean into more of the like operational, some of the like administrative type work and as it means to try to make that more efficient so that your human talent resources were like had more time to lean into the pieces that fell much more complicated to deploy AI to use, especially like in a service learning context. Is that the right way to think about it? Because I'm just so fascinated to think about how we scale service learning type experiences as well and I don't want to make sure I want to make sure we don't lose that thread. Yeah, oh no I'm glad you're drilling down on that. So I'll introduce one framework here to start organizing some of our thoughts. So in our work with lots of districts and schools across the country and states, we're noticing that often practitioners are approaching AI kind of the Y use AI for one of three different reasons. The first is the efficiency play and I think it's a little bit about what you're talking about. Can it streamline a system? Can it save me some time? Which is there's real value in that and it's in an optimum, it's an optimization play, right? But efficiency can it save me time? That's one. Second category, now we get into the more why we're in education teaching and learning and the second category is generally around outcomes. Can AI help us move the needle on outcomes and it could be about literacy outcomes, it could be about better personalization or differentiation, it could be about getting better at assessing the things that we found a hard time, a hard time previously assessing the things in the social emotional realm, but it's all around moving the needle on. And the last category that I think for me is always really important to talk about connected to the first is how can you think about or use AI to reimagine school. So the first two is about kind of existing or improving what is. The third is about reimagining what's possible with this powerful new technology that we have and it can be easy for and especially with most of generative AI right now lives in the private sector and as a result many to most of those companies are venture backed companies that have a timeline to get to profitability and so they need to solve real problems now which is great. And while solving the problems of next week or semester as a school practitioner, school leader, we also have to be thinking about a more aspirational vision and this brings me back to where we started of the real powerful ways that Gen AI could push and evolve education in the future on how does it change the structure of a school day? How does it change how could it change what school actually looks like how it connects to real life connects to community. So bringing it back to specifically your question, I would say this multiverse example falls into the for I'd say first two categories. It's in the realm of assessment right where there is lots of interesting promise around AI and its applications and the realm of assessment. So there's one there's the speeding of assessment in that can we get students feedback faster and save teachers time in doing so that's great. And I would say it also starts to bridge into the outcomes category as well and I'll tell you one story about actually some research that we just kicked off actually last week about feedback on student writing where AI can be a tool for teachers to get feedback on student writing much quicker and in this case it's not only about speed where say an eighth grader might have turned in their their two page five paragraph essay and the teacher now has many many essays to grade that it might take them two weeks to get turn around and get feedback to those students on their essay. Here's an example where could AI be effective in getting feedback to students faster measured in a few days rather than a couple weeks. And what that allows is the student can get three reps on their essay they can do submit one draft get feedback on that AI generated feedback teacher reviewed the student can take that integrate feedback submit a second draft same cycle AI generated feedback teacher reviewed they integrated that feedback and then get a third. So in that two weeks time where previously they got one rep on their writing with feedback in this case they've gotten three reps on it and cannot move the needle on their writing skill. I'm so grateful that you're the person doing the AI work with the background that you have because I think if if I were to try to speak for I think a voice that I hear out in the field I think a fear that many folks in education and I would put myself included into this vote is that the one of the potential scenarios we're locking towards with the deployment of AI and technology in the classroom is just every student on a laptop all day doing little modules sitting in and not really moving throughout the world gaining anything in the world right that outside of this like laptop that they have and I think that there's tremendous learning and value of course that one can gain by accessing and engaging with technology but the role of education and I think very much in the ways that you've described it's it's not just about the knowledge acquisition or even the skill development it's about the self-actualization that can happen throughout those experiences and I just think as an example about some of the most meaningful moments I have throughout my career helping young people apply and successfully attend colleges and so much about that experience is not about just applying to the schools and the x's and those of doing that it was about a lot of kids getting on a plane for the first time and having that experience and going to a completely new place and like just being in a new environment and all of the emotion and experiences that are coming up from them and then being able to connect that to skills throughout life that they're going to need to in life you're going to have to take risks and it's scary but it's also can lead to growth right in lessons like that that then allow them to have just growth and new just unlock new facets of what it means to be human on the earth and I think we cannot lose sight of that within education we need to continue to run towards that and technology I think can can play a meaningful role but it should not be replacing that or diminishing that and I just love that you are leading this work with knowing that that is going to be something you're going to hold very dearly as the person making decisions about that so just really wanted to double click that because I see why those two things are so important and I think a lot of folks listening to this are going to understand that as well. Oh I love that you highlighted that and I get so excited thinking about that because to me that's that's the whole ballgame there as far as using the opportunity to use AI to actually bring more human oriented child centric learning is is possible and we are at this key infection right now on you can envision a very dystopian world of as you said a bunch of kids with big headphones on steering and staring at individual computers plugged in quote unquote learning and AI empowered you can see a world that this creates way more powerful learning with kids outside of the classroom learning in teams inside their community helping teachers assess skills that we never knew possible how to assess that scale before but now we can understand how to help this child get better at their communication and this child get better and measure their progress and teamwork because of this powerful new technology and makes it I think I get really excited about that and I'll say one more thing and we can dig into this more in the future but there's a lot you know for the last at least two decades the holy grail in education often has been personalization of like let's personalize education which is wonderful like each human each learner has their own strength gaps areas for improvement blind spots and the more we can help young people accelerate when they're ready to accelerate and spend more time when they need to spend more time that's a good thing and and I think this is important around AI we should not confuse personalization of education with isolation of education and there is a hazard in confusing personalization with learning in isolation in that kind of plugged into my own laptop and I see a future where AI enables more team-based learning more learning through project that allows the development of all those skills that we know to be so essential for life, for career in the future around teamwork and leadership and conflict negotiation that if we played full personalization out to the learning through isolation, kids are not going to have the opportunity to get personalization, not isolation. We'll say, Chris, if I was a fly on the wall in your conversations since you've been in this role on this journey, you know, representing Stanford, you know, doing so much work with AI and education. Like, what would I see? What types of conversations have you been a part of? What are you talking about? So we started the Generative AI for Education Hub about in July of 2024. So it's been what's a quick math on that about nine months. And we've spent that period of time understanding what our key value add here is. There are a lot of players focusing on AI and education and there's a lot of work to be done. And so we wanted to be sure that we were playing a unique role that would complement all the other outstanding work being done by other research institutions, by other nonprofits, by other tech companies, etc. And so what we have established as the purpose of the Generative AI Hub is to be the trusted source for education leaders. So specifically, superintendant state and federal K-12 leaders. So we're talking system leaders that have the ability to scale impact to be that trusted source for them on what's working leveraging Generative AI to benefit student school somewhere. And we have three core priorities in that. One is research. Two is tools. Three is engaging. So breaking those down, research. One is we're in a very nascent stage with Generative AI and Education. And so we want to do two things here. We want to focus the spotlight on what we know that's research back right now around Generative AI and K-12. And actually, just in January, we launched the research repository. I'll get to more on that later. But shine the spotlight to evolve the conversation for more we've been for the last two and a half years, which is about use case generation around Generative AI, which is great. But evolve that to also include efficacy. So where are we starting to see impact? Where are we starting to see real promise? So we can double down in those areas. And then starting to conduct our own research as well. So we have two research projects that are in flight right now and we have a lot in the pipeline to come. So that's research tools tools. We're not talking like we're building ed tech tools here. We're talking about building tools for practitioners to help them make better decisions for the districts and schools and state education systems that they lead. So this is example policies or playbooks or best practices or things like that. And then lastly engagement, this is about bringing together for this work to be effective. We have to bring together ed techs. We have to bring practitioners, states and districts and researchers. So we can push forward what we're learning faster. And so the theme of a lot of these this work right now is speed to insight. Historically, education research has been not particularly fast and maybe very rigorous, maybe often finding wonderful insights and impactful insights. But what's different now is this technology is changing so fast. And that we could build a amazing research study over two school years, over three school years. And in we put our heads down, we do that work. And in December, 2027, we come out with some amazing insights on that. And I know superintendents and state leaders might say, that's wonderful. I wish I knew it two years ago. It's totally irrelevant now because the technology, the tools, the applications have changed entirely. And so our challenge is how do we break down the barriers to doing research and get these insights out faster. Even if they are not definitive yet, but they are early indicators of it. Here's what we know right now. And there were some great early models of this particularly started during the pandemic. The Center for reinventing public education did some great work during COVID on kind of rapid cycle research on what schools were doing at certain moments in time during the pandemic and sharing that out in really effective ways for practitioners to access to help schools evolve their best practices during COVID. And so I think that's a great model for how we can approach while not a pandemic, a different fast moving topic, which is generative AI. And then the second theme, the one theme of speed to research. And then the second theme that we've been talking a lot about is this is such an bridging research to practice is a commonly used term in education. I think often because it's so hard to do good research can happen. And it actually doesn't change what's happening in schools. And so something we talk about every day is ways that this can be made more practical, more actionable for the district CTO who's making buying decisions on if and what Gen AI powered K12 attack tool they might be considering. Yeah, it's fascinating. There's so much there that you're sharing and I'm as you said, I'm excited to get into their repository and what that's starting to shape up as in just a second here. But so much of what you just said resonates and I think like what's coming up for me is to transform education in the United States is just so difficult because of how decentralized the system is. And the spectrum and the context of schools is so vast and so large from urban districts to rural districts and in everything in between and they all have different and unique needs. And what I'm wondering is as you all are thinking about not just like making this practical, but like what do you ultimately it maybe the question underneath my question is we had a pandemic and even through that pandemic there was some changes such as like laptops or much more ubiquitous than they were before the pandemic at most schools. But what I'm also noticing is a lot of things immediately reverted back to what they were before. So the system is very persistent and what is it about the combination of these three things or your unique approach that maybe creates some confidence that you all can approach it in a different way to inform a system that has been really persistent over over the last hundred plus years is my not so easy question. Yeah, I would say no easy answer is there and I certainly feel like we have a unique gift at the at the Generative AI Hub in kind of how we're situated and our kind of peer departments that we get to work with. So a little background here. So our research lead is Dr. Susanna Lo. Susanna was a tenure professor at Stanford for a couple decades and then she she loved Stanford. This is pre-pandemic to lead the Annaberg Institute for Public Education at Brown University and she was there during the pandemic as well. And in the early days of the pandemic Susanna and a small team stood out something called the National Student Support Accelerator. And so the National Student Support Accelerator, their goal or what they were foreseen was okay kids going to remote virtual school a pandemic in flight here. There's massive potential for learning loss and worry. We know that there's a lot of good research around that the best way to mitigate and close those gaps is high impact tutoring. But they recognized that the barrier to high impact tutoring was operationalizing that for system leaders. So if a if a superintendent in the early days of COVID it's like oh how am I going to make sure that the students that need it the most don't fall further behind. They say okay high impact tutoring yeah well and good there's great research that seems expensive or that seems complicated for my district I don't know what to do. So they started the National Student Support Accelerator in mid 2020 and over the last four years they have seen great success and they built basically this recipe of research tools and engagement with in their case they approached they they had those same priorities research tools and engagement but where for the Gen AI hub we are research we are research first because that the field of research for Gen AI is so early for high impact tutoring there was a big body of good research around high impact tutoring so they went to tools first. And how do we actually operationalize this and help this make sense actionable and applicable to a school leader in in Tennessee, or a district leader in Washington DC, across a variety of different contexts across our country. And in the last four years, they saw massive influence around district policies, state policies, and even national recommendations around high impact tutoring. Is a response to the pandemic, initially, but to persist to be a highly effective tool to serve the students that need it most. So they can catch or close, close gaps that they have. And so through that, so I'd, so two years ago, Susanna came back to Stanford. And in coming back to Stanford, she came back to start the scale initiative, which is about systems change for advancing learning equity. And the NSSA team came to Stanford with her to be a component of the scale team. And to give the full origin story of the Gen AI hub, the NSSA was doing a lot of research around high impact tutoring too. And as we know, many of the very first Gen AI applications to education were around tutoring because of the obvious kind of V1 applications with the technology as it exists. And through that research, they recognized to state the, in many ways, the obvious of this Gen AI thing is gonna be a thing that we need to dedicate a whole focused body and hub to, which then initiated the Gen AI. But we're a sister, kind of department to the NSSA and being able to learn from them in helping create systems change that is contextualized within different districts, within different states, but also practical and actionable for their context. I think will be really helpful for us. And the last thing I'll say on that is, we're recording this right now in one or two days after the, by executive order, the Department of Education has been dissolved. And what that's going to create is a lot more onus on the states and opportunity at the states, a lot more leadership at the states. And as you said, which is so accurate to the US education system, we have always been in a very decentralized system that maybe the one glimmer of opportunity here is that there are gonna be 50 different laboratories of innovation for what Gen AI can and should look like. And our ability to be able to research what's happening in each one of those laboratories and understand the really promising bright spots and to be able to take the research on what's happening in this state or in this district that is outstanding, that's moving the needle on efficiency or student outcomes or reimagining schooling and then be able to convert that into tools to bring it to more system leaders across the country. That's our hope to have that system level change still around the country. - Yeah, I would especially double click on that last point, which is, I find just in my experience interacting with a lot of systems leaders and even in my own decision-making as someone who makes decisions that impact many teams, oftentimes one of the biggest motivators in those decision-making processes is gonna be what you're held accountable to, like your goals, those policies and those things and obviously schools, districts, they're looking at their report cards, they're looking at the highest needs, they're looking at what findings come from and that's gonna be a really large motivator of a lot of decisions that were made. But to your point, I think we're entering this season where the answer to those questions feels murkier than probably it's ever been before. And even at a state level, I mean, there's a lot of states as an example here in Massachusetts, I mean, over the last year there was a decision made to repeal back the MCAS, which is like a pretty well-known state standardized exam as a graduation requirement. And I think the impact of that is there's a lot of folks now corralling across the state asking themselves, well, if it's not the MCAS, then how will we define success and what is gonna be the standard for allowing our young people to graduate and the requirement we wanna hold them to? And I think to your point, in those moments where states and entities are looking for answers, having organizations such as the organization you're leading and other places and having that research, having those tools, I think it really helped to support their decision making when they're actually really looking for answers. And there isn't like set of a scribe frameworks. And so those conditions might make this like a very interesting time to be deploying these types of work streams as well. So maybe to dive further into this, you had talked about how as a part of launching this hub, a lot of that was informed by the sister hub and the work that they did in their 30 chains. But what I also heard you say is the other part of this was also about doing a landscape analysis and looking across the field and asking ourselves, what are some of the organizations across the world, the country doing, what are they not doing? And I'm curious, what are you noticing high level about where we seem to be building momentum and where we're not building any type of momentum? I love that you went there, Daniel. So this gets a little bit to our forming research agenda around Genai and K12 and how we're thinking about our priorities in different research projects we're taking on. So right now, we're largely thinking about our research happening in three different categories. There is, as you said, the landscape mapping. And this is understanding what's happening, what people are using, how take up is changing over time. Right now, we can all probably think back to LinkedIn posts we've seen in the last two weeks of such and such entity surveyed 1,000 teachers and 10% are using AI weekly. And then I swear like three days later, you see a different survey from a different, very reputable organization that surveyed 2,000 teachers and found out that 70% are using AI weekly. We see numbers all over the place. And generally, what we have right now to understand the landscape is a lot of survey data, which is a great place to start. And the first part of our research agenda is in mapping that landscape, we want to evolve where we are right now, which is a lot of surveys, to start looking at real user data and working with districts and ed texts on how much time people are spending on their platforms. Where are they clicking? What tools are they using? What is causing them to increase use or decrease use? How is their use changing over time? And this one will give us a good baseline understanding of what use looks like right now. See some contributors which are leading to is take up increasing or not. What is causing people to use it more or not? And they really importantly, start to identify areas of, oh, do in this maybe we learned that teachers are largely using it for lesson planning. Or maybe not, maybe we're learning that they're largely using it for assessment or for exit tickets. In learning where behavior is gravitating to, then that can inform the next stage and next category of our research agenda, which is around efficacy. So we can start to zero down on zero in a, okay, teachers or students are spending more time in these areas. So let's investigate efficacy around assessment or efficacy around lesson planning. So efficacy is next kind of research category. Is it moving the needle on the things we care about, you know, largely being learning outcomes? And then last category of research agenda as it's starting to take shape is what we're calling it enabling conditions. So what we want to understand and start to research around this are what are the conditions that need to be in place that help for highly effective AI take up in a district with a school within a state, you know, what are the policies that are in place, what are the PD practices for teachers, et cetera, et cetera. And so to review, we've got landscape mapping, we've got efficacy and we've got enabling conditions. So right now, as I mentioned, we have two research projects that are in flight and then we have a lot in the pipeline. What's in flight right now is in the category of landscape mapping. And I would say this is not definitive. This is just really early indicators. So nobody should hang their hat on this. And I'm sharing this only to get people curious to want to look for more research to come of this. But what we're starting to see some indicators on in comparing AI take up across contexts is that when we look across difference of, say, title one schools or more resource schools, generally speaking schools are across more well resource schools and Title I schools are trying AI AI out similar rates and they're sticking with it at similar rates so that we're not seeing a delta between higher need schools more well resource schools also across race ethnicity so districts with higher percentage of students of color versus districts that are more predominantly white students so also we are not seeing significant difference in schools trying out AI tools or like persisting with those AI tools the one area that we are seeing a more significant delta on is urban school districts and suburban school districts are both trying out AI and sticking with AI at similar rates but we're seeing that rural schools are the one area that are both trying and persisting with AI at much lower rates and so this is like very early insights as far as looking at take up across some different contexts but it's something that we want to investigate a lot more and it's and is an important early question to understand yeah I appreciate you walking me through especially that that example you know to contextualize that further because I do think there's gonna be a lot of maybe predictions or feelings about how AI is gonna be deployed I mean even to your point of even the LinkedIn timeline those surveys that you're mentioning and those I see that all the time as well so maybe let me also ask you this it seems like some of that initial AI hype is really subsided in the last I would say maybe about six this month especially for schools I think even before I think there was some kind of consensus that AI literacy at the very least was important that that actually is something we should prioritize in schools that we should make space for now I'm even seeing a little bit of pushback there of like is that actually really important and I'm just wondering are you all noticing the same thing and is that coming up in your convenings or in the research happening there I'm just wondering if that's something that you all are noticing as well I would fully agree with you that the blind height has passed which I think is a great thing because it can help us move to a more critical practical what does this mean stage so I'll introduce one framework that I think is really helpful just to analyze a couple different approaches schools are taking so Cousin which is led by Keith Krueger which is an association that supports district technology leaders across the country they do an annual survey about what is top of mind for technology leaders at districts and they are they are classifying schools kind of into one of three categories they're either the bandit outright category the let every flower bloom category of like well just let all the teachers experiment and do a bunch of different things or the ducking cover of like oh this feels really complicated it might pass over us let's just like put our heads down and hope it gets easier in the future and but I would say one we do know that AI is not going anywhere what its role will be and is in education is we are in early days of that what that looks like what best practices are there but we know in the world in the workforce world in generally college and career AI is going to play a role so simply from a fact of preparing young people for a future AI plays a role in there somewhere even if it's in the realm only of AI literacy and I do feel like I'm I talk to state and district technology leaders a lot and I do sense from all of them a pressure to do something and I I empathize with that and that's why at the beginning of the episode one of the things I led with is this permission to approach it with a beginner's mind of hey you are a leader you might be in a leadership role at your school at your district in your department and as a leader you might feel a pressure to have all the answers but I have a secret for all of you that is quite liberating I think nobody knows exactly where it's going even the heads of the largest LLAMs or tech companies don't know where this is going long-term so it's this great opportunity to truly approach it with a beginner's mind of getting curious and playing with the tools and asking questions about how could this be helpful how might this fit with what we're grappling with right now what our needs are what we're struggling with and I think that can be a more productive way to navigate the pressure of to do something because that is everywhere of like if AI is a core part of all education conferences now so many webinars and teacher PD sessions it's it's everywhere so that feeling do something exists but though like so what what do I do that's the hard part and that's where where we're in the space right now of what is AI mean for education where is the impact and what does this look like in practice I'll I will say one more thing though I recently learned this new term called technology overhang that Ethan Mollick shared Ethan Mollick who Wharton professor who's written a lot on AI generally because he's a college professor he also thinks about it specifically in the realm of education but but the general concept of the tech overhang is the text ability out pacing it's kind of day-to-day productivity and applications for what we do right now and the his his his theory or his his his proposal is that even if AI stopped right now and didn't develop any further there would be years of work for education to think about ways that it can be useful in education and cannot be because it's moving so fast and so to me that's just more reason to be leaning in asking her questions and engaging but not having the expectation of right now having all the answers yeah that resonates and I think to your point it's it's also just such a healthy and great orientation to even bring to conversations in in your school building right being able to say hey this is what we know this is what we don't know and being able to just model you know collaborative leadership and what that looks like and saying let's go on this journey together because I think to your point if we especially with with this point and I and I think this was especially felt after like Chad GPT first became live in 2022 in that first 18 months where it felt like policies were just you know have being written in pencil and adjusted on a daily basis because it was literally happening so I think folks have a little bit more clear opinions now so like what should or should not be included in their policies and that feels to have the dust appears to have settled a little bit there but to your point like what I'm you know wondering if folks are thinking about is even just in hearing you say like I this maybe this responsibility for a lot of folks that are and and and tech type roles to be able to say well we know there's a moral imperative for us to like lean in until like understand until I create guidance here but what I'm also hoping opening is equally true is that folks in more programmatic or education or educator facing roles also like leaning into feeling this responsibility because what I'm noticing in just conversations I've been a part of is this trend where I think AI is being very ed-techified and I think folks now when you talk about do you know what AI is they think oh yeah I've like played with magic school or I played with Chad GPT so yeah that's AI AI is Chad GPT and it's like no it's not just that it is that but it's not just that right and and thinking about all of the ways that AI is going to impact society and lessons learned from other ways where we've had general purpose technologies impacts are worlds positively or negatively and not having thought about that and I think the one that the two that really come top of mind for me where it feels like even if you don't believe that your school should be adopting any AI use cases you should still probably talk about these two things which is how do you handle like deep fakes in your school and like what's what's the education around that how do we how do we also talk about boundaries with like young people like leveraging that and what that is and how it's you know with just how to grapple with that and then I think we're very related is just relationships with technology right technology now and and I think we started to learn a whole lot from social media about the impact the technology can have on our young people and their mental health and now we're starting to see early indicators around AI and how a lot of these chatbots especially ones that are designed to be even you know more persuasive more compelling you know we look at tools like character AI which are the most pop some of the most popular AI tools that exist in the world right now and it is disproportionately young people that are engaging with tools like character right and they're engaging because they have chatbots that they can talk to a different person is that they're excited to connect with and I think we're entering this like potentially uncharted territory or at least uncharted to me territory where we are now creating like human like create relationships with technology and I don't know that we have appropriate boundaries or just guardrails for like what that should look like and what what a healthy versus non healthy relationship would look like and so I would really push schools even think about what some early conversations around even those topics could look like you know and how we can start to get ahead of that. Oh I think that is really important that you brought up young people's use of technology how social media plays in and kind of the role that that education leaders and educators need to be thinking about AI literacy and kind of learn from our past 20 years we sometimes use this comparison of two powerful technologies launched in the last 20 years and two very different ways that they played out and one was social media and this you know started in the mid-OOs give or take and then in many ways unleashed with the advent the smartphone and the ubiquity of the smartphone getting in kids pockets and increasing use and it radically changed behavior and there was little to no research done while it was unfolding and there was very little regulatory oversight beyond say section 230 that created boundaries for what this what responsible use look like the role of the provider etc etc and where we've landed with that and and I'm not going to debate like there are many contributors to these things but what we do know is young people's mental health feelings of isolation thoughts of self harm are at historically high and many indicators point to social media being a contributor so that's that's one play out there's a different play out with also technology that was that came out very quickly during the pandemic covid virus came out and greet focus operation warp speed focused on development of mRNA vaccines and these mRNA vaccines were grounded in really solid fast research on efficacy one and it was bounded within government oversight with clear boundaries to protect human health and safety and how did that play out they were developed in months and all data tells us millions and millions of lives were saved because of that so with those two contrasting pieces and how specifically research and the regulatory environment was handled very differently in both in in one case social media none of either in the other case appropriate levels of both and they ended in very different places and so we have this great opportunity with generative AI on one we need to early and often be researching what's working and what's not so we can direct our attention focus energy on what's working do more of that and pull back on where we're not seeing efficacy and where we might even be seen or and then also looking at the regulatory environments that help the innovation to happen while also protecting kids on data privacy deep fake harm etc etc and in our current state you know we're probably going to see much of that innovation around regulation becoming at the state level yeah well I mean well said I think you're absolutely right that if you know if I were to wait my magic wand and say what could have been different with the deployment of social media and how might it we have best position ourselves for systems to be able to like understand the impact of technology I think the the dichotomy you were kind of sharing there like very much sums it up well right which is a big a big enabling condition there is about research it's about being thoughtful it's about understanding what the impact is and how we might be pivoting for making smarter decisions as we continue to navigate the waters there so I mean that being said like I feel like we've been we've been hinted at it for a little while they're bearing the lead so I mean let's get into it I know one of the big things you all are are prepared to share that you're you have done a lot of work on is this research study repository what is that what is it supposed to do how is it set up walk us through it so the thank you for that setup Daniel so research repository we launched at the end of january and it is on our website sure we can link to this in the show notes but it is it's goal is to collect all the research that exists around gen ai and k12 and put it in one place where a user a practitioner can go and filter for who their user is what the application is what the developmental age is what the why is and then pull up all the research that exists connected to a topic so as a principle you might say we've been struggling with middle school math outcomes so what does the research say around gen ai tools that teachers can use as the user that is focused on middle school age and that is around numeracy around math outcomes and then it'll pull up all the papers that exist on this so right now the body of research as I alluded to earlier is quite small when we launched at the end of january we had 181 papers there and only 26 of those are our CTs so randomized controlled trials where we can have like clear takeaways that we feel more confident on and so it's early days we plan to update this monthly actually in the next week we plan to roll out a major update of the repository and significantly expand the pool of papers that are in there but like I was mentioning earlier in our conversation what we hope to do by providing this tool is give practitioners a resource when they feel like they're getting hit by a number of sales calls from edtech providers on hey I've got a great product that can solve this problem for you then you can say huh what does the research say or when you're debating something as a school leadership team and you're saying okay does ai play a role in this you can ask what does the research say and you know where you can go to get some answers to those questions you can go to the repository you can filter for the need and you can pull up and you might find one of two things one you might find you might find a bunch of great papers that that have asked the exact question you're curious about that can help inform you make some research back fine decisions on what tool you want to do that's great you also to to be honest you might come up with zero papers in the filter that you and based on all the filters you have there's four literally over four billion different combinations of ways to splice it but you might come up and be like oh this is a total blind spot right now that is also pertinent valuable knowledge to know hey um there is no research has been done into this area and that will help you approach your decision making on that issues that risk mitigation that buying decision more appropriately so that's where it stands right now and we're really excited for people to use it to get out in the world and we have some plans to improve it technology enabled in a bunch of different ways in coming months so also this is a little plug to for people to get their hands on it and then also share feedback with us on tell us how you're using it tell us what would be more valuable if we added it as a feature we see many ways to leverage AI to start to make sense of and synthesize the research that does exist because very quickly the volume of papers in there is going to become more than an individual practitioner can make sense of so you might filter for something come up with 25 papers and as a busy superintendent like i'm not going to read 25 50 page academic papers so can AI help us synthesize that to to glean the key insights on that topic across those 25 papers i'll tell you two use cases right now that we're hearing from users on how they're finding great efficacy with the repository so one right away heard from many school leaders on oh this is great i'm going there with questions or something i'm curious about i filter for what i need i identify eight papers i download those eight papers And then I upload them into a notebook LM I created notebook LM is a Google product for your listeners that might not be familiar It's a it's a great tool to to give a quote unquote notebook a discrete set of knowledge. So you give it 8 PDFs these eight papers and then The notebook allows you to use AI to engage with the content in those 8 PDFs So the user then uploads a PDFs to their notebook LM and they can chat with the papers and ask them questions They can ask for the high-level summary of those 8 papers. They can take away for their specific context They can make a podcast and then listen to it debate between two AI voices about the content They can engage within a bunch of different ways. So that's one real effective use case We've been hearing and then another with the launch of deep research by boast Google and open AI Specifically, I've heard several use cases of users using the deep deep research on open AI Through open AI's chat GPT platform and all they do is they point their Their chat towards the generative AI for education hub research repository at Stanford University They tell it to go look there and tell them the question. They're wondering about I am a high school principal looking for what AI supports there might be for developing resilience in my students or Developing more social connections to build school cohesion or to improve literacy outcomes You know, whatever thing that you're curious about to optimize my school and Through chat GPT research tell them the context the question that they're trying at that they're wondering about Tell it explicitly going to the generative AI research repository and Then it'll take you know five plus minutes to go sink and it comes back and the reports I've gotten from users They've often shared their chats with me or Quite good and what I would say is what gets me most excited is this as a tool to bringing us back to an earlier theme in our conversation Be that bridge from Historically denser, longer, sometimes maybe more theoretical Research papers and bridge build that bridge to the practitioner who are like give me the key takeaways or the actions I can do today to make progress towards this goal leveraging AI. What should I be worried about where should I be radar up and where can I lead it to? And just so that I'm understanding that last example because that that would be nice to me I played around with the deep research Functionalities a little bit, but maybe not as much as you so are you saying that some users are essentially saying in their problems like hey Chad you see like I specifically want you to find sources from the Stanford repository in the inquiry of this question And then it's like in and then in addition to answering the question. It's specifically going to the repository It's as simple as that. I played around with an affair back to of I'm you know quote me on this some version of this you know go to the generative AI for education Hub researcher repository at the Stanford accelerator for learning I want you to pull research from that resource and help me investigate X question here's some more details about the context I'm working within Send ready go and then the results they get back sources cited links back to the papers in our repository So you there's a level of transparency and reliability through that, which is I think pretty powerful tool I can't help but just like reflect in this moment that even some of the early conclusions We thought we were drawing about artificial intelligence. We're already already being challenged It's like we kept saying a bunch of the narrative out there is AI can really help with these like low-to-mid cognitive level skills But it can't help as much with you know fact checking information And I think this is an example of how that may not be very true for very long right with deep research Functionalities and being able to direct it to really really good sources of information and doing some of that So I can't help but just laugh it to your earlier point of even if even for folks who try to keep a pulse On things and do you can't help but feel like you're at square one a couple months later with just the new developments and innovations that are happening It is so true. It is so true And that's why here. I'll get meta for a second, but you know, I've been in one theme I hope people are taking away as approached this with a beginner's mind You don't need to have all the answers But just get curious and play around and so me in Eleading the generative AI for education how we just built this repository and I'm telling all your listeners Hey, go play around with the repository and tell us what you're learning Tell us how you're using it and tell us how it could be better Because we don't have all the answers on how that could be used But we want to learn from you all and what you're finding out to then like iterate and improve opponent So try and do a practice when I preach there Yeah, it sounds great No, and I and maybe just so that we can understand a little bit of their requirements So when for the papers that ultimately make their way on the repository like what what type of screening is involved to understand the rigor of those sources that are Added to the repository versus what is not added? Yeah, that is really important question So curation is particularly important now Because we could very quickly Fill that with an insane volume of research that very quickly will make it less usable right because we don't have a AI tool built into it right now to like synthesize and take away and link key learnings So we on our team we have a talented research data scientist Benjamin Liva Who works for the hop who has been the the key charge in in building the repository and we are in where actively improving upon our sourcing Selection and curation the repository so in early days when the pool was pretty small It was very as we can say handmade in that we basically did a scrape of all the journals On anything related to AI and education and then hand selected them Like went through line-by-line on abstracts and said this was relevant. This is relevant. This is it. This is relevant etc And really for the first cycle that's possible we are already starting to see a hockey stick Increase in the volume of genai research coming out and the vast majority of the research in the repository right now Was published in 2023 and 2024 and so it's it's very recent and 2025 I'm sure will be a lot more research than 2023 and 2024 combined So we are building a system that is both more reliable and scalable as the research Increases and I will bore people with the details, but in short we have a process that we can run monthly that goes out and looks at all the journals out there Where there's published academic research or this is important Sources like ed working papers that have pre peer reviewed but again We're trying to get speed to insights and so we want to get stuff as soon as it's out So it pulls from all of those and then just dumps a huge body of all these papers are generally about AI and education And then we have a series of prompts that analyze all those papers Grade all those papers based on to our target audience is For the genai and research hub the questions we anticipate that they're curious about and all the filters That we have on the repository to tag them accordingly And then we have a protocol for Then scoring them and then above a certain score Gets into the repository automatically and then around the margins doing hand curation Of this is in this is not and then improving the prompt and the scoring process as we go So that will allow us to again, this is changing fast that it's only accelerating So we need to build this in a way that we can deliver on the speed that this field of study is changing So leveraging AI to do so is important Yeah No, and I appreciate you walking us through that because I think especially in this moment where we were Hanting that earlier. I think a lot of vital sources of data potentially at the federal level may be phasing out Or may not be was up to date as they've been before and so I think there's a moment here where rigorous sources of Analysis and data and research are going to be really important something. I think system leaders really values Chris, we've I feel like we've hit so many things today. What haven't we talked about that you feel is important to to share I'll just need one more piece as far as maybe this is this is scaffolding for your listeners on How to engage with the repository in a really productive way and then I'll just give a couple shout outs to two papers in there To keep this really grounded in the research. So Right now our first step to help you So there's a few filters that you can go into there right now. One being who is the user so you can filter for student or teacher or school leader or family caregiver. And if you filter just for student, then it'll pull the papers that fit that. But at the top, it'll also have here the key insights that synthesize. Like what we know about Gen AI and students right now, or you could filter for teachers. So that's important. You could also under the filter of study design. You could filter for quasi experimental or literature review or RCT randomized control trial. And if you just filter say for RCT, then it'll pull up all the papers. In this case, 26 of 181 right now or RCTs. And this is really powerful. It'll at the top. It'll give you here the key insights on what we know from RCTs are on Gen AI and education right now. And this is a great just lay of the land. This is what is known. This is what research tells me right now on AI and kitchen. It's a great starting spot. And just starting to go through a couple of those key filters in 30 minutes. You could feel like, oh, I get a cent, like an overall sense on what is the research say in these several categories. So that's a great place to start. And I'll get on now like drilling down to a couple important RCTs. And both of these are coming out of some of our gifted researchers and professors at Stanford. But I also think they're super important as far as thinking about AI and its role in education. And specifically that I think a lot of promise around AI right now is specifically around the teacher. When we bring AI to students, that can feel harder, higher risk, federal law, things, FERPA get involved. So it just very quickly gets a lot more complicated. But AI to teachers creates real opportunity. It creates a way to empower and give agency to teachers to maybe give them some freedom in saving them time for them to get creative. And specifically two areas that show promise are around teacher augmentation. So AI that doesn't face this student, but AI that supports the teacher to work better human to human with students or or just teacher learner. So one example of the teacher augmentation role is this important paper. Some of your listeners probably have heard of it already called tutor co pilot Rose Wang and Susanna Loeb were two of the key researchers on this. And at a super high level, I'll frame it and then give two key takeaways. So this studied a tutoring, a digital tutoring platform where tutors worked with fourth grade learners. And it's an RCT. So we had some tutors that worked with fourth grade learner learners as standard. And then we had some tutors that work with fourth grade learners, but the tutor had an AI that was effectively listening in. And after the student would say something, then the AI would provide three suggestions for how the tutor could respond that were grounded in good pedagogy and good tutoring practice. The tutor could choose to ignore that and say whatever response they chose or they could choose to take one of those suggestions. At risk of oversimplifying two key takeaways, the treatment, so the group that had the AI, the tutors that had the AI support, there were noticeable increases in now outcomes for those fourth grade students that were tutored by those tutors. One, and two, I think the more interesting one was the tutoring provider had previously benchmarked their tutors from high performing meeting expectations and low performing. This AI support really didn't have much impact on the high performing tutors, they did their thing, they continue to be effective, but this AI support brought the low performing tutors to meeting expectations. Real implications for fast tracking talent into teaching to bringing people up to speed to actually having real impact on kids faster, sooner in a more effective way, so that's one category around teacher augmentation. Last paper I'll mention because it's also I think a real promising area for AI and education and the category is teacher PD teacher professional development. So Dorah Dempsky in 2023, so this is now year and a half, almost two years old, but I think it's still really relevant. And so Dorah and team did a mixed, mixed methods RCT in classrooms that used teach FX. And so teach FX for those not familiar is an AI tool that listens to audio in a classroom, listens to what the teacher is saying specifically. And then at the end of a session will give a report out to the teacher on here was your some simple metrics like here was your talk time as compared to your volume of student talk time. And then also like what type of talking were you doing were you lecturing being didactic were you asking questions more getting students to think and also at risk of oversimplifying but to cutting to key takeaways here that I think is really interesting is in this RCT those teachers that use the teach FX tool to get feedback. The AI generated feedback led to a 20% increase teachers use of deep questioning techniques so more advanced teaching strategies to help develop more deep thinking deep learning on the student side. So I share those is just to make it tangible to make a practical to RCTs that I think are meaningful in the way of teacher augmentation or teacher PD that show promise for future research and impact in the future in education. Yeah, I mean you got my will turn in so much with those soon I think it affirms a lot of what I've been learning in different spaces to I think well my first maybe cheeky reaction is you mean to tell me that we're still going to need teachers in the classroom in the future. That is the headline yes I will I will I will put all my chips into the center of the table of teachers role is only going to be more important human teachers role will only be more important that is what I that's the world I want to see in the future. Could not agree more I will also throw in a couple of initial anecdotes that were coming up for me I think especially in that initial study you had referenced around just the teacher augmentation and the performance boost for the the lower the new coming performers and getting them to me expect it actually very aligned. So person you mentioned earlier I know Ethan mallick he actually did a study with harbor looking at consultants with Boston Boston consulting group they essentially found the same thing that using AI and allowing consultancy is AI it didn't really help the great consultants be any greater but it really helps the consultants who were like below the curve like get up to like during the curve with the access to those tools so it's interesting to hear just even in different different context how there might be some similar findings there. Just my mind is was immediately turning so much to just think about implications for teacher certification programs alternative certification programs knowing that currently with the way the system is set up in the United States there's just so much turnover and the teaching profession so you have a lot of teachers that fall into that like zero to five years of teaching experience and whatever you can do to get them to. To really confident the classroom using those like more really really critical check for understanding type strategies as quickly as possible so they feel confident and they're really like impacted the performance that feels like such a really vital resource and then very connected to the second study that you had mentioned. I think the audio is very interesting and I've also been exposed to conversations where folks are thinking about the video components too which is like you know especially as we we we get more AI tools that have access to text the video what the implications of being able to record just to teach your teacher teach for 60 minutes and then it just offers its perspective and feedback based on some of the things not just but now not just considering the context of your words in your your voice but also just your presence in those things you might be signaling just in how you're showing up in your voice. And how valuable some of that might be in a school based context where teachers may only have access to feedback from saying assistant principal or an instructional coach potentially on a weekly basis at best and now getting that on the daily cadence if they they aspire to have something like that. I think that is so true and I love that you brought it at the end there to in future worlds and whether this is student facing or teacher facing. I don't see this as that humans are removed from the teacher feedback coaching process entirely, but let's get practical today. Like if a teacher is getting observation and feedback from another human being once a week, that's really special. Like that is like 99th percentile in schools that are able to do that. There are many schools where maybe that happens once a year. And with through technology, could teachers get way more reps on that feedback with much greater frequency through technology once a day, once a week. And then that time of that previously one peer could come in and give, observe them once a year, that could be reallocated to that human being helping them make sense of the feedback they've gotten and to coach them through how they're gonna take action on what they've learned to build future strategies to improve their practice, et cetera, et cetera. So it's a great example of like the augmentation and support rather than the replacement. - And I think another thought that had been coming up for me is you were sharing an actually very serendipitously connected to like how we started this conversation just around self-actualization and these like more meaningful experiences for young people. I once read a blog post by the person who's like leading AI at Sacramento State, Sasha, Sudorkin and I had him on for a conversation last year. And one thing he had wrote this blog about how one of the potentials for AI as well for the role of the teacher is it's not that their role has diminished at all in this context and I think that's very much what we're saying as well, but it evolves, it transforms, right? Where they're leaning in versus where they can lean out. Now it becomes a transform that I think where he really hypothesized is that it really can unlock for the teachers like open up so much more of that relational work, right? Which is like why I think so many teachers get into teaching and education is to like mentor young people to like be there to share life advice to help them on their journey. And they just don't have time in the day to do those things. And so I see in addition to like some of the pieces you were naming in these studies of helping you to become sharper at your craft, I think the other layer of this is outsourcing some of the pieces that they don't really, they shouldn't be doing it at all. And just the mastery in being able to do this more officially is going to open up so much more of that like relationship building that life advice. Those experiences just connecting like as a wise adult helping our next generation that I think so many teachers like hunger forum will keep them energized and connected to the work, which doesn't happen at the rate that I think we aspire for it to happen. And so that was also just I think unlocked for me even further in some of these studies. Just affirming that no, that's where we should be going. That's where we should be putting our chips. And let's let's and hopefully the research continues to support that or affirm that that's the way we should be going as well. Yeah, couldn't agree more. That's excellent. Chris, I feel like we hit so many topics today. Any closing thoughts before we enter a time together here? I would say I would leave people also in the spirit of making this very tangible. I would leave people with one very simple three-point framework of just what we're observing and talking to a lot of school and district leaders across the country that is starting to show up but is pretty universal, too strong of a word, but a pattern. In that often schools are focusing in one to three areas as they're starting to dip their toe into AI. So I offer this as one way that those school and district and state leaders can start getting applied. So one, AI literacy. And specifically AI literacy for all. So thinking about all your stakeholder groups is not only AI literacy for students, only AI literacy for teachers, but like it's school leadership, teachers, students, the community and family caregivers, but AI literacy for all. So starting to think about that. Two, and this is classic like early adoption change model but lots of schools and districts are starting to see success. Here is their designating an early adopter group. It's a group of early adopter teachers, maybe some instructional leaders or district school leaders. And I think where it gets really interesting is also involving students because students can be at the real leading edge here, but it's an early adopter group. It's an experimentation group and give them a little more freedom to explore and see what's possible. And it can really help school and district leaders, too, they can tap into that group and see where the future might be going. They don't have to be doing all the experimentation self, but they can use this as an early indication group. So that's two, the experimenting group. And identifying that and they come together say once a month. And three, getting really practical and schools are playing around with operational systems back in the office. Ways to apply it because it can feel very tangible and also lower stakes because we only have adults engaging with the tool. So we're mitigating risk there. The a lot of the business applications, business systems, tools can be a bit more defined. And all schools and districts have business systems that they have to take care of. And so that can be a way that can feel more accessible to some schools and districts to start experimenting with of what if does AI play a role here and give them more confidence to roll it out in the things we truly care about. Your listeners to visit the repository and share feedback with us at the Generative AI for education health. Thanks, Chris. Very much appreciate your time today. Thank you. Why we're all here, which is around teaching at learning. So those are just three ways of patterns and what we're seeing in some early adoption for schools. And Daniel, I've really enjoyed our conversation and encourage your listeners to visit the repository and share feedback with us at the Generative AI for education health. Thanks, Chris. Thank you.