Episode 7: On Defining Success

[MUSIC] You're listening to Community Crafted Education, where we explore alternatives to conventional K-12 education and their potential impacts on society and culture. I'm Eric Wairn, Director of the National Hybrid Schools Project. On this episode, we're joined by Matt Lee from Kennesaw State University. We'll talk about how families choose schools and how we might define success in community crafted education models. [MUSIC] Well, hello. Today we're joined by Matt Lee from Kennesaw State University's Education Economics Center. Matt, thanks for joining us today. >> Hey, Eric. Good to be with you. >> Yeah, so typically on this podcast, the point is to talk with people who are kind of outside of school choice world about the effects of various aspects of interesting school models on their particular areas of work. Now, you're very much inside school choice world, but I think you're doing some really interesting stuff that is going to have an impact on how we think about designing schools and choosing schools and measuring schools and things like that. So, could you just tell us in general about your background and your work at Kennesaw State? >> Yeah, I'd love to. So, I've been at Kennesaw State now for a couple of years working with the Education Economics Center and with you and John and the National Hybrid School Project. Doing a lot of policy-related research on really focusing on the area of school choice. And hybrid schools tend to be schools that people choose, so it really fits in with that. Before I came to Kennesaw State University, I was applying my research skills as the Director of Research for the Association of Christian Schools International. And before that, I was doing my PhD with the Department of Education before at the University of Arkansas. So, that program really focused on applying economic research methods to the area of education policy. And I think one of the kind of unique things about the Arkansas program was that it had a lot of faculty that had done a lot of their significant career research in the area of school choice. So, I knew that that was an area that I wanted to study. And so, I got a really rigorous education from that program and now I get to work with some awesome folks like you and Ben, John and some other folks at Kennesaw putting those skills to work. Yeah, no, great work. So, we're very glad you're here, Matt. So, I want to talk about a specific method that you work on. But first, maybe we should talk about just generally, how do parents choose schools? Like, what do they consider when they're making these choices? Yeah, that's a great question. So, I guess it depends on the kind of context that the parents find themselves in. I think a lot of the times, the parents just kind of end up in their residential, these kind of district school. But even to end up in that catchment area, they had to make some choices about where to live and things like that. I think prior research is pretty clear and pretty consistent about a few different themes. One of those being that parents invariably consider some form of academic quality as an important metric for how they choose schools. There was also a study done by Hastings and Weinstein that was published in Harvard University's quarterly journal of economics that showed that parents are sensitive to the way in which information is presented. And so, if there's information presented about which schools have higher test score levels or which schools really contribute more to students' growth in their learning, that parents kind of adjusted their decisions about which school that they would choose based on those things. So, parents invariably emphasize some form of academic quality in their, in the way in which they choose their schools. But interestingly, it's not always the most important factor that they consider. So, again, it depends on the context in which parents are choosing. And you also have this kind of Maslow's hierarchy of needs going on too, where, for example, if they feel like their students are not, if their children are not in a safe environment, then safety is going to be the number one thing that they emphasize even over academic quality. But beyond that, you have a lot of parents that choose on some sort of moral or value formation. Again, I was working in a private Christian school context. And those parents really emphasize the importance of spiritual formation, hand in hand with academic quality when they show schools. You have other parents making considerations like distance that they would have to travel to the school, or if they're in a private school setting about how much tuition they might have to pay. And also the kinds of programs or extracurricular opportunities that the school offers. And so, there's a little bit of cost-benefit analysis that a parent goes through and thinking about the things that they desire from the school, including academics, maybe value formation or extracurricular opportunities. But then that preference is constrained by the choices that are available to them as well as the kind of cost to them, whether tuition dollars or time in terms of travel that it might take for them to choose that particular school. Yeah, so parents are looking at all kinds of things, right? So academics are important. Physical safety is important. Other things, tuition might be a constraint that kind of makes the choice for them in some cases. But they're thinking about a lot of things, right? So let's talk about the other side then. How do other groups like governments, state or local governments, but also groups like philanthropy or nonprofits or researchers like us? How do those groups typically define success in schools? What are the parents they should like? Let me answer that question, but tell me that joke that at least like the economist find funny. So maybe the average person won't find funny, but the guy is walking down a street at night and he comes across a guy who happens to be an economist who's on his knees under the street light, looking for his keys. And so the guy says, "Hey, what are you doing under this light?" I drop my keys way back over there, but I'm looking for my keys here. And the guy that happens upon him said, "Well, if you dropped your keys back there, why are you looking for the keys here?" And the guy says, "Well, this is where the light is shining." And so to answer your question about how do governments and researchers and other organizations, how do they value success? Well, oftentimes they value success by where the light is shining. And so a lot of the times the question comes down to data access and availability, and some of the easiest data that there is to collect is related to outcomes like student test scores, and their grades, their GPA, their college going, the circulation, the graduation rates, things like that. So these are things that are important, I would say. Parents certainly value them to some degree, but parents don't necessarily value them as ultimate ends. Sometimes they value them as proximal ends to some other more important outcome. But for, again, government organizations, researchers, other organizations, and nonprofits, this is where the light is shining. And so when it comes to defining success for these guys, oftentimes they look where the light is shining at the data that's easiest to collect. And so success becomes some sort of metric about the data that's easiest to collect. Again, data that's important, but that might not be ultimate for parents, data like test scores, graduation rates, and things like that. Yeah, perfect. So we know there's sort of, at least potentially some kind of disconnect between what parents want or say they want, and what these other groups care about or say they care about. So now I want to get into the real reason I wanted to talk to you today is about these conjoined analyses that you're kind of on the cutting edge of applying to education policy, right? So we can ask parents, you know, we've surveyed parents all the time about these things that you mentioned about what you care about in schools, and you might present them with a list of things like academics, test scores, distance, tuition, physical safety, and we could ask them which of these things are important to you. And in some cases, parents would check all of them, right? In most cases, they probably would. A little bit more sophisticated version would be to ask them to kind of rank these things, but that gives us some more information. But you are working on another kind of analysis that I think really can tease out what parents care about and how much they care about it. So could you talk a little bit about what you're doing in this area? Yeah, you made some really good points there in terms of there's a lot of descriptive work that's done about the kind of outcomes that parents value. And like you said, more sophisticated descriptive work looks at how they would rank different sort of outcomes. But the limitation with rank order descriptive analysis is that, you know, a parent can list five things, one, two, three, four, five, and we could see in order what their preference for those things were. But we don't really have a good sense of how much more they valued the number one thing versus the number number two thing all the way down to the number five thing. So as you say, this is this is kind of where Conjuret comes in. So Conjuret analysis has pretty widely been done in fields like marketing or political science, but like you say, it really hasn't been applied to the field of education or any education policy, all that much. Let me try to kind of abstractly explain how it works, and then I'll explain it with a couple of examples, and then we can talk more about how it works in this community crafted education space. So a conjoint analysis is a fully randomized stated preference as an experiment. So if you're filling out a survey, you would come across a set of choices that's presented to you. Now these two choices have a bunch of different characteristics, and all of those characteristics have been randomly assigned to the choices that are available to the survey responded. So based on the choices that they have available to them, the survey responded would choose A, B, or C, depending on how many options are presented to them. Now because those characteristics are all randomly assigned, a researcher can now analyze on the back end to say, hey, this group of people that we're looking at in the aggregate, when a particular characteristic showed up, it caused them to be more likely to choose that option. And so we can see with causal inference, how important a given characteristic was to that individual, or to the group as a whole. And so now we get a sense not only in terms of causal inference, but also in terms of how much weight a survey responded would place on each characteristic, right? So a characteristic that's more important and consistently leading to the respondent choosing that choice would have a much bigger, what we call, effect size, than something else that basically the survey responded is ignoring. So let me explain now in terms of how marketing researchers and political science researchers have used it. So in marketing, saying you're doing research for a big tech company that's trying to figure out what kind of products that they want to come to market next year. Well, they might, for example, present a survey responded with two different TVs. And the TVs all have different attributes that are randomly assigned to them. It might be something like size or screen resolution or different features that it offers. And then of course, it would also involve price. And so with conjured analysis, a market researcher can then figure out how important these characteristics are to the person who's going to be buying, and then how much they're willing to pay for each of those given characteristics. In political science, for example, it's often done with election research. So a survey responded might be presented with two different political candidates. And then they might have their views on different issues randomly assigned to them. And so a researcher then can look at that data and figure out, well, how much did different issues matter to the survey respondent? And we can quickly figure out, well, is it the economy, is it immigration, is it education, is it something else? What are the issues that people are really deciding who to vote for on? So for that second example, it would be something like candidate A is for immigration or for increased immigration for less education funding, for higher taxes, or something like that, just randomly assigned by the survey software. Right. The same thing for candidate B. And then by picking, you kind of-- and the other end can see which of the issues the people care about. That's exactly right. Yeah. And for-- in lay listeners, I think that randomization is really important for researchers, because it allows us to say, basically, that when an attribute showed up, it caused the survey responded to be more likely or less likely to choose a given candidate or school or program or whatever it might be that we're looking at. So for the community crafted education space, I think this research method is really important for at least three reasons. One is kind of a practical thing. And then two and three, I think, are just kind of like a philosophical alignment. So one, I think, as more education-freedom programs are passed in the states, we're seeing that most states are adopting these programs in a universal form. So students are universally eligible for these programs. And in the past, enrollment in these programs has been constrained by some sort of limit, right? They might have been means tested, where they might only be available for students that attended a D or F-rated school in the previous year, or something like that. And so by limiting the number of students that can participate, they could have this randomization lottery that researchers would use to study how effective the program was. Well, we're seeing that kind of taken off the table for researchers now with these universal programs. So we can no longer randomly assign students to receive, let's say, an education savings account or not. They're all eligible to receive it. And so that method for evaluating program effectiveness is kind of going away for researchers. So one is a practical consideration. Two and three, I think, are more philosophical alignment. Which is to say, when a researcher examines a program and says that it produces this kind of effect on, let's say, student test scores, implicit in that kind of analysis is saying, we can validate your choice and measure how good or bad your choice was by saying that your choice either increased test scores or decreased test scores or increased graduate you know, you get the points, I went in so forth. And if the program or if the schools aren't perfectly aligned with the outcome that's being measured, then even if they do a good job at improving that metric, it might not be fully captured by the method, which is only focusing on a particular outcome or a particular set of outcomes. And so there's this implicit, we get to validate your choice kind of thing going on with these kinds of evaluations. And third, I think, very similar. On second though, just to put some more meat on that, that's just an example would be, right? Like, we're going to test you on Egypt in fourth grade. So hopefully your school studying Egypt in fourth grade or your kids are going to score poorly on this. Yeah, that's right. You might not want to do Egypt in fourth grade. We might want to do it somewhere else. And it could be test scores in general. The school might say, we want our students to do well on test scores, but we're not going to overly prepare them for that test because we want them to have a holistic experience or we want to focus on some of their outcome that we as a school community value. Those things aren't going to get picked up by that sort of evaluation. So there's a misalignment in terms of the way in which we're evaluating schools and maybe what the schools or the parents or the students want to be evaluated on. And then I think the third thing related to that second point is it implies that there are choices that are good and choices that are bad and that perhaps some people should or should not have choices based on the kind of decision that they make. Now, I agree that people can make good or bad decisions. I make better or worse decisions on one day or another. So that's definitely true. But I think also the choice is inherently precious. The freedom for a parent to choose the kind of educational experience that they want for their kids is something that we should implicitly want to protect and to promote as a society because at this point, we've seen the full weight of empirical evidence from research that says parents are effective choosers. The schools that they choose tend to improve test scores, they tend to improve graduation rates. They tend to even produce longer term benefits for health, labor market outcomes. You know, it's like-- They make out changes. Yeah. And so at this point, I think we know that parents do a good job choosing schools for their kids. And so now the interesting thing to study is the way in which the parents make that choice. Because I think that choice is inherently a valuable thing. It's an inherently interesting thing. And so the conduit method now comes in. And now we can say, let's take apart that choice. We know it's valuable. Now we want to understand how parents go about this. So in the community crafted education space, right? Parents are choosing schools for all sorts of reasons. They're coming together in these voluntary communities and crafting schools for their kids that are designed to serve very unique and interesting purposes. A conduit research design can come in. It doesn't have to look at any particular student outcome. It's very flexible in the way in which it describes the way that parents make these kinds of choices. And now we can see and describe with a rigorous method and this causal design to say, this is how parents make choices for their kids in this context, where that context are another. Yeah, so this is great. So can you give us some examples-- you've done a few of these now. Can you give us some examples of things that you've seen when you run these kinds of analyses? Can be with community crafted schools or other kinds of schools. But what kinds of things are we learning by doing this kind of analysis? Yeah. I'll give you a few examples. So one of them was in my time with ACSI. I published it with Alison Johnson and Albert Chang. We looked at parents of students who were enrolled in private Christian schools. And we designed a conduit that was intended to understand how those parents made those kinds of decisions for their kids. And we found that their stated preference was, they told us spiritual formation is the number one thing. But then second or third on their ranked order list was always academic quality. The conjoint was consistent with that result because spiritual formation produced the largest effect sizes for their choice. But it was followed really closely by academic quality. So spiritual formation might have been like 30 to 32 percentage points increased likelihood of making a choice, but then academic quality was something like 28 to 30 percentage points. So really, really close, but in the order that they were telling us. And so this really showed, again, consistently with prior research that parents always consider some form of academic quality. Again, it might not be the most important reason that they're making that choice, but that student learning is always something that they want. In the hybrid school context with you and John, we've looked at hybrid school leaders. We've done a couple of interesting ones so far. Last year, we asked hybrid school leaders how they define success. And we presented them with a few different options in terms of what their graduates might look like. It might look like, again, some high-test score performance or going on to a higher education. It might have been that kind of religious component of having a spiritual life because what some 60 or 70% of hybrid schools say that they have some religious purpose. And again, I think consistent with what we've seen so far, academic quality was, again, valued by hybrid school leaders in terms of saying a student was successful or not. But then also, we saw a lot of differences in terms of the kind of school. Some schools really valued that religious life component. Some schools really valued the civic life component. Some schools, more than other schools, valued college, or elite college attendance and things like that. One project that we've got going on looks at hybrid school leaders and teacher characteristics to get a sense of how hybrid school leaders hire teachers for their schools. And some of the interesting things that we find there, hybrid school leaders really value hiring teachers or they desire hiring teachers that have some sort of a post-secondary degree, so a bachelor's degree, or a graduate degree of master's or a doctorate. But the interesting thing there was that hybrid school leaders really didn't pull out a stock in whether or not the teacher candidate had a degree in the field of education. So a teacher candidate who studied English, for example, and wanted to become an English teacher, maybe they put more stock in hiring that kind of teacher, than a teacher candidate that studied the science of education, for example. They also really valued teacher candidates that had experience. One of the things that I found interesting there was, they even valued teachers that had experience in conventional schools. So even though hybrid schools are this unique space that blend a conventional learning model for two or three days a week with a homeschooling model for two or three days a week, the hybrid school leaders actually thought that prior experience in a conventional school be a public or private, didn't matter, was actually a good preparation for them to be effective hybrid school teachers. So yeah, these are kind of some of the projects that we've got going on looking at the conjured method in some of these community crafted spaces. Yeah, so it's interesting. I want to talk about the teacher degree point for a minute. So this is an interesting finding, right? That these, on their own, without any requirement or law or anything like that, these hybrid school leaders are wanting people that have some kind of degree, but not necessarily an education degree, right? So if I'm a state regulator, I might want to know that, right? Or if I'm in a creditor, right? So this is something that we learn through the conjoint that might have a practical effect on certain kinds of requirements that these schools are facing. Are there other things that you can think of that might affect school, defining school success or accountability with some of these new education savings account plans? It feels like there's always this kind of gravity that pulls us back towards test scores and no child left behind style accountability. But we can find out in almost infinite number of other things that people care about through these conjoint analyses. So are there other ways that these kind of might inform policymakers or creditors or funders in terms of what how we might think about success in these schools? And I think just to lay my cards on the table, I generally think that parental satisfaction is the best form of accountability, right? We've kind of tried the accountability thing in policy in the past. And we actually find that government is just kind of an OK regulator in terms of holding schools accountable to high performance, right? That parents are actually pretty good choosers of schools. And they're really excellent at holding schools accountable. And I think one of the interesting things about some of this research is that parents don't put zero stock in test scores, right? They actually consider test scores to be some sort of important information that's being provided about their student learning. And I think that's consistent with-- there's a lot of research in psychology, for example, that looks at the interesting thing about psychologists is that they think about test scores as an input and not just as an output. They think in education research, we just tend to think about test scores as an output to maximize, but student performance on a test score, how they do on a math section versus how they do on a verbal section, things like that. That can actually tell us very interesting things about their potential for future learning or for their labor market potential and success. Books at the right inventions and patents that they'll get degrees that they'll attain later on. It actually tells us a lot about the students themselves, the students themselves. And I think parents, similar to that research, value test scores as some sort of proximal measure of student success. I did this conjoint method with some researchers from the University of Buckingham in the United Kingdom, James Tulley and Danny Schaehl and Andrea Doegar. And we had one conjoint there with them, specifically looking at low cost, low tuition, private schools, in which we had test scores and also labor market outcomes alongside a few of these other outcomes as a measure of student success. And what we found was that when the labor market outcomes were in there, is my kid employed? Are they self-employed, running a business? Are they meaningfully employed, gainfully employed, things like that? That basically all of the parent preference went to the labor market outcomes and away from the test scores. So labor market outcomes had these huge effect sizes. And test scores basically had no effect sizes. Well, we did a very similar conjoint with Angela Watson at Johns Hopkins University now looking at homeschoolers. And in this context, we actually-- we got rid of the labor market outcomes, but we kept the test scores in there. And the interesting thing that we found there was when labor market outcomes weren't part of the conjoint design, parents actually put a lot of stock in the test score outcomes. Because I think, again, the test score outcomes kind of told them something meaningful about how the student was going to do maybe in terms of college growing or in terms of their employment. And so parents do value test scores as an outcome. And I think why this is important, why that kind of finding-- why the finding that we found with hybrid school leaders that still want to hire teachers with a bachelor's degree, if I'm a regulator or if I'm a legislator, that puts me-- that should put me-- a lot at ease. Because coming out of the accountability era, I think the concern that a lot of folks have, maybe justifiedly, maybe not, is that, well, if we let these folks hire whoever they want to, how do we know that they're going to hire qualified people? Or if we let parents choose whatever schools that they want to, how do we know that they're going to choose schools that produce meaningful outcomes? Well, we find that in these kind of community crafted spaces which tend to be very unregulated spaces, that even without the accountability pressures of some sort of centralized bureaucrat coming to knock on their door, making sure that they're doing all the things right, they already have this desire to hire someone with a degree that they think is going to meaningfully help their school run in an efficient and effective way. Or parents are already looking for some sort of measures that are going to help them understand that their students are really learning without the pressure of an external regulator that's holding the school or the school leader accountable for some sort of hiring practice or some sort of student outcome practice. School leaders in unregulated spaces are already doing things that they think will help their school be successful. And parents in unregulated spaces are already doing things that they think will help their child be successful. And the parent being the closest to the child, I think, shares the greatest interest in the child's success, much more so than a distant bureaucrat or legislator. And so I think that parent is going to be much more invested. in making sure that they choose a school that's going to be meaningful and beneficial for their child. Yeah, so you've hit on like, I think this is the ultimate like important point, right? So you mentioned earlier the data problem of these universal access, ESAs, it's taking away a researcher's tool. And the way these schools are set up, right, as home school groups, as private schools, as charter schools, as public schools, they show up in all kinds of different data sets or not at all. But these are researchers problems and policy makers problems. And one reaction is to try to kind of re-standardize everything. But it sounds like what your work is showing is a better reaction is instead to actually ask them what they care about. And then we're learning that they care about we probably pretty good things for their kids, right? Yeah, they care about student learning. They care about the future potential of their child. It doesn't have to be college going. It could be getting a job in some other way. They care about civic engagement. They care about community engagement. All the things that prior research has kind of told us, and I think just our natural intuition would tell us, are good things that we want a healthy education system to instill in the next generation. These are things that parents care about. Whether or not there's some sort of a regulator or accountability bureaucrat that's holding them to some standards, they want their child to flourish in these ways. Yeah, perfect. All right. So last question. Right now, what's the most interesting thing happening in your work at this moment that people involved in these new school models would be interested in hearing about? I told myself I would think a little bit more about this question, but. I mean, you started by saying you find my work interesting and that makes you and my mother at the very least. So actually, I'm not sure my mother is aware of the kind of work that I'm doing, but. Yeah, I think, you know, we want to take these kinds of methods and apply them to these choice spaces a little bit more broadly, right? I think the interesting thing about school choice, maybe the wonderful thing about school choice and policy now is that school choices very suddenly become too narrow of a term to describe the kind of education revolution that we're witnessing, right? These universal universally eligible programs, education savings accounts that are popping up in these states that. A lot of states have passed have really made flexible learning more possible. It's more possible now than it has ever been before for parents and their community to I keep popping on this term, but I think it's an excellent name for the podcast, right? It's more possible now than ever before for communities to craft education for the students and the children in that community because these education savings accounts are allowing parents to use these funds, not just to pay tuition dollars, but to hire tutors or to go and field trips by curriculum resources. So, I think, really, to augment and enlarge what education can possibly mean for their kids. And I think we're seeing this explosion of like we used to think education had to fit inside this box. It had to mean, you know, five days a week, it had to mean six hours, butts and chairs surrounded by four walls, a blackboard or a whiteboard or now a smart board, you know, at the front of the room. And adults, a professionally trained adult with an education degree standing in front and teaching 15 and then it became 20, then it became 25, then it became 30 students in a classroom. And we're seeing that these new education savings accounts are really turning that conversation on its head because now all of a sudden it's possible for us to totally reinvent what education means for kids. And I think that's a wonderful thing, right? So, school choices become very quickly this too narrow term because it implies that it has to look like a school and it has to be a particular choice. We're now, I think we're seeing much more flexible expressions of education freedom, right? I want my kid to be educated in a context without walls. I want them to be in nature, you know, or I want them to be traveling in a broad maybe there, some sort of high performing athlete or musician and they need the flexibility to learn about, you know, important things while they're in Paris competing at the Olympics or something like that. So, I think what school choice used to me has now exploded to be just totally education freedom for parents to figure out and reinvent and revolutionize what kind of education they want, what education can look like for their kids. You know, I hope in case you were going to be continuing to work in that I would school space, but I think we're also going to start exploring some more of these options. We're launching a new research initiative education freedom research lab in the fall to really expand the kinds of innovative expressions of these new education options for kids. I think these conjured experiments will continue to be an important way in which we look at how people in a space where they're free to make choices, how do they make those choices, right? And that could be school leaders, that can be parents, that can be other stakeholders that are involved. And I think there are lots of interesting choices to make and I think we want to be looking at all of that. I think, you know, to close out, I think if there are any parents in particular school leaders, one of the ways in which we can really expand the scope of our research is if we can start to get more data from parents in terms of understanding the ways in which they navigate the space. So the more cooperation that we can get from school leaders to include their parents in some of this data collection that were engaged in can be really, really helpful and really, really powerful, I think, for demonstrating hopefully the full spectrum of creativity of the ways in which parents and school leaders and these communities design education spaces for the next generation of kids. Perfect. Yeah, that is a lot and I still think it's interesting. Your mom is right. Matt Lee from Kennesaw State. Thank you for joining us today. Thanks for having me here. This is great conversation. Yeah, see you soon. Thank you for listening to Community Crafted Education, a podcast hosted by the National Hybrid Schools Project and Kennesaw State University's Michael J. Cole's College of Business. If you have any questions about the topics we discussed on today's show or want to learn more about what we do, please look up the National Hybrid Schools Project or reach out to us directly at Cole'scollege.com/hybrid.