#7 Press Pause: Rethinking Systems and Evaluation with Emily Gates

[Music] You're listening to the It Depends Podcast with Matentineal. Talking practice without pretending it's perfect. Place to go when your question only has one answer. It Depends. [Music] On today's episode of The It Depends Podcast we chat with associate professor Emily Gates from Boston College in the United States. Emily is an evaluator, educator, and systems think of his work bridges theory and practice with a focus on how evaluation can better reflect the realities of large-scale systems change. She's also just co-written a book. An evaluative inquiry for systemic change, a guide to shift beyond fixes to lasting value. I'll link to it and other things we discuss in the episode can all be found on the show notes. We hope you enjoy our conversation with Emily. Be sure to follow us wherever you get your podcasts to keep up to date on the latest episodes and to check back on those that you might have missed. [Music] Hi, I'm Emily. We're so thrilled to have you here with us and appreciating the time difference. I hope you're not feeling too tired after a long day doing all the wonderful things that you're doing. No, I'm happy to be here. Thank you, Matentineal. That's great. Thank you so much. I think the best way for people to learn how you are is, you know, direct. For the benefit of those listening, how did you end up with this? How did you get into this wonderful world of systems and evaluation? Yeah, so I'll try to keep it a shortish version, but it was almost two decades ago now. So a lot of people will say they stumbled into the evaluation field. It's like they're an accidental evaluator. I'm kind of an anomaly in the sense that I totally sought out the evaluation fields and not just the evaluation fields, but the research on evaluation space, even though I didn't know it was called that. When I was an undergraduate, I had a couple opportunities to do fellowship. So one of them I worked at a youth shelter as an intern. Another one I worked at a community organization. And as an intern at both places, I basically got to learn about the work the nonprofit was doing. For the youth shelter, I participated in direct services, like spent time with you. It's in programming with them, engaged with the leadership staff, the board, all the different points of view. And I kept hearing how much evaluation sucked. And I was like 19 years old. I didn't know what evaluation was, but I mean, I was like, it seems like it wouldn't suck. You care about what you're doing. You're trying to make an impact. You're here for a reason. And you're a values are personally lined with the work you're doing. You want to make a difference. Is that not what evaluation is about? And I was hearing from people, you know, frontline practitioners and leaders in the nonprofit space. This was in Florida at the time that what resonated with them in terms of the meaning and the why behind the work and the real questions of impact didn't show up in evaluation. So I ended up as an undergraduate doing some research on evaluation. Again, not knowing it was called that time and surveyed nonprofit leaders and interviewed them. And looked at basically why evaluation sucked for frontline practitioners, particularly those that were working in complex issues. I called it community building spaces at the time, but some of my main findings from that early study was around like it depends on what nonprofits do. Those that were engaged in deeper work harder work were more resistant to outcome based evaluation, which was the model of evaluation that was pushed. So fast forward, I had a route that eventually took me into grad school for evaluation. I worked as a teacher for some time, a few years out in California after undergrad. And I wanted to do evaluation there and they said I could do it on the weekend. You know, my day job was to be a teacher and to actually follow up and find out if teaching was making a difference was not my day job. That was on the weekend. So I I sought, you know, a PhD to actually figure out how to do evaluation, but I was skeptical from the start because I really wanted to do evaluation better. I wanted to understand why we evaluate the way we evaluate. And so it was also starting to ask systems questions, which we could get into, but I didn't know they were systems questions at the time. When I studied, I did my PhD at University of Illinois, or Banna Champagne, which is in the US and has a really strong reputation and culturally responsive evaluation, mixed methods, democratic traditions. Like a lot of really exciting evaluation work. And most of it was program evaluation. So then when I was doing my doctorate, that's when I really got into system science and systems thinking and I had to do a lot of independent study and even comprise a committee of people outside my university and outside of the evaluation field to sort of navigate forward in the systems and evaluation space. I finished in 2016. So it was like when systems and evaluation was getting going in the broader evaluation field. And then right now I'm an academic. So I worked at CDC and public health for a couple years and I'm at Boston College now. So I'm in an academic role because I actually get to do sort of the heart of what I care about, which is not just consult and do evaluation, but actually think about how to do evaluation better. So I engage with students around those questions, right? Think about what's the future of evaluation? Well, it's next. So I do a mix of writing, doing some studies with practitioners who practice and then teach courses. I will also just flag that I think he might be one of the first guests. So we've had that specifically had the phrase it depends in the exact right spot. I didn't notice that in my talking. So I was like, it was intentional when we named the podcast, but the thing I find quite interesting is that you mentioned about like that idea of asking systems questions without knowing that those systems questions. And I think that's also quite common for a lot of evaluators who aren't, yeah, sort of in that function. They kind of have that natural precluded to ask evaluative questions without sort of knowing that that is a whole field of practicing and and things like that. And it's like a natural inclination to want to know this questions of why and things like that you fell into evaluation, not quite. But did you, would you say that you fell into like the systems bit of it and the topic area? Honestly, I still feel that I had to seek it out because I had really great mentors, but I just didn't understand the boundary around interventions and and in the program evaluation space learning evaluation. And you know, it usually begins with an intervention that has a goal or a set of purposes and intended outcomes. And I was like, but where is it situated and they said, oh, yes, context matters, of course, context matters. Let's consider the contextual factors, but it didn't land with me well that the starting place was the intervention or the program. And no, my sort of intellectual community in grad school wasn't familiar with the system sciences at the time. Tom Schwant was one of my mentors and became a really lifelong colleague and friend. And he engaged in the system science and the scholarship with me like in parallel when I was in grad school. So asking systems questions, I think in seeking out sort of training and systems theories and methodologies, I see a two different things. So even though there's a natural intuitive way of being systemic and engaging systemically, I do think sort of a discipline systems practice is something different and for that I had to seek it out. Your story Emily resonates with me so much because I also came out of my undergrad and did my first placement. It was in mental health and our colon drug and I was given a project which was an evaluation and it was purely an outcomes focused evaluation. And as we would all know, it is so difficult in that field, particularly our colon drugs to have positive outcomes that are long term and to show them. And it was so deflating for practitioners to put the effort in and sit down with the evaluator and invest their time into the evaluation to have a report that did not capture the impacts and the outcomes that they saw and that they felt. And that was actually my first exposure to the flaws of what evaluation can be if it's not done probably if it's not considered in the context and if we don't consider the wider system. So I was listening and I really felt that because I had a similar experience. Well, it's a meal just to briefly add I mean what I didn't talk about I was talking mostly from the practitioner perspective, but it does so much damage right in terms of funding when decisions are made then based on those outcomes and anyways, yeah, like thinking about the ripple effects out of the consequences of of narrowly focusing on outcomes or sort of presuming that being able to capture outcomes means you're doing better or more impactful change that doesn't make sense. Those are two different things. Yeah, absolutely. And I think something that I still carry with me is sometimes around recommendations and the comfort level of making recommendations when you don't feel that the evaluation has actually captured the impacts in the first place and then on top of it potentially changing funding decisions, you have to make recommendations that are truly not sitting well with you. And that is something that I could talk about for ages that I'm still sort of working through myself in my own evaluation journey. Yeah, I mean, I'm happy to engage with a little bit and then branch sort of beyond because yes, like in in my training and orientation, we talked a lot about whether to make recommendations, we talk about that in my evaluation practice with students. But to me, it's not so much about the recommendations like the idea of single double and triple loop learning comes up for me as salient because you know single loop learning being if you're doing things right. A lot of evaluation is premised around program design program implementation and intended outcomes to address a pre specified social problem or need right that sort of orientation to what about evaluation is and how it fits in social problem solving. Then you could come up with recommendations that were pretty focused around tweaks to the program design implementation or learnings in relation to outcomes, intended or unintended. But my concern is less about the decision to make recommendations or not and more about that underpinning model because recommendations within that model can lead practitioners or interveners or funders and commissioners of interventions sort of keep going with the set of assumptions they have and with the intervention design and strategy and roll out that they've got. And that's sort of been my concern. Is interrogating that. We could call it a mental model, but it's not. It's like a mega frame, if you will, that shapes the way that we understand issues and address them. So to me, recommendations that are real probes around-- but they shouldn't come at the recommendation point. So it's also about the role of the evaluator and when to engage around that. But thinking about recommendations that also can take whoever's engaged in the evaluative process further in their thinking around the question-- sort of double and triple loop questions around whether they're the right things to be doing and the underpinning assumptions and values that are the basis for thinking that those are the right things to be doing. How do we engage in that space more? And what does that mean for recommendations? I don't know that the recommendations are the right language, like some sort of inquiry probes or something. But then the idea also about just sort of delivering them to people doesn't make sense as well. If they're more instrumental in that first loop space, it makes sense to make them or not make them or phrase them in a particular way. But I think if they're higher order, then the format for the recommendation process, to sort of inform adaptation, would have to shift to. We could go on forever, couldn't we? So you mentioned earlier some of your working systems. So we are really interested in knowing more about this. We know that you're quite prominent in the systems thinking and evaluation space. So for you, what is a system and what is thinking and then what is systems thinking? What a question. So all first, repressive by saying that people in the systems science systems thinking community have engaged with us and heavily debated and disagree. So there's no one universal definition. I mean, to me, the system is-- well, I'm going to talk about system and systemic. So a system is a set of things and elements, activities that somehow have some coherence, some relationship to each other towards a purpose, towards an end, or an outcome, something like that. And an important piece of the definition is that somebody sees it as a system. So there's a sort of discussion we could have around whether systems exist or whether systems are frames that we as people use to make sense of the complexity of the world around us. I take sort of both and stance to that. Yeah, systems exist. And systems are frames we use to engage with the complexity of the world around us. So I also like the term systemic versus systems thinking. So I think about systemic as both an orientation and an object. So that sort of captures both thinking that systemic is how we engage, how we understand, how we relate, and what it is we're talking about, what it is we're looking at, or what it is we're trying to engage with. So that's sort of the both-and-piece. The thinking-- I don't know. I don't know so much about the thinking piece. I know that there are people who have a sort of real cognitive focus on systems thinking and cognitive then sort of locating systems thinking in individuals or in small groups, interpersonally, thinking about systems thinking as literally a cognitive process or a social cognitive process. I'm much messier in how I think about it. I mean, I think about systems thinking as a way of engaging in the world and making sense of it together. And there's different types of things that support that. And there's some classic ways of engaging. So you've got to attend to relationships and interconnections in your processes. You've got to shift perspective in terms of world view and value. And it's important to be transparent about boundaries and where you set them, how you set them, and what the consequences are. So that sort of interlular relationships perspectives and boundaries are key aspects of what's known as systems thinking. As I said, I'm a little bit looser about it. And it's like people do that together. Sometimes it's really using formal systems tools. And sometimes we do that almost organically in a conversation. But there are-- I think we can get better at it. And I guess the next part of that question is what does it mean for evaluators to systems thinking? I know you touched on that you don't necessarily resonate with the word "think so much." So whether it's systems practice or whatever terminology you tend to use, especially in your teaching, how does that vary for the different roles in evaluation? So you've obviously got the evaluators themselves. You've got the commissioners, or the funders, or the implementers. What's your perspective there? I'm going to be saying it depends multiple times in this segment. But to me, the boundaries piece is usually the starting. What it means to engage in systems thinking a systemic orientation systems practice or all of the above is to reject the given boundaries. And that is not what we usually do. And I say we, including myself, I say we in thinking about evaluation as a commission activity, just like you describe it. So evaluators in the way that we've professionalized, or the field has professionalized, we traditionally exist either in academic spaces, research spaces, or consultant spaces. And there are a whole bunch of boundaries that shape what we do and how we see what we do. And that oftentimes is framed by developers of interventions and funders and commissioners granting cycles, the request for proposals, whatever you call it. So there's a bunch of boundaries. And then we enter in conversation with different groups. And there are already assumptions made. And I would call those boundary assumptions around what the fundamental issues are, whose issues they are, where they're located in a larger system or situation, why they matter. So there's sort of behind the scenes of a particular issue intervention. And then there are usually boundaries around what we want to know and why. So what success means, what counts as evidence, rather than thinking about evaluation as a technical exercise, to me, a systemic orientation is it's a critical exercise. And it critical means naming and questioning those boundaries and saying, are those right for our work moving forward. So I'm leading with that. And I want to name that that comes out of critical systems thinking in critical systems practice. It's the least attended area of systems practice. But then yes, beyond that, or sort of what I would say are the more obvious things, which are to make connections. So locate things in context, think about factors, relationships, dynamics, feedback loops, right? The way that those shape processes, yes. So that's important in this ability to shift perspective and engage with shift of perspective, whether that's cultural, disciplinary, et cetera. All those are parts of the practice. But to me, if we don't question boundaries, and if we don't have the power and the professional autonomy to reshape those together, we can't really do systems evaluation. It's really interesting to me that you mentioned things in that way, because I mean, Teneal and I, we were literally delivering some training yesterday to some people around some of the systems thinking stuff. And one of the things that resonated with the group the most was that systems thinking isn't just like mapping. It's not just making maps. There's so much more to it. And I think a big part of it was-- because we did mention critical systems heuristics and some of those boundary critique ideas. And I think people-- it's kind of almost the systems thinking as a practice is now at this stage. We're kind of-- it's kind of designed thinking a little bit. People have heard the phrase. They know some of the outputs or some of the diagrams or some of the images. And so they're like, oh, systems thinking is the maps, right? That's what you do. Do you make a map? And I think actually, that's a very surface level response to a situation. There's so much more that you can do to think systemically that isn't just about making a map or making a diagram. There's actually a lot of questions and perspectives or positions that you can take on something that are arguably more important or more transformative. Yes, maps can be helpful as a constructor, as a representation of something. But without the rest of it, it can be a bit-- a bit not unhelpful, but maybe not as transformative as you might think that it would be. Yeah, Matt, can I pick up on something you're saying there? Please. Teach me. OK, go on. No, it's not about-- it's just me. It's not about teaching you at all. So I've done systems mapping mostly with STEM faculty, like faculty that are trained in the sciences engineering fields, right? In higher education, and they're thinking about-- there used to be a long and ongoing line of funding here, federally, in terms of broadening participation in the science fields, science technology, engineering, and math. And then we would do systems thinking workshops just in the way you're saying. And the experience of what it meant to engage in systems practice was so bounded within someone's understanding of what their professional role was. So it was like, well, yes, I'll do this inside my classroom. because I have some say over what happens in my classroom. And I'm like, "Oh, that's pretty interesting. You have limits in parameters around which you can engage right systemically." And at the same time, everybody's saying that. So like, there's people who are department heads, there are people who are deans or are provost in higher education all saying that. So sort of one of the things that strikes me about what you're saying in terms of systems mapping and a concept that I've really liked and engaged with is this idea of stewardship. Like, part of it's almost like a prereq for a systems practice is to interrogate how you see your role. And I will focus on professional role, although we could talk about personal spaces, how you see your role and how your organization or your discipline or your sector is siloed, right? Is split and divided in the way that we work. And then we sort of have professional roles that are also split in siloed. So that we don't really do systems practice together in any sort of generative or potentially transformative way. Instead, we do mapping and then take off the slice of the map that we currently have a job description that enables us to take action right in that area. So anyways, this sort of concept of stewardship is the way that I see it and why I like it is an sort of an argument that I like to make, including in my own work with faculty with students in an academic setting, is like, we make systems every single day, every interaction we have, you know, thinking from buying things to voting to engaging in processes in school and work, whatever space it is. But especially professionally. Like, if we segment things and continue to only do what's in our lane, so to speak, then we're not going to reshape systems. Like to me, we have to think about the co-ownership and the co-shaping. And that is like, yeah, some sort of prerequisite, I think, for doing systems mapping. And I get it, it takes power, it takes resources, it takes time, like, but we can talk about that too. I know that's a later thread in terms of why commissioners, why funders have to be at the table. Yeah, but I just wanted to name that because I'm seeing that a lot too. And it sucks because it reduces the potential of systems practice. But I also think, if we keep sort of preaching or pushing things like systems practice in workshop formats without restructuring the way work is set up or organizations are set up, then there's not as much potential as well. You know, it's like, all we can do is do it within the lanes, unless we recreate the lanes to some extent. I mean, it's getting towards the idea of like incremental or gradual shifts rather than like something a bit more substantial that like, yeah, we can tweak the, we can optimize, we can make it a little bit better, a little bit more efficient by changing bits and pieces here. But it's like, we're not actually going to substantially address the issue, like the root cause or whatever language you want to use. Unless we think about it differently or challenge our roles or, you know, and I mean, that idea that like a funders role, for instance, it's like, you know, we fund in this way. But we want you to do radically transformative work, but here's how we fund. And it's like, well, those are put, yes, it's great to say you want that, but then the way in which you are doing that is actually limiting or restricting people as well. But anyway, we could go down that rabbit hole forever. And I wanted to put to you a bit of a simpler question, the ones that Taneel's put to you. And then I would say, you know, for a lot of the people listening who are in the evaluation space, particularly here in Australia, but I know that I could come a question all over all over the places. Obviously, people want to change systems. They want to transform them or whatever else. And inevitably the follow up question is like, but how do we evaluate that? How do we evaluate systems change? So I was wondering if you could, you know, in like 30 seconds or less, just answer for the world. Emily, how do we, how do we evaluate systems change? And I'm being facetious. You can take more than 30 seconds. That's totally fine. But I guess like to put it back to you in a more useful sense, like what does it mean to you when we say or ask that question, how do we evaluate systems change? Like what does that mean in practice? Yeah. So 30 seconds. It can be more than 30 seconds, maybe 45 to 60 if you need. That's fine. I mean, so I'll just say a couple of things. I think the first thing is like, I don't want to engage in evaluating it unless there's some sense that it's happening. And so I want to sort of start by saying it's not automatically happening. So yes, on one hand, I could critique myself because I said it's both an orientation and an object. Right. Like so we're changing systems while we're engaging systemically in a change process. But I have heard the idea that, you know, we're doing a bunch of things. We've got strategy in different areas. There are different interventions at different scales. And now it's time to evaluate system change. But from a design perspective, from the development perspective, there wasn't actually systemic things happening in the process. So I can talk about the book that I finished with Pablo that's coming out soon. That's all about how to evaluate system change. But sort of we had to start with what is system change. We use the phrase system change in the book to highlight the meaning. But it itself is not just doing a bunch of interventions and then looking for greater impact. Right. So so my my first piece of how to evaluate system change is see if see what support there is for system change happening and how you can support the change process in being more systemically in being more systemic. So you mentioned mapping like we identify myself and a bunch of graduate students have reviewed a whole bunch of practice guidance to sort of first say what the heck is systemic change in action and practice. And it's not just a phrase and it's not just like once you do a bunch of interventions are doing it. I've repeated myself there. But so we found the idea of systems mapping being a piece. But even before systems mapping, it's that engagement with complexity in terms of the situation and thinking about what change could mean to whom in terms of the visioning process going through systems mapping to identify leverage areas leverage points being part of it. But also a kind of futures orientation to navigate forward and then thinking about where does evaluate fit. So then how do you evaluate? So if you know some systemic change processes are are happening, where do you plug in? So again, I'm I'm trying to sort of pause and slow down because system change evaluating system change whatever language gets conflated with like capturing big hard to measure impact. And I don't believe that that's what it is or that's all it is. Right. So like you can you can evaluate systemic change in terms of supporting a group's capacity to engage systemically in a change process in any phase of the process. Right. So if there was problematic exclusion in the boundary definitions early on of what the goals of an intervention are, right. So something gets you know cast as I'll speak in the STEM space and say mentioned that already oftentimes interventions get designed as ways to fix students. Like student ability, student access, student resources and funding and opportunity. And they don't actually locate the issues in the institutional culture, the department culture, etc. So early on you can evaluate systemic change by helping to interrogate the boundaries around the intervention. I would call that evaluating systemic change. You can do that at any stage of the process. That's not satisfying to most people because they do want to answer the question of like how do you evaluate the bigger longer term hard to measure impact. The thing that I will say about that is that there's no way like there's no one way. There are a lot of ways. And I don't know if this is jumping ahead to another question. But again, in the book with Pablo, we looked at a lot of evaluation models and approaches. There's I will offend people in the evaluation fields, but like we've weighed too many labels, buzzwords, names, jargon, acronyms for not even this like there are not fundamental distinctions between them, right? So what we tried to do is say what are the core fundamentals? Like what has to be there if you're going to do evaluating of system change or systemic change? And we boil it to five main things, which is like you have to be clear about purpose. You've got to have your questions, your criteria for what counts as value need to be there. You've got to think critically about the evidence basis for answering those and have a sense making process. So it's a giant interpence kind of answer because it's like well, what are we talking about with systemic change? Where is the systemic change process? And then how do you bring the evaluative elements into that systemic change process? So I will stop there for now because it is it's it's it depends answer to me a sort of better question is not just how do you do it? But it's the the question that is inside the question around what good means or what success means. So like we can evaluate system change by bringing our evaluative elements into systemic change processes, but what we lack in both theory and practice in my rate of sort of where the field is at is is a working idea about what success means like what what is good system change and I think I'd like to talk more about that of how to handle that, because I see that as a value and it is responsibility more so than the methods we bring to answer whatever the questions are. Yeah. Yeah. I mean, that question of like, how do we do it? It's like, it's coming from a place of methods. Like, what's the method that we would use to do this? Yes. Like, that's the implicit question or the sort of the underlying question that people have. But I think what's quite interesting is, with taking a sort of a high level, like, you know, here are the five considerations that are being clear on those, is you mentioned earlier that if anything, arguably, anything can be a systemic and its orientation, there is this idea, obviously, of, you know, we can have systems change in a place, like, quite localized, and we could be looking at it in that way. But to me, there's also a deeper question, which is, and this is one of my favorite sort of ideas that introduced me to a little while ago around, like, there's complexity in terms of experience, like experiential complexity being that people will experience something differently, and that, like, there are a range of factors and reasons underpinning that person, all those groups as to why. And the idea that system change is, I think, fringe universally as, like, a positive for all, as opposed to recognizing, like, there are systems that exist for reasons that benefit some and not others. And that, you know, we can, I think, and this is going to sound like maybe incredible, but I think we can kind of get caught up in a bit of an echo chamber, it'd been like, we all want to change the system and make the world a better place, and not recognize it, like, it's not a universal view as to what good looks like, as you mentioned, and that there are some who actually will disagree that things need to change, and not to say, like, it's because they're vested interests, say, commercially, but just because, like, they're not really that hard done by, you know, context relative to others, and they're like, is there really that much of a problem? Like, yeah, some tweaks, it's fine, like, it's not really that big of a deal, but for others, obviously, they're, and like, that, that different perspective on the context or the system to change, is not actually that straightforward, depending on who you are, and I think, you know, for groups that are typically marginalized, like, it's more clearer that things need to change quite radically, but for others, you know, they haven't had that same experience, they're like, no, no, no, like, is there actually that much of a problem? Like, seems fine. Yeah, so I mean, like, we could also just say, and like, real anchoring in systems practice would laugh at the phrase of systems change, right? So I said I use systemic change, finished a book about it, but the world is always changing, and we bring a systemic orientation to the world. So the idea of systemic change is nonsensical, it's almost meaningless, right? 'Cause systems are in flux, they're changing, the world is complex, it's changing. It doesn't inherently mean, right, anything, let alone it being good, right? So we've got to sort of set up what it means. Yeah, so let me share this piece too, and then see how it lands or what resonates, but sort of going back to that idea of methods, like people want answers to how to evaluate system change. And that's, yeah, assuming like, change to system is happening, I wanna know, and have some sense reliably of how much is happening, to what extent, in what ways, et cetera, what factors contribute, all of those things. And not first saying, like position, so what that does is it positions the evaluator is the answerer to the question. Like evaluators hold expertise in methods, yes, that's true. And methods are used to gather and analyze data to provide evidence to answer, you know, to answer questions. That's also true, but I don't think that's the role of the evaluator in the evaluative space with system change. Pablo and I use the phrase of critical deliberative, like critique and deliberation. So it's also not, and I can sort of engage a little bit with some of what Julian King's podcast was about, but like evaluators are there to engage with questions of value centrally, right? To what extent, and what's the value, to make determinations of value? That's the core judgment of what evaluation is or definition of what evaluation is. But we sort of expand the frame and say, it's not to assess value, it's to actually develop value. And to develop value is to interrogate the meanings of value from a pluralistic perspective, and to render critique, and to support discussion around the way value gets defined, the consequences of how it gets defined. And to me, that's as important or more important than the data and methods question. So another way of answering your non-30s, another way of adding to my non-30s, second answer of how to evaluate system change is to support deliberative discussion about what matters and who was involved. So what does systemic change mean to what extend is there agreement? Who's left out of that process? What are the consequences of defining success in particular ways? And ensuring that there's ongoing space to be integrative to multiple world views and underlying values, but also to advocate for who's excluded, what world views are excluded, especially in Western or colonial and indigenous context, or depending on the context, but in particular, thinking about indigenous right to self-determination and spaces and who should be having the most power of defining success is not, yeah, is people most affected and also sort of victim or experience to the injustices that are byproducts of the way unequal systems are designed and implemented, yeah. - Right, could I just pose to a term that came to mind when you were describing there as to what the evaluator role is in this? I think you're like, it's like less evaluator in terms of rendering judgments and more like negotiator, like there's like a negotiated space within this question of what does it mean to evaluate systems? And as you said, there's lots of different questions that emerge from that that will require compromise by different groups and things like that and oftentimes, that facilitated process to come to a compromise or a negotiated process. And so I wonder if there's less evaluative junctions being rendered and more by groups and more negotiated or something, I don't know if that resonates with you, but it's negotiated as long as the negotiation has got some open-endedness to it. So like, I mean, we use the phrase evaluative inquiry instead of evaluation as an also a way of opening up the idea of an evaluation culminating in some sort of conclusion or judgment. And I'll say like I've heard pushback and engaged in conversations with the idea of like negotiator or like sort of playing a deliberative role, meaning that the evaluator themselves or the evaluation team doesn't advance values, right? Or doesn't advocate. And I have made the argument and some of this is from earlier work with Tom Schwanton, our book on valuing and evaluating like, we emphasize that the evaluator's role is that space-holding role. It's not a neutral role, like absolutely values are endorsed and pushed for a forward, but it's a kind of fluid thing where to me, it's like, once there's a lot of agreement or alignment in one space, the job of the evaluators to say, "Huh, what did you leave out?" Or what's the consequence of thinking that way? Or what's the underpinning assumption? And so you're almost like playing counterpoint to whatever the center thing is, but that's what helps group self-cratique as they move forward. And so it is a negotiation, but without sort of ever landing on a resolution. And that's also, I think, a long view on what systemic change means because I don't think it can be something that begins, happens and ends. Like, systemic change is an ongoing way of engaging and change. And so there needs to be space for negotiation in an ongoing way or negotiating or co-negotiating or however we would call it. And you've got me thinking off the back of that about the role, I suppose, of commissioners in evaluation as well. And we were just talking about value and how success can be defined by the value. But when there's so many different perspectives in the system, then value is going to look different for so many different people. And I suppose it's just got me thinking that often commissioners have to assume what success looks like for the people in the program, for example, if there is a huge diverse audience or diverse population, and they'll put that together and sort of send that out to the evaluators. And yeah, I guess I'm questioning whether those assumptions have just been made by commissioners about what value is for other people or how that's been designed. And then how you as an evaluator, especially if it's a systems change evaluation can, I guess, take a critical lens to that and how you can work with commissioners, I suppose. So I guess what are some considerations, maybe, that commissioners of evaluations of systems work might need to consider? Yeah, I was thinking a few things as you were talking to, Taneel. So let me start by saying a bit bluntly. It's not about you, right? It's not about the commissioner. It's not about their strategy. It's not about their mission. They play a very important role, especially when they-- like, I'm thinking also the difference between philanthropy and government. So another thing on my mind right is what's happening to public institutions and public spaces. So in democratically funded government evaluation, in some ways it can be. about the commissioner because the commissioner is holding space for democratic institutions and the commissioner exists on behalf of the public, right? But in philanthropic spaces where they're private investments or social impact spaces, right? Or private investment spaces, there's not that same democratic socially responsive ethos. And so there's a sense of like what the commissioner puts forward maybe extremely well-intentioned and well-resourced. And that's where I would really say it's not about you, it's not about the commissioner. So to me, I want to return to that idea of stewardship too. It's like a really big shift in role. But here we're not talking about professional role. We're talking about institutional organizational role. And even sector role. I really don't think that one, you know, funder or commissioner in one sector of society should be defining success for a system change on any complex issue. But it's nonsensical, right? Like it doesn't make it doesn't make sense. So then what does the commissioner do? Will they create forum and space or do different types of listening to curate and amplify what success could mean from those different vantage points and then figure out what their contribution and their relationship is to that, right? So I also think that's happening. I don't think that that's new. I think that there are different funding, you know, different funders that are making that shift or pivoting or repositioning the way they engage in social change processes to shift in that way. And it's a much more collaborative model. It also goes against the way that our economic markets are set up. And the way that that shapes the way resources are deployed and received and acted on. So there's a whole bunch that also has to shift. I get it. Each actor in what we're talking about is constrained to different extents. But yeah, it's shifting role. It's engaging in dialogue. It's thinking collaboratively like this is where like we talk about contribution versus attribution on the method side. But there's the prerequisite for contribution too, right? Like to really look at contribution, we've got to care from that stewardship angle and from the wider angle around our relationship to systems and have a longer view in terms of of what our vision is values are, like what we care about, what we're working towards. You mentioned that each actor is kind of constrained. But it brings to mind. And one of the people talk about the systems too big, we can't change it. If you go to Google and type in like ferromagnetic materials, there are great images that come up of basically like our magnetism works. But basically, you know, arrows are pointing all in different directions. And then for something to become magnetized, they all point in the same direction. And it's this idea I think of like, yeah, if you've got a space where everyone's pointing in a different direction, of course, nothing's going to shift. It's all too big and too hard. But it is about orientation. And like if we're all working in the same direction or pointing in the same direction or come, can come to some degree of collective agreement that economic systems need to shift, then it can happen. It's just that it's hard because there's a lot of inertia behind them and how they've been set up and things like that. So yeah, there's lots of critical reflection required for like those bigger radical shifts. But I think you've hired some really important things throughout this round. Like it can happen at any level, but like we shouldn't just default to system changes. The big stuff only that's long term and hard to foresee, but it is actually much, it's different, it's different to that. I mean, people in therapy rate counseling family systems like, you know, our bodies are systems, right? Our families are households are systems, our neighborhoods are systems, right? So like, it's not an abstract thing. We exist in these nested systems if we to call them that or we encounter a complex situations at different levels and you can bring a systemic orientation to that. I don't want to sort of give the impression that system change and my understanding of systemic change. Like we attend a process, we do it in collaboration. There are different systems, tools, practices that we can bring to bear and that's it. So another, I don't know that it's a red flag, but it's a pattern that I've been seeing because I did both, like a big interdisciplinary scan of system change and transformation, what's happening in the academic literature, in the evaluation space, and then did a parallel scan in the practice space, as I mentioned across all these different guides. And there's a disconnect in each of those spaces, but like the disconnect that I want to name is the lack of underpinning science behind the ideas of shifting system. So like there is space and need for the accumulation of knowledge around systems in particular areas and issues and that be brought into systemic change processes. And I want to hone in on it around leverage areas leverage point. So like this is something that I'm trying to engage in now more recently, but like the idea of finding leverage and influencing leverage the way I talked about it in a very sort of simple way is like do it, do some systems mapping and identify those, but there is so much learning that can be had, I think, interdisciplinary, interdisciplinary to think about what leverage really means and looks like and what it really means to do it. And yeah, I just want to see sort of more theory, I mean, I can say this in academic, greatly more theory in academic work or empirical work in the space, lifting up from cases and from practice in that space because there are risks to trying to then like get overexcited and have bigger impact around ideas. Like you can have strategy that has more in higher ambition. And at the same time, there's risk of consequences in doing that work. And I still think there's very much a need for interdisciplinary and cross sector learning from that. It's not that like everybody all at once does system change as processes. And then we change macro level systems, right? Like so it's that yeah, anyways, I just want to just sort of name that because there's we looked at the sustainability sciences for example where there's like really rich work and public health is in other fields like there's really rich work that could be lifted up and engaged with cross disciplines. And you've obviously given us a lot of your time, but there is as you've said a world of stuff out there, you've mentioned a couple of times that you've got a book that you've written. So I'd be great to hear a little bit about that, but also like title when it's coming out all those sorts of things, but then also maybe like other sort of essential reading or things you'd be suggesting people check out. So the book is called a value of inquiry for a systemic change. And we talked about those terms already from evaluation to a value of inquiry and a systemic orientation to change as well as changing systems. And then we have a subtitle which I can see if I can remember it's it's something like I won't remember. I like a guide from fixes to lasting value something like that. So what we do in the book is we really critique what we call a fixed mindset and a fixed orientation, which is really this like diagnosis problem designed an intervention implemented and evaluated. And so evaluation existing in this very narrow bounded problem solving process. We say first we have to sort of tear that open, break it apart, do something different to really engage in systemic change. And the book is for funders, evaluators, practitioners, people who are you know in the social change space who are like I know I want to shift system right I want to change system. What does that mean? What does that look like? And our goal was not just to create practical process right like there is practical process out there. We tried to distill it in terms of what could be supported by the academic scholarship and the state of practice now to the fundamentals in both system change and evaluation. So there's that. And then we also tried to create a bit of space for dialogue. So it's also really tricky when we don't use the same language and we don't understand concepts in the same way. You know across discipline, but across role. So the other thing we really tried to do and we did that through you know different rounds of peer review and bringing in different types of input into the creation of the book is to try and create what we hope is some sort of grounding for conversations between funders, practitioners, and evaluators. So you could read a chapter, read a section or you know engage in some engage with something in there and have a conversation and then sort of have a shared space to go forward from. I think you also asked me about other resources. I would characterize it as like it doesn't have to be the best resource but maybe one that you particularly like. Maybe that's a good way to think about like one one that you think it's good for people who are maybe entering the space. So not like a not intro to but maybe one that's like a good thought starter. For instance. Yeah, I mean there's just like too many things I can mention so I will go with the ones that I if thought about you know coming into this call because I mean I have like sort of endless amounts of recommendations for things because that's what I get to do. Dabble in both evaluation and the systems face a lot and read a lot. I have to shout out Tom Schwann's book on evaluation foundations. So it's from 2015. I still use it in my theory class. This is for people on the line or listening who are evaluators who are trying to get a broader sense of the evaluation field and who are like I get it. It's not just about methods but what is is it then really and what do I need to care about so his book it's called evaluation. foundation's a life of the mind for practice. And that's what it is, like a life of the mind for practice. And it centered around big debates, you know, what counts as evidence, the relationship between politics and policymaking, how that influences the way we evaluate, like a lot of what was in Julian King's episode in relation to values is connects with his chapter on values, et cetera. Our ideas of use and influence, right? And what underpins that. So the evaluation foundation's books will give people in evaluation and funders a really good sense of all the different ways of thinking about evaluation and also that it's not a menu of models, but instead it's a stance, it's a way of relating to the core issues and debates in our area. So that's one. And then I think it's the Amidior Network did a systems practice guide. So I can link that so that you can link it for readers to, I mean, there are, I think we've found 30 or 40 guides at this point in the systems practice space or changing systems. I like the Amidior Networks systems practice one. It's really great. And so those people who are interested in the system change systems practice space, I would say to look at that. And then I didn't name explicitly, but I did some of my training at the Open University's systems thinking in practice course. So if people don't know that they've got free stuff, they have stuff that comes at different time, amount of time investment in terms of degrees and cost intuition, but like you can also look at the books that are affiliated with it. They've got ones on fundamental systems thinkers, array isons work, others. So yeah, I would recommend that for, you know, evaluators or anybody who's like, I want to sort of expand my understanding of systems theory and systems methods. I was just interested to know actually when your book comes out, Emily, because you didn't say and I like to flag it so I can make sure we get it. - Yeah, well, I, it's supposed to come out now like next week. So it's published by Sage in the research and evaluation method section. And if you look up a value inquiry for a systemic change, you can order it or pre-order it through Sage as the publisher and then it'll be on the other places where you find and buy books. And there'll be a Kindle Edition and that type of thing if you prefer that way. - Well, finally, we always ask this question at the end for our guests. There are any words of wisdom or guidance that you would like to leave us with. - I think my word of wisdom is to pause. So there is a lot of enthusiasm around doing evaluation differently. And evaluation and evaluators have a good bit of responsibility for the way we do evaluation now. And if you don't think that's the case and you're an evaluator, ask clients, ask practitioners, ask other people because the way in which evaluation happens, yes, there are other factors, but there's a degree of responsibility. I think the evaluation feeling evaluators have for the way evaluation is practiced. So pause and I would like to see more critical reflection and engagement with what does it really mean to do things differently? How confident are we in different ways of doing things differently? And there's also enthusiasm about our role in transformation, which is really important, thinking about transformation of government, social ecological systems, right? The space between government, social systems and ecological systems. And we know we need transformation for the future of humanity, the future of the planet. And in that space, I would also say, pause. And we talked about this before. It is not a universal, there's no such thing as transformation that benefits all, especially when we talk about ecology and non-human species. So yeah, maybe it's not the wisest words of wisdom, but I appreciate the enthusiasm happening in doing evaluation differently and engaging in transformation is a large sort of shared vision for the future of evaluation as a field. And I think we could do more pause and more critical engagement with what our responsibility is, reflectivity on that, et cetera. Not concise either, but let's just take a moment and self critique a bit more. Yeah, that's exactly what Matt and I were talking about on the last day of the evaluation conference. You go along to the conference and you're hit with often all this sort of rebranding of existing types of methods or models or new ones. And it's like, well, hang on a minute. Let's just think what is actually needed, what is actually new versus, like I mentioned, rebranded. And yeah, it's a lot to digest. So again, coming off the back of the AES conference that sits with me. Yeah, I mean, the thing I find interesting about the idea of new, new named things is that I think there is a benefit sometimes in naming something when it's been used in a particular way in a particular context so that it's like, you know, we did interviews, but it's like, well, like we did them in a particular way in a particular context for a particular purpose. And we've named it this, but it's kind of a shortcut, as opposed to presenting the idea that like this is a radically different way of working. It's like, well, the interviews, you've just done them in a particular way and like you've called them a thing in the evaluation report, which is fine. And you know, there are reasons why sometimes it's good to do that, but there's a difference there between I think we have radically reshaped what it means to interview someone. It's like, well, no, it's like, it's not, it's still an interview, like it's, you know, it's not that different. Yeah. I mean, I can disclose that, I mean, it took over two years to work on this book and repeatedly I got feedback in Pablo and I together out feedback of like, oh, you're doing a new evaluation model and we were like, no, and they were like, well, then wait, what's the point? You're doing both theory and practice. It's not a methods book. So it's got to be a new evaluation model as if that's the only type of evaluation book that exists, either methods or an evaluation approach. And I mentioned Tom Schwann's book because I, to some extent, wanted us to follow in that way. Like, yes, we want to guide practice, but these are fundamentals. Like these are sort of like I said, these are the things that like out of the 40 system change frameworks, pick five and you're going to see the things we cover in the book on system change, right? The same with evaluation, complexity informs monitoring, complexity aware monitoring, like developmental evaluation, all the different labels, blue marble evaluation, systems evaluation, there's many, many, many more. But the same fundamental evaluative elements that are in our book will be there. And you need to do those to do good evaluation, right? So anyways, like it both resonates and it's hard to, it was, it was important in authoring work to try and, and hold the positioning that this is not a thing that can be branded as a model or as an approach. Because there's, yeah, anyways, I'll leave that for another story in sort of the publishing space as well. We really like brands and models. Somehow there's an idea that that is, that is the ultimate way to contribute. Yeah, everyone loves an acronym, right? If it's an acronym, then it's a thing. Yeah, or an acronym. This has been a first-person consulting production. To find out more about us, check out our website, at www.fbconsulting.com.au. You've been listening to the It Depends podcast with Matt and Taneel. Till next time.