How to do a literature review

[music] Good morning, Nick. How you doing? You must be jet-elected something. You were in Europe. Good morning, my brother. Yeah, I was. I got home and yesterday was 4th of July. You know this holiday, right? You asked me what I was. Yeah, well, Independence Day or something. Yeah, and it's when we. Independent from England. And we celebrate by blowing off fireworks, and I have a teenage son. So his thing every year, and he had a buddy over here with him, and we had our family members and a bunch of people and blowing off fireworks last night. So I had to get off the airplane and then stay up until six o'clock in the morning, Europe time blowing off fireworks. Oh, jeez. Yeah. And you're not in the South of France yet? No, no, no, no. Today's our kids' final school day. So I just picked them up and I took them for lunch because to celebrate their grades or whatever. And now they're playing with Lego. So which is a cool age, right? So they're playing with Lego. Daddy gets to record his hot cars. Everyone's happy. And when are you going to your little French villa? Well, we do it. Well, yeah, the villa. The villa. We will go to France, but for a little bit, and then we'll do a pretty cool trip across the Alps, basically. So we started France, we go via Switzerland and then visit our good French Stefan in Austria and then basically go all the way to the to the border of Italy, right? So we'll spend three weeks traversing the Alps from west to east if you want. And then we'll come back and then we'll go to AOM, right? So that's just a couple of weeks leading up to AOM. I'll see you in Chicago. So you're like Hannibal. You're going to go from Yeah, I brought my elephant with me and I'm ready. Yeah. Yeah. Hey, that's the title of the episode. Hey, buddy, speaking of fireworks, I got to tell you this. I got to tell everyone this. So, you know, my role on this podcast is of course to be very critical of everything inside. And I cherish that role very, very much. But you know, honestly, I don't know why I was doing this, but a couple of days ago, it's actually going to your web page. And I think what I was I wanted to see if you had a new paper coming out, because let's face it, you have a new paper coming out every month or so. Anyway, didn't find a paper. And then I saw you have this award, you know, tag on your web page. So I clicked on that. And then you won every goddamn award. There is this super impressive. You won the you won the dissertation award. I didn't know that in 2009. You won the sequence award. This is a one year after me. I mean, I didn't win it. That's the thing. I came second. Okay. Second, after Galostra Hasinga, she won and I was runner up. And then the famous Nicholas Burrenty. And then you won, you know, all editor of the awards is ISR and MISQ. You won service awards. You won several best paper and best theory papers of the year. People, whenever this gentleman opens his mouth, you should shut up and listen. Hey, that's what I've been telling you now for four years. I'm serious, man. Very, very impressive. It's really cool. What a career you're having. Well, thank you, my brother. That's very nice. Very kind. Thanks for bringing that up. And, you know, in return, I looked at your website for awards. No, that's good. No, I haven't actually done that. But I'm sure you've won many awards as well. That's many. You've won me. You're going to do these things so I could reciprocate. Yeah, I was just, you know, it's really, it's really impressive when you get to see it. I don't know. So I think is it, is it like a sports metaphor? Like when your career is over, you look back and, you know, and actually value your awards or do you think about them much at this point? Yeah, I don't know. I just moved offices. So I have all of my awards and I had, I'm running on a bookshelf space. So I had to like shove them all on one little shelf. And my wife does, my wife puts them in a shoot carton and then puts it in the in the basement seriously. I have very important sports trophy and then the carton in the basement. There you go. And that's really what happens with these awards. We would, you know, they, they're pretty meaningless. Except if you're in the bubble, when you win them, they're a big thing. Yeah, probably. The funny thing is when I won that dissertation award, I wasn't there. I was working on a paper. I was actually, you know, which paper I was working on? It was the qualitative meta analysis that was published a decade later in MIS quarterly. What? Seriously? You were working with that in 2008. Yeah, because it was in my dissertation. And, and I win this award. And I was in my hotel room working on a paper, Kale. My advisor went and picked it up for me. And, you know, when they give the award ceremony, he picked it up. And then he sends me a note saying, this is the most important day of your life. You're not even here because I didn't go to the ISIS reception. So, yeah. So what happened to me? This was Ithus 2008, which was in Paris in France. And I at that time, I think this was my first ISIS. So I thought the awards thing, I knew I was being nominated. I was in the top three or whatever. So you get an email beforehand. Yeah. No one told me that. Yes. Okay. So they didn't tell me. But I thought that the Watts cup at the end of the conference. So the second day of the conference, I was like, hey, let's go to the loop. You know, what's the more police? And I get a phone call from Michael, my advisor. He's like, dude, where are you? We're having a watch lunch. I mean, what is it? It's Tuesday. So I rushed back to the hotel. I go into this big award lunch lunch on thing that they do. And then, you know, in time, so I'm there in time. I go to my sheet. I sit there and then they make everyone, I don't know why they still do that. But back then, the top three candidates were the Watts. They made them stand up in the room of 1500 people. So 500 people sitting eating the smog dashboard and three little kids stand up. And then one of the wins, which was gal, fair enough, right? Awesome. And then I very quietly sit down against. Hey, that's it. But what it, but what it meant. So when you're one of three, you had a great dissertation and for whatever disposition or taste or whatever political injury happens in these committees, they pick one over the others. That sort of happens with all these best paper and best whatever. Right. So that is true. That is true. Maybe one inside in the in the AIS best paper awards, because I'm in the, you know, the genius scholars pick that. And it's not very political. So what happens is they get out of the genius scholars to take about 20 or 30 of them, like volunteers. I'm usually one of them. And then they give me, let's say 10 papers to read, five to 10, all of which they check for conflicts of interest. So I wouldn't look at your paper or, you know, whatever Andrews or whatever. So I pick 10 papers. And they give you ranking scale. I can't remember like important significance, timing and stuff like this. You're supposed to rank them. And they do this with 40 people across all the papers. And then they just look at the scores. That's it. And a story, right? That's ridiculous. Well, it's at least not political. It's not fairly, but it's fairly unwise. If you look at what we did with AOM, it's a smaller group. We had like five people Academy management. And we had a whole, we had all of our networks generate. It's kind of like what we do with the podcast, but a little more formal, right? So of the five people, they had their networks. We generated a big set of papers. Then from those papers, we generated a smaller set where we're new. They were CTO members. So now we had that smaller set of papers than everyone read everyone. And you picked your favorite one in something you're a specialist in. And then your favorite one in an area that's not your specialty. So now each person now, we do, isn't it? Yeah. So now each person nominated to, and then we had a call where we discussed each one. So then we knew which ones everyone nominated for best paper. And then we would discuss them. And then we kind of came up with, okay, this is why I like this. This is why I don't like this. And whatever. And really we actually started somehow an advanced articles and advanced snuck in. And then we actually decided we're going to give the best paper to this articles in advance. And then I pulled it up. I'm like, guys, this is an article is an advanced. We can't give a best published paper for last year to a paper that is not published yet. Right? So that articles in advance thing is confusing. You're true. But on the other hand, like the way that the AIS runs it, or I think the AISQ does it as well, they pick a paper that by the time the award is out, it's been published for two years. So they always almost too, because it's in December and they pick the year that's already complete, which is the year before that. And I think that's not timely enough. Right? So I want to see what's the best paper this year, not, you know, whatever, what's online now. So I know what it'd be like, it's hard to find it. And it doesn't matter too much, I think. Like, who cares whether it's online, advance, positive. Well, we ended up not doing it. Right? We ended up picking one that we said, all right. We'll put that one in next year's. But we, yeah. So you brought up, you brought up qualitative meta analysis. You worked on your 2019 qualitative meta analysis in 2009. That's amazing. That's 11 years. I also, I didn't know that it was your PhD work. What I do know about this paper is when it came out, I think I came up this year and said, like, what's a qualitative meta analysis that doesn't exist? The meta analysis is statistical technique. And it took until this year, it's February, right? In February this year when I was in Georgia, you came down and you gave a PhD class on qualitative meta analysis. And I sat in and I did the reading. I was a really good student. And, you know, I learned, I learned about qualitative meta analysis. It's really cool. And to be honest, I've never heard about it. And it was so funny this session because at the beginning, so this is in the short form, how the session went. Okay. I'm going to do qualitative meta analysis. Record, you're here. Yeah. And qualitative meta analysis is bullshit. And then we talked for two hours. You wrote that on the on the whiteboard. Qualitative meta analysis bullshit. Yeah. Yeah. It records the pigeon bullshit. So we put that up. And then the idea was at the end of the session, will record still say it's bullshit. And you said, no, it's not. So you started with this bullshit. Very cool. Not some of that might have been theater. Hopefully you didn't actually think it was was bullshit at the time we were. No, no, no, no, but I see I seriously hadn't really heard of it. I've seen your paper, of course. But I really thought you just flipped the idea of a meta-analysis a little bit 'cause now I know that you actually took this methodology from the book, et cetera. So I really didn't know that that existed. - Yeah, my dissertation, I actually innovated it in my dissertation because, you know, I was doing it at enterprise systems and as you've mentioned many times, there are a million studies on enterprise systems. So I had a particular view, but, you know, you have to review the literature in such a way that, you know, your particular view clearly has a, but, you know what, a lot of the literature are these qualitative papers. And the problem with qualitative papers is that one will give you this really rich story and then they'll conclude with something about power, you know, it looks like manager versus people in worker power, you know, right? And then another paper will tell you a really similar story, but they'll conclude with something about whatever socio-materiality and ensembles and what, right? Then another paper will tell you a very similar story and give you some argument about, say, institutional alignment and misalignment, right? So the stories in the data, all these qualitative papers that are coming out interpretive, they're ethnographies, whatever, they're telling stories about ERP, but they're using different theoretical lenses. And I kind of was like, well, I don't wanna look at their theoretical lens, in a sense. I wanna see what their data is saying and I wanna compile this to see if things are agreeing. And this was one of the exercises during my dissertation. And I'm like, all right, how do I do this? And I was already starting to do the task when I discovered the Noblit and R.A.s book. It's 1988 and there, I think, educational researchers, or at least one of the two is, one might be a sociologist, the others, an educational researcher. And they were looking at schools and that sort of thing and looking at each school to see what works in that school. And they're seeing that there are a bunch of, and they were ethnographers, right? So to them, they call it a metathnography. And they're like, well, one ethnography, but we can compile all these ethnographies as data and we can draw our own conclusions. So that's what I used as kind of the language. And then, yeah, the idea is that in qualitative data, it like any data, any complex inductive situation, this is the under-determination problem. This is quine had this under-determination problem that, and what quine said was that any given data set in the world, you can draw an infinite number of equally acceptable conclusions, right? So that's what quine said, the famous philosopher, people since have said, well, they're not all equally valid, but you can draw more than one valid conclusion from the same data. And that's the case. So that's what qualitative researchers are doing. They're looking at cases that are often very similar to each other, but they're drawing much different valid conclusions. So whereas a typical lit review reviews their conclusions, we ignore their conclusions and look right at their data. Which is what quantitative meta-analysis does, too, right? We're not looking at their, we're looking at, yeah. You know, how we talked once about meta-analysis, we made the point that we don't see a lot of meta-analysis in IS because we don't have, you know, it's not a high paradigm field, right? So I know there's a meta-analysis of tamry search, there's one on trust, I believe, that we gave one of our awards to Sanzaza, you know, IT turnover into a professional turnover, attention stuff like this, but there's really just a handful. Now I'm thinking the qualitative meta-analysis, that's more something, right? 'Cause we do have many cases about, you know, terms digital transformation, for example, there's gazillion of them, there's one in healthcare, there's one in education, there's one in banking, there's a bunch in service, whatever, right? So this could be a really powerful tool for us to review our, you know, cumulative knowledge tradition a bit better, 'cause you know that I've been saying a lot, I think I said it last year, it would be good to say that it takes stock with 50 years old. What do we know? You know, like I am actually a fan, that's the paradox, I think. I don't like seeing literature reviews, but I'm actually a fan of literature, 'cause there's so value, they're so helpful. If someone just says, this is what we know, you know, this is what it means, and this is what we still need to know and stuff like this, right? So I do think that everyone hates seeing them as a reviewer editor, but I actually would like to see more than we have. - Well, and all right, so, you know, I've been pushing on this podcast, this new genre of computationally intensive theory construction. Well, what is that? You're taking a bunch of data, and you're generating a new interpretation of that bunch of data. So what do you do with a qualitative meta-analysis? In a given domain, a whole bunch of people have written qualitative case studies, you're generating an original kind of theory construction from that. So to me, it's the same thing, and it was really my first start in thinking about it. And just to go back to that paper, I just want to kind of close the loop because it's kind of a fun story. Andrew Burton Jones was the editor of the track at ISIS when I submitted that paper, and it went to ISIS, and he really liked it. And then he was a senior editor at MIS Quarterly many years later, and I was focused on getting the empirical stuff out of my dissertation, and usually you don't publish the review of the literature part. So I kept thinking one day I might publish it, whatever years went by. I got a couple papers published from my dissertation. Andrew was a senior editor at MIS Quarterly in like 2015 or something, and he said, Nick, I remember you had this ISIS paper, whatever happened to that. That was really interesting. - So he came to you? - He asked me, is that the greatest dude in the world? He asked me, he goes, "What ever happened with that?" And I go, "Do you want it at MIS Quarterly?" He goes, "Well, if you send it to me, I'll send it out to review in a way." So he actually brought it up. - I think that's what editors should be doing. We talked about this before, right? And Mashaq does that as well, I believe. She runs around like, "Send it to me, send it to me." I'm trying to do that as well, if when I see a really good one, I was like, "What are you doing with this paper?" Come on, send it to me. - Send it to me. Send it to me, right? So I think that's great. It's a great story. So everyone should be doing. So many years, so I published the decade later, I had to update everything. But here's an interesting thing. I was at, this is the problem with asking people to send it to you. Last week, I was at the kin, you know, the Samus school, you and Amsterdam, Marlene Huzman and Hans Barrens and that whole group, right? Wonderful summer school. It's an institution at this point in I.S., right? Every summer. - I agree. And I asked her too, I asked her, afterwards, I'm like, "Thank you for inviting me." And she looks at me and she's like, "I didn't invite you." What do you mean you didn't invite me? You just showed up? Yeah, no, she said, "You invited yourself." I'm like, "What?" And apparently at like a conference a year ago, or ALM, I'm like, "What am I gonna be invited?" So she felt bad and had to invite me, apparently. That is amazing. Because I'll use this podcast here, because I looked at the today, like because one of my students was there and one other girl that I work with. Anyway, and they, you know, talk, everyone talks very highly of the event, right? So some of my students have been there previously. It's fantastic. As you said, it's a really good institution in IAS, individual innovation. It's great. And I said that today, I was like, "Why am I never asked?" Yeah, ask Marlene. Tell Marlene you're doing it next year. I just invited myself to come and I'm said, "I'll take the bloody train." It's like 50 bucks, I'll be there. To do it. Yeah, so, but here I am as an editor at the place, literally everything that the young people were doing. I'm like, "Yes, and it's me and I'm as cute "so I'm gonna have a bunch of papers "and I might skew now, which I probably won't even get assigned to me." But tell me this. So whenever I go to some of these events, I usually go out and tell all these young, I say like, everything you're doing is amazing in comparison to what everyone was doing 20 years ago. Do you agree? This is cool topic. Absolutely. I saw yesterday as something, there's a new paper in IASR on technology and conspiracy, by, you know, like another young German swimmer, trying, I think. And I was like, "Man, what a great idea." Like, it's a great idea. It's a great paper, of course, and you know, it's in IASR, it's fantastic. Everything's fantastic. And it's great. It's great topics. They're really interesting. I literally looked at it, downloaded, read it, in the minute I saw it, 'cause it's so interesting. And that's, you know, I can't say the same thing about, oh my God, there's another tan paper in 2002. You know, that just wasn't as interesting. No, I think it's because we've moved from the IT function, which if you look at like the '90s, it was all about the kind of the IT function. And my work in enterprise systems was all about IT oriented. So, I mean, I thought it was organizational side, but it really, who's interested in enterprise systems, IT people, right? And then what's happened in the last decade? Well, there's the big data thing, which everyone's interested in. But it's AI, blockchain, you know, all these cool technologies that aren't necessarily the IT function anymore, but they're-- And they're also more interesting dependent variables, right? I mean, the dependent variable has changed. You know, only the technology is in it, dependent variable. It's not only value or return investment, or alignment in implementation or something like this. No, it's about a number of conspiracies. That's a great dependent variable. Or, you know, likely to fall for them or whatever, right? Or, you know, chances of political reorientation, you know, there's some really cool, very important dependent variable on the board. Much better, for example, than my hate feel like marketing, willingness to buy, you know, conversion rate. That's less fun. No, you know, it's important I get it, but I'm very happy that we have a wider array of dependent variables than they do. I really am. Yeah. All right, so let's go back to kind of reviews and there's that qualitative meta-analysis, which you kindly-- you like now. You don't think it's bullshit anymore. And it's a good way to look at a bunch of case studies without necessarily doing it. doing them yourself in an area. And I did one on enterprise system implementation. I did another one on escalation research, right? Where, so those are published. And I've actually been talking to people and perhaps I'll do another one at some point with some folks. But that's one way of taking what we know in the literature and starting to, and this is less about, and meta-analyses are less about what we know and what we found, more about taking the substance, right? And a quantitative meta-analysis, it's taking the data itself, right? And re-analysing it in your, right? Reconciling different measures with each other so that you can kind of aggregate across the findings, not their framing, not their conclusions, but you're actually taking their data. And you're reconciling the data with each other, and then you're drawing conclusions with essentially a higher sample than taking just one. So it's the same intuition, I think, with my formulation of a qualitative meta-analysis. There's some guys in J-A-I-S, and we can post the, it's Ryan Nelson and some other folks where they looked at a few different types of qualitative aggregation, right? You basically go from just doing a lit review all the way to like a meta- ethnography kind of approach. - Yeah, and you know, folks, the background to this is that a couple episodes back, we had JVB on, and one of the things he's known for, he's known for many things, but one of them is this idea of a structured systematic literature review. And it's enormously popular, but it also drew a lot of criticism for a way, or for becoming such a popular, what, like popular in a sense of many people doing it, way of reviewing the literature, following, you know, basically very simple clear process model, you know, filtering for abstract, and it drew a lot of criticism, and you and I, we have both criticises. I guess the point for that day was, not so much did discourage people from doing this, but also shown like there's many ways of reviewing the literature, and to me, it all depends on what you wanna do, like what's your objective here, what's your goal, why are you reviewing the literature, and then I'll tell you how to do it, right? So the structured literature review is one way, and to be very clear, the one thing I really hate about what people do with this approach is, they use it as a shortcut to avoid reading the goddamn paper, right? - And be direct. - So I read this, what the happened, like they're doing all this title, key, what abstract search, and then they're not reading, and they, and I think that is the fundamental mistake, so put it other way, if you're, you wanna review the literature, you have to read, there's no other way around it, right? So I don't know how else to do it, I can't have a tool to it for me, I can't have AI summarized it for me, 'cause, you know, like I need to read it, that's, if you don't like that, then you're in the wrong profession, I'm sorry, right? Is that, is that, is that a say? - I think you're right, I know you're right, that's absolutely, and here's my problem, so we were joking when JVB was on board. - That he started to do that? - To do that, obviously. - Yeah, because we keep getting these structured lit review papers. Now, I read his little conference paper, and I read the CAIS that he came up with afterwards, right? And I actually don't have much of a problem with what he wrote. I think the problem is, is with, well, there's, all right, there are two things, the first is that it's about reviewing the literature, all right? So let's think about what that means, that there is a literature, and now I wanna understand what it says. So who reviews the literature in this way? Well, it's someone who doesn't know the literature. It's a new PhD student, and I wanna capture a domain, so I'm going to use this approach to capture the knowledge in the domain. It's kind of anti, I don't know. It's either very high paradigm, so it's like we all have the same theory, we all have the same theory, right? And I'm just trying to figure out what everyone has done, right? So let's say it's a medical journal, and I want to aggregate our knowledge in a particular thing. So I wanna make sure I capture all the knowledge that's out there about that thing, all right? That's maybe the paradigm that you're drawing from, if you're gonna do an exhaustive search of the literature. The problem is, that's not what we do, right? We don't have a high paradigm. There's a phenomenon, and then there are different theories with the phenomenon, and there are different conclusions. The phenomenon is not one thing. Think about anything we talk about, whether it's AI, AI is just a type of IT. Blockchain is a type of database, right? These phenomena are not distinct. So if I'm gonna do a exhaustive lit review of blockchain, I'm not really doing an exhaustive lit review of blockchain, am I? Because if blockchain is an instance of the broader category of data, right, then I'd have to go back to whatever World War II, and since it's to talk about how we're going to, and then I would have to survey everything we know about data, and then distributed data, and then the instance of blockchain. And then it's like these phenomenon-driven lit reviews don't make sense that they're exhaustive. So you have to come up with kind of boundary conditions, and the boundary condition saying that it's the current phenomenon, it's like, let's say you're gonna do a structured one on digital twins or something. You'll end up with a very weak review of just what people saying about digital twins. You're not actually getting at the substance of what we know about digital technologies, representing and mirroring physical realities, right? To do that, you'd have to go look at representation theory. To do that, you have to go look at bungee, to go, right, so these phenomenon-based, yeah, bingo, whatever is there. So these phenomenon-based structured lit reviews, I think make no sense. Now if you're doing a theory-oriented structured lit review, like the one you did with representation theory, that makes sense, because now you're saying, here's a community, it's the IS community. Here's a theory, representation theory. Now how has it been used and what do we know? All right, to me that makes sense. But to say blockchain, now let's do a lit review, well, no, there's like a bunch of different fields. It's like, where do you try to get the line? I'll give you an ominous. I fully agree with you, and I'll give you another one. The other, again, and let's be very clear about this. I also don't think there's anything wrong with the East's paper or the shares paper. In fact, that's quite nice, both of them. Again, so the paper, how people apply it, I think that has a couple of issues. So one of the things in, one of the ways in which it is being applied, either one of the two paper, is as a justification for people to do a search across the basket of eight or 10 or 11 papers. So they sort of like, they refer to the paper, which actually doesn't say that, but they say like, look, based on this, we follow this methodology and we'll limit our search to the basket of eight, 'cause that's IS. Well, first of all, it's not. That's not the IS field, this is just a subset. Second, IS, our field doesn't have boundaries. Like we are by definition cross-sectional. So if you do anything in our field, everything people do in business is relevant. In fact, anything people do in social sciences relevant and everything that people do in relation to IT. All the engineering stuff is relevant, all the computer science stuff is relevant, all the stuff engineering stuff is relevant. So if you're not doing this, you're literally not covering what has been done. It's ridiculous, right? So AI, there's a gazillion research on AI from within computer science, all the way to sociology, philosophy everywhere. So what point is there of saying, like I look at the European journal, the JIT, and the JIT says, it's just, no, you're not getting anywhere. Right? Your job is not to write a paper about what our six journals say or don't say. Your job is to create knowledge and to understand that knowledge we already have. Right? So it's not a good idea. It's wrong. I agree 100%. And that's the issue of this structured lit review. Right? You come up with a word that you like, digital twin blockchain, pick the phenomenon, and then you do a literature review of that word in IS. So that's the first thing you do. So I think it's just premised on the wrong idea. But then the second thing is, you then, what do you say? You critique IS and you say, look, these groups, these folks look at this, these folks look at this and they should be looking, and then what's your big conclusion? Well, the literature should be looking at other things or it needs more work in certain areas. It's like, that's not a contribution. That's just the very beginning of starting to do research on digital twins or blockchain, would be doing a lit review, thinking about what literature needs, then go freaking do it. Right? Don't write a paper to us telling us what it needs. So that's the thing that I get sometimes, especially at conferences, is this critical structured lit review. It's like, oh, now I'm going to criticize the literature. So not only did I come up with some small subset, phenomenon oriented, supposedly exhaustive of just IS, but now what I'm going to do is criticize it, which usually means pointing out things it doesn't do. It doesn't do this, it doesn't do this, and it's like, oh, so, and then they cite JBB. Now I think in part, it might be JBB's fault. And if you go look at, and I'm looking at the CIS paper, an under method perspective, what he does is says there are two kinds of reviews. There's the narrative review and the systematic review. And I guess what I'm basically, and he says the narrative review is sometimes called conventional literature, right? So I think this is the problem. And you tell me if I'm right, I think the only kind of lit review we should be doing in IS is narrative review. And if we're going to do a structured lit review, the only time we should do it is if theory. Not phenomenon oriented structured lit review, but theory oriented structured. I like that. Because in a way, that's what you said about the, you know, you mentioned our representation reviews. And I mentioned this a couple of times on air and a bunch of episodes. In fact, there's multiple reviews, right? We didn't write a review. We wrote two. And there's a third one, again, using a different method. So, in fact, on the same body, there's a narrative review. There is a meta-analysis, and there is a, you know, a frame, or an integrative review. We reinterpret the literature, right? In terms of, you know, it's a success of failure. That's the framing we use. So we reinterpreted it. We did straight-up narrative review. And a, a, a, a, a, a, a, a straight-up statistical meta-analysis. One of the slightly different subsets. The big difference, and here's the big difference, is really on the ground how we did it. Now, we started with a theory, and the theory was published in a paper. And in fact, it was published in 1987 or something, right? So what we did, our starting point, we went to the 1987 paper on Google Scholar and said, like, "Tell me all the citing papers." And I can't remember, but I think it was at that time, it was about 1100. Right? And we crossed, there's basically three main early papers, late 80s. And we crossed, there were about 800 also papers citing these three papers. Right. We downloaded every single one of them. I think five of them were in French or something, so we didn't read them. But all the other 795, we read. And we know that is the literature that deals with the theory, because all of them refer to this, you know, to this to the main bodies. But half of them were just superficial references. Yeah, of course, right. But we read them to figure that out. So first you had to read them, and then, you know, we dropped the, depending on the type of review we did, we dropped about a half of them because there were, you know, sort of like in the conclusion, you said, "Oh, by the way, there's also representation theory, side here." Right. And then we read them with substantive citations, but you know, the only way to figure that out, the point was that we started with every single paper that we could reasonably expect to actually work with this theory, and then we read all of them. Took a year. It took a year. How far, you know, like even for fast readers, it's 800 papers. I still have a gigantic data. And by the way, also another common mistake. No, I didn't give it to a student to read. Like, what like a system is something? Like, how the hell is that going to help you do your literature review? You need to know that literature. Not here, you know, in the end of the day, you're student of literature, but you don't. What the hell? That's not how it works. Yeah. That's some of these things you have to do yourself. Yep. So I think that that's the problem is we need a narrative understanding of phenomena, but then we do this structured lit review of phenomena, which is like, now with that said, I'm thinking of like digital transformation. What if you did a review of everything we know about digital transformation? Well, first of all, that would be huge, but it is a phenomenon in a sense. And I think we do need some, someone at some point to give us a summary of what we actually know. Like, I want to review what we already know about digital transformation. Let's say you want to write that paper. How do you do it? How do you, uh, first of all, I would, the first thing is I would try to do this qualitative meta analysis. I would try to find, I'm pretty sure you could find a hundred, two hundred, reasonably good cases, rich cases where you have enough data in there that you could do this qualitative meta analysis. If you just explain to me, forget their conclusion interpretation strike, go straight back to the data and then re-earned it. The other one you could do, which is now what I'm actually trying to do, not on digital transformation, but is what people call an integrated review. The integrated review is the type of review you see on AOM annals. You know, the literature review journal that they have AOM annals. The types of reviews you publish, they're not structured, they're not narrative, they're not meta analysis, they're, they're integrative. And what that means is they basically say like these are the things that are being discussed by different conversions. And I'm integrating all of this to do one or two things. Either I'm trying to do an abduc, abducation, you know, I'm trying to create a verdict, you know, or I'm trying to reorient. So I'm thinking of some of the annals papers. I know Paulie and Ernie has one. Yeah, a bunch of these, they're all beautiful, but that's the difference. There's a structure going on there, but I think it's a narrative review. It's what they're doing. They tell a story in every one of these papers. It's like his paper is like here's the way that we thought about it. Here, then there's this other way of thinking about it. Then there's a, and he builds it up like a, and then there's this, I don't know. Well, you can call it a type of narrative review, I think. There's a very nice, a very recent editorial, I guess, or a method paper on this type of review that the one that it's written by the editors of the AOMN. And it's Matthew Cronin and Elizabeth George, which is cool, because it's one of my favorite book authors, Elizabeth George, it's an American writing English crime novels. But I don't think it's her. Anyway, so that's a very nice paper where they talk about this type of review. And I think the difference to a narrative review, it's actually more conversation based. It's about who are my conversions, which are the communities that are formed? And how do they talk about this broad phenomenon, say digital transformation? So the integrated review goes beyond paradigms, metaphors, schools of thoughts, you know, and just tries to integrate all of them together. It's organizing. So it's, I guess that's the key that I would take away from what you're saying. And the reviews that I like are organizing based on conversations. It's not as if there's like just a table saying they say this, they say this, they say, it's more like, all right, here's a conversation. And this is what the conclusions are from this conversation. Here's another conversation that emphasized different aspects of it. Here are the conclusions from that other conversation. And this is really nice because what it does, this type of review, you can only do, first of all, if you really know the conversations and if you really know what individual scholars say about a particular phenomenon. Second, the level, the scope of these reviews is big. You know, you mentioned Paul Yanadi, Emma vast. They did this integrated review of social media. Like, there's a lot of stuff that people talked about in social media, right? And they integrate all of that. There's of course, Wanda Olicavsky, very famous social media reality. That's another one of these. Like technology and organizing, like how broad can you possibly be, right? So if you have a topic like this, then and if you really know the literature, like you really know the, the, the, the points and the arguments that individual scholars make, then I think you can do it integrated. And you know why? Here's why, record, because it's not a list. The moment whatever you're doing turns into a laundry list. You're probably doing the wrong thing. If you're telling an integrative story based on tons of knowledge, now you have the laundry list on your computer, right? There's clearly a list somewhere of a bunch of papers. And then you've organized them in some way. But the thing you write is a story, right? If what you're publishing starts looking like a laundry list, I think that's a problem. And I guess that's my issue with a lot of these structured lit reviews. The basis upon which they use inclusion criteria, right? And the, that they do their, their review is a problem. And then they have these laundry lists. And then they're critical. They're like, and then their conclusion is the gap, which is awful. It's like, oh, and we're not doing this other thing that we should be doing. Well, why don't you go do it? You know? Yeah. Just two more examples of this. Right? So again, integrate review. Nick, Nicholas Foss, famous guy, he wrote integrated review on microphoneation. That's a huge topic area, right? There's another one on, you know, like undenemy capabilities. Yeah, it's a, it's a, it's a huge conversation. And this particular case, a theoretical conversation where lots of people say different, lots of things. And the key point here is the same with a lot of list argument is not every paper, not every viewpoint, not every argument is equal. And you got to work that out, right? And again, you can only do this if you already master the field. I don't think you can do a, this type of review is like, I'm going to do it and now I start reading. No, you can do, you could write such a paper after you've been working this field. And you know what the conversation have been for the last 10, 20 years or something like that, right? Again, it's not a, it's probably good reason why there's old senior scholars writing these types of reads. It's very difficult for PGD students to just to have that handle of these conversations. And it's more than just knowing that there's 27 papers on blockchain, you know. And it's, it's funny. You need, here's, here's one thing that I think a lot of, especially younger researchers don't realize. They think we're in a game where what you can do is knock out a bunch of papers and you can have a superficial understanding of a few different things and continue to knock out these papers and just keep your superficial understanding. And the problem is that works for some people for some time, maybe. But that's not the way to be a scholar. That's not the way to have a career. Like I see what we're doing. The point of this profession is not to write papers. The point of this profession is to create knowledge. Well, and I see it with you. You know, I never really paid much attention to representation theory. We're talking about that one. And then we brought in, we, we are working with you now with some colleagues, you and I are in a research project. And you're our representation theory, which is becoming kind of the main theory of our paper. And you're the expert. You don't need, like I read this stuff. I'm not sure. You know this stuff inside and out. You know it's history. You know where you came from. It's like you are, there's a handful of people in IS who are experts. Like they know this shit about representation theory fundamentally. Right. I would say that's what you want to be. If you're doing this research endeavor, it's not about doing a structured lit review, dismissing something, then moving on to your next structured lit review. review. And oh, if you happen to do the first structured lip review in I don't know, some good journal, but maybe not the elite journals, but it's the first one on a particular topic. Then you get a million citations, even if you do a bad job. So it's like then people are racing to do the first structured lit review. This is that clickbait stuff, right? So so that they can get their million citations. That's not your goal as a scholar. Your goal as a scholar is to know something and be one of the touches like you with representation theory. There's probably five, maybe six people. Yeah, Jeff, Roman, Andrew, Ron, of course, that's about it. And maybe my son. In our little corner of the world, which is information systems research, there's whatever 4,000 researchers of those 4,000 researchers. There are five legit experts, maybe eight. Maybe there's some junior ones you're not paying attention to. And there's like eight, maybe there's 10 legit experts who know representation theory fundamentally. I think it's your job as a scholar. When you're writing a dissertation, when you're taking your dissertation and you're writing papers from it, to be one of those five to 10 people in your niche of the world. And if you're not, you're probably not going to achieve at the level of Yarn record, right? You're not going to be in the top journals. You're not going to have a career where you have multiple papers in a stream. You know, what you'll be is one of these people who, you know, mishmashes every paper is new. You're learning the literature brand new for each paper and your, you know, my institutional theory stuff. That's what I presented in the Amsterdam. It's what it's still relevant to this day. It was relevant when I did my dissertation. It's going to be relevant when you and I retire. And of course, everything you said about me is the same for you, right? So I know you with you institutional theory. And you know, we also work on a project where you are the expert bringing in for this. And you know this too, right? And again, how many of these people are there? Maybe 10, you know, like the, the, the, the, the, it gets very, it gets very lonely there. So I want to, I want to talk just a few helpful hits for people that get started, right? It's easy for us to sit in a cell like, hey, you can't do that as a student and you shouldn't do a structural interview either because it's bad. Do do do do. So what can you do? So what I, one of the paper, I truly love and I go back to it all the time. It's actually a, a, a, a review, a paper about how to do literature reviews by Gipare and others. Yeah, Gipare from a, a GZ in Montreal, he's done a few papers on how to do literature reviews. I think he handled the review section of JS for a while. Anyway, so he has a, he has a paper in I&M of all places called the typology of literary reviews. Oh, cool. And I don't even know this paper. It's fantastic people because what it is in the middle of this paper, you only have to go to one page where it's a table. And it says like, he has, he's 10 types of literature reviews. Here's the goal. Here's the type of review. Here's what you need to do. It's fantastic, right? It, I'll use it all the time when I talk with students, it's like, okay, you want to read it? All right, what are we trying to do? What is our goal? Okay, well, then we need to do a scoping or narrative or a theoretical or critical or meta analysis, whatever. Or a scoping review. That's the type. And then these are the guidelines that pertain to this type. It's really useful just to understand the different types. Yeah. And I think that of these, I'm just looking at that table. First of all, I'm going to read this and this sounds awesome, but I'll tell you this, I guess, and this is what I would do is, is my bias as someone who has to deal with people doing reviews. I'd be like, if you're going to send me a descriptive review, you have to point, right? You're, I'm going to reject your descriptive review. I'm going to reject your critical review unless you're saying something, right? I'm going to, so I'm going to reject probably half of these reviews. Of course, of course, because the goals are not all equal, right? They're not, right? So, yeah, he talks about in this table that we're looking at right now. He has four goals, summarized literature, aggregate or integrate the literature, built new explanations or critically assess the literature. I would completely agree with you, but this is, this is from your vantage point of what gets published in the quarterly. This is, you know, that may or may not be a goal, right? So it's still, of course, for some, we need to, like, I just published a scoping review last year with my student in her first year, we scoped the literature and AI development. What do we know and what are we not now? It's simple about developing AI system as opposed to standard information system, AI development, oversized development, both scoping review. Well, we don't, we don't even want to submit that to the quarterly. No, we still did it. Like, we, you know, we, we wanted to do this and was a good point to summarize that. And because we were doing it anyway, we wrote a little paper about it, but no, that's not going to appear in, in Isar or something, right? Yeah. And I'm being a little critical because if you're a second year PhD student, you pick an area of your dissertation, understanding the literature in a structured way, makes sense. And if you write it up, that, excuse me. And that makes sense. And then if you send it to a conference, it'll get you in a conference, you present, you won't get in an MI quarterly. That's like the basis upon which you're going to do your real work that will get you an MI quarterly. But you can send it to another journal and absolutely. Right. So I'm being too negative, aren't I? You're not negative, but you're only, you know, you're being an American. Everything that's not in these two journals doesn't exist in your world. The other paper that I actually think is kind of helpful is Suzanne Rivar. She was the theory and reviews editor for what, you know, people in the day, Suzanne Rivar, she handled the theory review section for the quarterly irrespective of what the theory and review was about. So she handled all the theory and review paper. They changed that now, you know, if it's about institutional theory and it's a theory and review, it goes to Nick, you know, it represents, it would go to me. So now every editor is about a topic area could also get a theory review. But back in the day, it was different. So our theory and review paper, my first was handled by Suzanne because she was the handing editor. Anyway, so at some stage, she wrote an editorial. It's called the Ion's, the Ion's of theory construction. Where she talks about what's the difference between a literature view and a theory generating a review like a theory paper, a theory development paper, right? Because the section the quarterly is misleading. They don't want reviews. What they want is new theories, right? So and that's helpful for people just to find two, three, four characters to understand what is the difference here? Like, you can't, you know, you can't do a summarizing scoping review and send it to the section. That's not what they're looking for. What they're looking for. You have to link to our podcast with Dorothy Leidener from last year, right? Because she had that whatever, what the literature says, what they found and what it means. So she had a distinction. So if people are interested in doing strong lit reviews, but even what you just said quarterly, if they're doing institutional theory review, they'll send it to me. They're doing a representation theory review that sent it to you. What if they're doing a, I don't know, digital transformation review that sent it to either of us, right? They're always sent it to you. He has written theoretical papers as well. I know, is that I have to go to this other thing and then I'm reading that. So I'm actually reading a lot when I'm writing. No one gets to see that. But that's also why when I sent my draft, my draft version, usually after every sentence, you see the two or three references, because that's the outcome of my reading process. So I'm reading a lot only when I'm writing. I don't sit down like you, you always say that you read first because you don't know what to write. I write first, 'cause I don't know what to read. And that's a different. So one comes and one comes from expertise and one comes from learning as you're going, right? Of course, in representation theory, I don't. I know what the, like I just write and then put in the reference, 'cause I know them by heart, but another I don't. So this is, so for example, that paper when I'm doing the institutional section, I might just leave placeholders because I'm not sure, because here's the deal. You make a particular claim. There are probably five eight papers that you could potentially put after the claim, but you don't want to use different papers for each claim. You need to really pick the conversation. So you just have placeholders while you tell your story. Once you figure out where your story is and where those critical, you know, not just they said something, but it's a key point of their paper and you have that later in your paper, you can bring those up towards the front, right? So, so yeah, I think if you, if you have a mastery of the literature, the references don't matter as much. You know you'll stick a reference here and there that is the appropriate reference for your paper. But yeah, for me, it's all about. reading and it's all about understanding what you want to say and how you want to say it and how you're going to use the literature to make your point. So this is one thing and this is the Watson and Weber or Weber and Watson, Webster and Watson. And they had this distinction and this was a very important distinction for me when you're doing the lit review of a paper, the distinction and I forget what it was but it was two types of they said you're reading and having a whole centric and concept centric wasn't it? Yes, that's it. A paper centric and concept centric. Yeah, exactly. So paper centric has this view of okay, now I'm going to report to you from the literature. So if like I'm doing an experiment in psychology and it's a high paradigm thing and it's like all right, this one, I found this or this one found this. Now this is what I found, right? That's like medicine. That's like psychology, right? It's those kind of experimentalist hypothesis, incremental, well-defined, high paradigm space. So then you can have a paper centric review. For us, or at least the genres that I generally play in, which is theory heavy, that doesn't help much very often. Paper centric reviews. Instead, you have to, it's all about rhetorical moves and how you're positioning yourself with respect to authors. For every paper centric saying, you know, record at all said such and such, you need to set that up with 12 conceptually oriented sentences that have the, you know, pulling an argument together. So I think that idea of conceptually structured lit reviews is, I don't know, to me the key of setting up a list. It is the key in there. You can have like two or three author centric. Exactly. And let me make that very practice. Because again, this is one of the things we did in our theme review, the 2017 representation review. Like people think of it as normally when people do a literature review, they build an excel spreadsheet and every row is a paper, right? So it's like, I'll burn it all blah blah blah. And then the columns are the things that they code for or whatever. Now the concept centric flips that. Yeah. So you have on the rows of concepts and the columns are maybe papers, right? And in our review paper, you actually see this in the penics, I think, if I remember correctly, we have the actual concept matrix is that we build from these, you know, got no so many papers there were. And they organize by concept. Yeah, the rows are concepts. The rows are not papers. And then in the columns, we have a bunch of papers that deal with their particular concept, right? So you know, think about the excel table and flip it by 90 degrees. That's the difference. And my argument is if you want to write a compelling interesting argument, it's understand those concepts fundamentally, understand the relevant relationships those concepts have to other concepts. And then you have the front into a literature, a contribution to literature. If it's all about what people are doing, then you're not disciplined in a way that will make it obvious what your contribution is. And the only doable way, which is something you said earlier, and the only way to get at this really is if you pick a topic area, say a theory of phenomena and stick with it. Right? Right? Because although it just gets too much work, you can't do this for 20 different things. Right? Well, and for me, I have to say it's theory. And I know I always err on the side of theory and I know I have a bias, but I don't know too many people who have really made a career and we respect them. And I guess I'm putting my scholarly blinders on, because I guess if you go to computer science and other places, there are people who pick a technology and made an entire career about the technology. Right? And there are a few of those in IS as well. But to me, at least in the top journals and information systems and an organizational side, which is where my, the right to theory, the conceptual understanding of the domain and similar domains is what I think enables people to just be hugely productive in their careers. I agree. And you're a shining example of like a beacon of that, of course. And if I compare myself to you, I think the one big difference a bit and big move I had is you mentioned representation theory, sure, but it's actually, you know, like I'm not actually doing a lot with that. Some of the papers have come out recently because they take years. So because I did shift and I shifted about 10 years ago to this, you know, degeneration architectural, you know, young gens, senes and clark, this kind of audio. And that is now my go to theory. So I feel almost as comfortable with this entire architectural view of degeneration and digital material and all that stuff as I have felt with representation. But if you think about it, that's a 20 year career or 18 years. And it's two. And they're both theories, by the way. So it's the same thing. So these are the two areas. And I don't have a third. I don't think I don't think anyone looking at me would say I would have a 30, right? So you can't do 10 of those. I don't know how you could. And that's actually our overlap because modularity theory is something I've always been fascinated by. And that's how I'm backing into architecture stuff. So the stuff I'm doing, I'm not really sure. And yeah, yeah. All right, dude. This is the end of our season. It is. It's the seventh. We finished seven seasons, 80 plus episodes. And you know, I looked up the stats. We're still growing. I find it amazing. It's not exponentially. You know, like I don't think we get a series D backing from an investment or anything like that. But it's still growing and people keep on coming back. So 4000 rough people generally say the information systems field is around 4000 people worldwide. Right. So and you told me at one point when I asked you about how many are listening, it's around the order of three or four thousand each episode. Right. So and some episodes more. Well, it's pretty constant. I think, yeah, every year we have a favorite episode. I think the big one from last year was the one with Paul Anati. Oh, was it this year? Anyway, so we've had so many of them. But I wanted to use you, but I just wanted to say this year was probably Jason Thatcher. You showed me that that we got like a 40% Jason Thatcher bump didn't we? So we got a little bump when he when he announced it on LinkedIn, that might just be reached dude. Yeah. See how many, you know, like in the long one, whether you get listed to more, there is there's some patterns in terms of what, you know, which people, people love when we talk about how to write papers, you know, like these are the click baity ones. Maybe it's the title. Hey, I wanted to say thanks. Thanks to all the listeners, man. I mean, we it's it's great. We keep on doing this because people listen to it. If people stop listening, we'd stop doing it. I mean, I think that's a fact and just shout out to, you know, everyone that is apparently still excited for this, you know, at the 490 Wednesday when a new episode comes out. And it's it's been good. Look, look at Nick. Nick's been coming to Europe every three weeks because people keep inviting him. And I refuse to believe it's for scholarship. I think it's all because of famous podcast host. Of course it is. And then I go to these places and people bring up references from our podcast, like obscure things we say to each other and then someone brings them up to me. Oh, well, all right. And then the final shout out, I just because I'm a nerd. Last night, I came home and I saw there's a new issue of the Scandinavian Journal of IS. And they have this beautiful special issue in honor of all a Hanseth. All a Hanseth is known for is an infrastructure guy. I already wrote to him key infrastructure papers. And this is a beautiful special issue if you interested in the history of the field, especially the European history because they take the opportunity go back to and talk about the Scandinavian tradition. There's a great paper by Kale about the Scandinavianized tradition by all of himself. Eric Monteiro has a piece in there. Johnny Janis has a piece. It's a wonderful special issue. There's probably intellectual merit there too. But really, if you're interested in our field, have a look at this. We'll put a link in the show notes and just learn more about where we've come from. And again, thanks to those people that organized, I think Tina was involved. And you know, like people care and people do stuff. There's also an AI historian, AI's history website where I looked up you being an award winner and you could look up all the award winners to anyway. So there's people doing a lot of service, lots of cool stuff. And if people really want to make this profession their own and spend their life in a profession, carry in it, big thanks to all the people that curate and do these things. Yeah, just pretty, I'm going to check it out. All right, my brother, we have fun in the South of France and you're a little bit. Are you going to take some time off? Are you just going to be an American in work? Yeah, as long as you get a tumor or something? I always take time off, but it's just a couple days here and there. I don't do these like two weeks of vacate, right? I have a three day threshold for vacationing. So yeah, three days, the threshold for vacation. So you and I, we're going to meet in Chicago for AOM. That's the next time we talk. And then after that, we'll get back to doing the podcast. And we already have a couple of guests lined up for the the wintersies, don't we? Yeah, we do. All right, well, we'll let people know what that is when we come back. Absolutely. All right, Nick, it was a pleasure and an honor as usual. Take care, folks. All right, my brother. See you.