Why do government projects fail?

Welcome to the APM Podcast. APM is the charter body for the project profession. My name is Emma DeVeter. I'm the editor of Project APM's Co-3 Journal and Your Host. In this podcast, I'm speaking to Jonathan Simcock, a highly experienced and respected project leader in the UK, who has held executive roles in oil and gas, energy, utilities and telecommunication sectors. He has been involved in the largest UK government project since 2007 when he took over what became the Infrastructure and Projects Authority and is now the National Infrastructure and Service Transformation Authority or NISTA. Until last year, he was chair with a submarine delivery agency in the MOD. His book The Delivery Gap, Why Government Projects really fail and what can be done about it, comes out later this month and is available for pre-order now. When the UK is facing low growth, poor productivity and increasingly insecure world and climate change, closing the delivery gap is an imperative as important as any of the other challenges of statesmanship he argues. So we at the APM podcast could not wait to get into the studio and ask him what he's learned about delivering government projects, whether they're all doomed to fail and what it takes to deliver some of the largest and most complex projects successfully. Welcome, Jonathan. Thanks for joining us at the APM podcast. How are you? I'm very well. Thank you ever so much for inviting me. No, it's a pleasure. I would like to begin by asking you briefly about your career and what you're doing now. So I spent the first two-thirds of my career probably in oil and gas. We didn't always call what I was doing projects in those days, but they were really. Big maintenance project, we would call them now capital projects. I did quite a lot of mergers and acquisitions and then I did quite a lot of transformation, internal transformation in non-aggressive businesses, but then relevant to this conversation in 2007 by a series of accidents I ended up as the major projects director in something was called the Officer Government Commerce, which as you say, turned into the major projects authority then the infrastructure projects authority and has now been merged into the new body. Mr, I have to say in 2007 I was spectacularly underqualified for that job. I think I got it because no civil servant at the right grade had applied, but since then I've been leading, governing, advising, assuring on great big government projects, one way or another. As you say, I was chair of the submarine delivery authority until last year, and earlier this year I stepped away from the board of Salafiore to nuclear sight in in the northwest. What I'm doing now, I've spending quite a lot of time promoting the delivery gap, and meanwhile waiting for the next challenge probably in an advisory or a non-executive role. Let's talk about your book, so it's called the delivery gap. What do you mean by that? What is the delivery gap? Why did you want to write the book now? I think that the level of public debate about great big projects is terribly shallow, very superficial in the media. If you listen as I do to political podcasts or read political books, then the reason we have problems in big projects, it's all to do is, oh, because we only have some decent project managers, or if only politicians would definitely have read some decent politicians, or if we sorted out the planning system, there is some truth of course in all of these things, but they betray a superficial understanding of what is the challenges that really go on in great big projects. So I had in mind to write a book of explaining, you know, why is this so hard? But my first working title was not the delivery gap, it was how hard can it be? But then in the wake of the HS2 failure, I'm going to call it failure, I realized that in my heart, I'm sometimes for these great big projects expecting them to fail, and I'll come back to what I mean by failure in a minute, and that felt really uncomfortable, particularly in the context that you mentioned, which is, you know, we're in a country with low growth, with poor productivity, with a desperate need for infrastructure investment, with growing defense threats, we need projects. So to start, to have this sort of mindset that of course with these things never work, that's really unhealthy. So I wanted to get under the skin of what really is going on here, and I wanted it to be an insiders book, you know, there are good project management books from academia, there are some quite good public sector project books from sort of journalistic view, but I wanted to spend time, and I have spent time talking with the ministers, the axel ministers, typically senior officials, the delivery leaders, the civil servants, the supply chain to get a more of an insider's view about what really is going on here, and I think my conclusion out of all of that is good people are struggling in a system that's failing, and it's to do with the incentive structures that drive tendencies and behaviors that lead to failure, and if we only think about improving the process and sorting out the planning system and so on, then when we're not going to get to groups with the root causes here, the book is structured with seven case studies at the beginning, which case studies see a cover in the book, so I cover Crossrail, the Elizabeth line, I cover the Palace of Westminster Restoration and Renewal project, I cover the Universal Credit project, completely different kind of project, I cover the smart meter implementation program, which I'm definitely one of the guilty, I cover the huge project for privatising the management of all of our nuclear legacy sites, including Sellafield, where I was on the board, I cover the National Programme for IT and the Health Service, massive top-down rewiring of the Health Service, and I cover the program for the student tax submarines, so you'll notice from these that I'm talking about huge projects, most of these projects are 10 billion plus, and that's really the light I'm trying to shine is on that kind of huge endeavour, so that's what motivated me to write the book, you asked me what I mean by the delivery gap, success and failure of projects is not a simple subject, actually it's worth the conversation and it's alright, but in context of this book what I really mean is as simple as did we do what we said we were going to do, on the day we said we were going to do it for the amount of money that we promised, did we do what it says on the 10, a successful project, or can I beat its promise? Well if that promise gets redefined over many many years, so I think if the promise is redefined, so if we're halfway through something and say look we don't really want a blue one, we've decided we now want a red one, and the project says well okay if you want a red one that's fine but it's going to cost two billion more and arrive five years later and everyone says yes we still want to do it, that can still be a successful project, however as you know as well as I do Emma, any experience project professional will tell you that the best way to kill a project is you change your mind halfway through about what you're trying to achieve, so the worst thing a project leader can do is to not face up to the realities of what change means, not force if I can use that word, the stakeholders, the sponsors, the funders of the project, to the consequences of the change they want to make and fudge it to pretend that while we can sort of still do a blue one and it might be a little bit more expensive but it probably won't, because that's what everybody wants to hear, so there's something really important about professional confidence and candor. So actually you're describing some behaviors there that I wanted to circle background to because you refer to incentives and behaviors and perhaps the wrong behaviors are being encouraged and perhaps not having candor is one of those behaviors. So could you tell us a bit more about that? I spoke to a leader of a project that everybody almost everybody listen to this project will have will have heard of. Who would describe the systemic pressure on them to under cook their estimates? Now he was absolutely clear that he wasn't going to do that. But he was also honest enough to say you shouldn't underestimate the ability to sort of get yourself to believe what you need to believe in order to get to where you want to get to. And and so the the example that he's using I'm keeping it very anonymous here. Well, you know, I'm going to say that it's, you know, X billion and X years. And I know in my I know that if I said it was X plus four billion and years plus three that I wouldn't get next year's funding. And without actually funding out of no project. So he's so he's not lying, but those pressures exist in the system and it's something else described to me. This tendency is limbo dancing that that's kind of what we do in in developing a big government projects. Why do these pressures exist in the first place and is it because of or estimation lack of clarity over the purpose of the project political desires and ambition. And but compounded with this kind of miscommunication and lack of trust because people don't stick around long enough. What would stop this happening in the first place? All of the above all of the above. So if if a strong minister. So I'm now thinking about the case of universal credit, a strong minister with a very strong vision. Determined to overcome any barrier to get going and to make this project survive and to get it to get it going. And nobody in the system was strong enough to stand up to that. Not by saying, well, you know, your project plan says this, well, my project plan says that because it was too early for a project plan. There is, you haven't got a project yet. I don't know how we're going to do it. And the incentives in the system don't encourage anybody to stand up. What can we, what can we do about these things? When I started writing, after a little while, I felt I'm not really, I'm not really satisfied with this. And I realized that what was happening was I spent so many years writing gateway reviews or structured advice for senior officials or for ministers that I was cramping myself into a sort of a style of writing, which where I was continually hedging and thinking, well, how will that read to a minister? And well, how realistic will that be in the political context into which I'm delivering it? And could I rephrase this in a way, which makes it gentler? And I had a conversation with a literary agent in the process of trying to get a publisher. And he said, that's really not what you're trying to do here. You're not trying to say, you're not trying to produce a sort of a set of simple recommendations which are bound to be easily applicable in the environment you're talking to. You're trying to advance the debate. You're trying to get people thinking about things in a different way. And suddenly I found myself much freer to think about things which, yeah, I wouldn't look particularly easy if they landed on the desk in Mr. the day after tomorrow, but I came into that conversation because you asked me, what can we do about this? One of the things I feel more and more strongly about is that the players in this game with the probable exception of the politicians, there should be, we should be subject to a kind of a duty of candor. Now, it shouldn't be okay to allow a business case to go forward with estimates that are undercooked. Where we want sort of those, they're undercooked, but this is what we need in order to get the project going. And this is the criticism of individual project people, let's say it is to do with the system. And so one of the things I think we need is rather than the mountain of project assurance, which any big project in government gets drowned by, I think there should be something that is genuinely independent. That the professional knows is going to be exposed, but to people to a small group of experienced people who can hold them to account. That doesn't exist at the moment. The National Audit Office is a wonderful institution and full of wisdom that nobody particularly takes much notice of. But it's auditing, it's backward looking. I'm talking about something that would feel a bit more like a kind of project equivalent of the Office of Budget Responsibility. You know, I'm constantly, everybody will hate this idea, but do you see what I mean? Is that not part of Mr's role? I'm sure it is. And you know, you can follow that thread right back to my time in, in number on horse cars road in the Treasury. But this does not independent. You know, everybody involved in that is that decision making works for the Prime Minister. No, it's, so it's, it's, it's, yeah, it's just not independent. Yeah, completely outside of that. And what were you going to say about Crossroad? I think I don't know if I mentioned this earlier on, but everyone I guess remembers is that for first of all, just to be completely clear, Crossroad is a fantastic piece of infrastructure. You know, one that then should be proud of and the people that belt it should be proud of. And it's, you know, delivering above it, above it's volumes and it's, and it's passionate and others and so on. And the, and the early phases of the Crossroad project went really well. The simple phases were a, were a miracle. However, in August 2018, the project was still saying that it was going to open the central section of the, of the railway in December 2018. Four form in four months time. And at that point, the underlying problems became undeniable and surfaced publicly. And the, the overall railway was not fully opened until 2023. Billions of pounds more spent and years more consumed. And yes, it's a wonderful railway, but that is not okay. So the people leading that program, some of them knew but weren't prepared to say. And some of them, you know, had managed to convince themselves that they, that they didn't know that is, that is not all right. That's the kind of, that's the kind of behaviors, if you like, the tendencies, which we've got to, we've got to overcome. I use this as an extreme example, but, but just to be clear, this is absolutely not a unique situation. I have sits at around so many tables where everybody kind of knows that the schedule that they're looking at is not a realistic one. And then, and then the project leaders will look at each other and say, well, you know, if we rebase line now and say it's going to take another year. Then in a year, we'll have lost all of that float and would be back to say to rebase line it again. There's this fear of facing reality. Toscan obvious question and the fear of facing that reality is the fear of what, what are their consequences they're seeking to avoid. There is a genuine concern around project leadership tables in a situation where the schedule is under real pressure and where its likelihood of delivery has become very low or even non-existence. There is a genuine concern around that table that if you acknowledge that reality and say to use the cross-relay example, it's not going to be 2018. It's going to be 2020. There is, there is a concern around that table that a year later, all of that new float, all of that new schedule contingency would have got swallowed up. And that will be facing exactly the same situation again and then it will be 2020, 2020. And that is, there is, there is some truth in that line. And it is a, it is a, it's a genuine tension in a big project to say at what point is the right point to. baseline schedule. It's a valuable and important conversation. However, there is something else going on with that table and that is, if I acknowledge this isn't going to get delivered, then I have to go and tell the sponsor, this isn't going to get delivered. He has to then go and tell the minister or in this case the mayor that isn't going to get delivered. We'll end up with a team of outsiders will be sent in to work out why we've done such a bad job and what we should do differently. There's going to distract us for six months and I might lose my job. So these subtler tendencies, the subtler incentives encourage the delusion, encourage the obfuscation, encourage the unreasonable optimism that is endemic in a lot of government projects. This is known about, isn't it, that people have this optimism bias. Why does this still happen? What fundamentally needs to change then? Yeah. So I love the phrase optimism bias. It sounds doesn't it? It sounds like a kind of a very human sort of tendency to look on the bright side. It's quite a warm thing. It sounds nice, doesn't it? What we're talking about in these kind of situations and I'm going to use a deliberately nasty way of describing it. It's a kind of a collective conspiracy to dilute ourselves and not only ourselves, but our sort of stakeholders and funders. And that's why I think this happens a lot less in the private sector than it does in the still happens in the private sector just to be clear, but it happens as often in the private sector than the public sector. So coming back to my requirement for Canada, let's imagine a slightly different world than the one we're in. Let's imagine that the business case for this project, rather than being buried in bureaucracy and whitehall, is insofar as it could be done from a confidentiality point of view in the public domain. And that the schedule has the name of the project director, you know, in the public domain. This is the schedule which I say I can deliver. Hold me to this. And the, you know, the cost estimations in a similar way. And that that person knows that it is, it is, it is going to be viewed by these office of budget responsibility, genuinely independent people who are reporting to parliament, rather than reporting to government say. And then couple lap with that everybody I've just talked about is likely to be in their jobs for four or five years rather than 18 months. So you can't, so there's much less incentive to sort of just keep your head down and hope, and hope that it'll all be all right when I leave it. What we've changed there is the incentives in the system. And it's those incentives, it changes in those incentives, which will change the behaviors I naively assert. Do you think all the attention on data analytics and AI is one of those things that people hoped for project delivery with? So I'm going to take you back bloody decades because I talk about the astute program. The astute, you know, is the tax submarines that form the tax submarine part of the Royal Navy's nuclear defense fleet. And by the time we finish the astute program, there are still a couple of submarines still in the yard. Those seven submarines will have taken pretty much exactly twice as long as the previous class of seven submarines, the Trafalgar class took back in the sort of 80s and 90s. One of the things people thought right at the beginning of the astute program, and I recognize I'm talking 25 years ago now was rather than using warehouses full of draftsmen with beautiful drafting boards and pens and eggs. We're going to use this 3D CAD process. It's going to be much better. And of course, 3D CAD is much better and it's completely replaced hundreds of draftsmen with drawing boards. But at the time, it introduced a huge delay in the program because the technology wasn't really up to something as complex as a nuclear submarine. You could use it for a simple ship, but it wouldn't do a nuclear submarine and there weren't enough people skilled enough to use it. And so design went very much more slowly than it did. My suspicion is that same pattern, hopefully it'll happen faster, we see some of that pattern with AI and learning technologies. If we come back in 10 years time, it'd elevate a huge impact. But we'll go down a lot of rabbit holes first if we're not careful. And so I don't encourage people to experiment, to try, but not to lose sight of the fundamentals of designing and building complex projects. The kind of prevalent culture attitude that these projects are deemed to fail. How do you shift that? Is it about finding pockets of success and reminding people about those? The APM salary survey this year showed that there was a great distrust in the success of major project, government projects. So it's kind of entrenched and expected. So put issues in someone who's trying to lead a project, maybe not a massive scale, but faces this expectation of failure. What on earth do you do to try and make the people you're working with think that no, we can do this? Well, the things I'm about to mention, you know, anybody who's been living in this world for 10 years will recognize them. So things like, you know, the delivery professions, including project management, being government should have the same kind of esteem in government as policy development. Ministers and civil servants should spend longer in their jobs. The challenge of a project should be matched by the capability and resources that are required to deliver it. You know, this is not rocket science. Project shouldn't be initiated without clarity about what they're trying to achieve. The estimation, the cost and schedule estimation of projects should genuinely take accountability of uncertainty, which means that early on, you're going to have very wide bounds of cost estimate and the tendency will for everybody want you to be much, much more specific and exact before you know really what you're doing. So these are all really well known, unanticipated pressures on cost and schedule should be brought out into the open and acknowledged and dealt with as you go along. We've been talking mostly in this conversation about why these underlying things prevent these well-known things being addressed. We talked about duty of Canada. We don't need to talk about that anymore. Let me talk about two others. The first is if you're all that project leader, place a really high value on simplicity. Try to avoid the tenders, the pressures in the system, which will lead you to more and more complexity. I'm going to give you again, I'll be brief, an example that I, unfortunately, know a lot about, which is the smart metering program, where in pursuit of theoretical perfection, we developed a complex solution which became functionally undeliberable at least in the context of the timescale so that the government wanted to deliver. So rather than finishing the project in 2020, you know, it'll be the current timeline says 80% of homes in 20, 26, but I'm not holding my breath. So we could have done something much, much simpler. It wouldn't have had all the bells and whistles of benefits, but then it would have been delivered, you know, a decade earlier and therefore, you know, who knows what we're doing with technology by now. So, so one really important thing is prioritizing simplicity over complexity and the other, which is less in the hands of the project leader perhaps, is so I don't know if you've noticed how many times big projects get announced. How does HS2 announce between its first conception in sort of 2009 and an actual formal business decision being made in 20? 20 Boris Johnson approved phase one in 2020. It must have been announced, you know, 20 times. So the world takes a huge account of announcements, even though they don't really mean very much, and takes almost no account of actual formal commitments, business commitments. And if I were working for a day, I would correct that balance so that I mentioned before publishing of business cases, formal, proper formal announcements of when when ministers of genuinely made a clear, not not political intent, but an actual investment decision. That should be very clear, very explicit and very, very public. Because you're hugely experienced, I wanted to find out for me what perhaps have been the most important career lessons, maybe one or two that you've learnt. So either personally is it as a kind of project person and/or related to projects as well. So if you could pass on any advice, wisdom to project professionals out there who perhaps aren't as senior as you, maybe something you would have told a younger version of yourself. Well, I guess the first thing I'd say is, you know, stick with it. You know, these are fantastic careers being involved in these huge endeavors, world scale projects. Don't let, oh, grumpy grey head, men like me put you off, you know, stick with it. That's the first message. The second is, I think, so professionalism, we've come such a long way with professionalism in the last 20 years, but professionalism is more than a collection of sort of niche technical capabilities, scheduling or cost estimating or risk management or whatever. I talked before about parity of esteem in a public sector context, parity of esteem with the sort of the policy and what one of my project friends used called policy la la land. So that parity of esteem starts with you if you're a project professional. So if the policy won't can't explain how what they have written down as the scope of their project is really going to deliver what they think it well. You have to call them out on it. If the minister is unrealistic and I know it's easy to say and it's hard to do, you have to step up and say so. So there's an element of courage that I would encourage people to adopt. And the final thing I would say is so be skeptical from time to time when you need to be but don't ever let yourself drift into cynicism. Cynicism kills big projects. And in a way it's kind of why I wanted to write the book because I could feel that tendency in myself, the sort of this will never work tendency. And that we can't afford that as a country we can't afford it. And it is there are things that we can do that we have to do to do better. Johnson that's a great way to end the podcast. I mean it just leaves me to say thank you for your time. And for such a thoughtful conversation and obviously wish you the best of luck with your books. So thank you very much. It's a great pleasure. Thanks ever so much and I'm sorry we meandered around a little bit. No that's the that's the fun of it I think. I expect that so I plan for that some contingency contingency planning. Well that's what we need. So thanks again for joining us and to you for listening to the APM podcast. Just to mention again that his book The Delivery Gap, Why Government Projects Really Fail and What Can Be Done About It, Publishers in September 2025. Don't forget to look up for more episodes or to rate and review wherever you get your podcasts. We'd welcome you to get in touch with your comments, feedback and suggestions by emailing us at [email protected]. Spotify users can also send us feedback directly within the Spotify app. This podcast has been brought to you by APM, the childhood body for the project profession. For more information on APM visit apm.org.uk.