The SpecOps Method
Hello. I'm Ryan Koch, and this is Civic Tech Chat, a show that looks at the way technology, politics, and policy impacts the world around us. The tools we use, the way services are delivered, and how we talk about and set policy all shape our society. We'll gather around and have a chat about these things together and more. Before we get started, I do want to let you all know that we've started a Discord for the podcast.
Ryan Koch:There will be a link with an invite down in the episode description. Do feel free to go check that out. It's a small community right now, but hoping to grow it. It's a great way to reach out to me and let me know things that you might want us to cover or to just hang out and talk about Civic Tech. Mark, thanks so much for joining us here on Civic Tech Chat.
Ryan Koch:Could you introduce yourself and tell us a bit about what you do?
Mark Headd:Sure. My name is Mark Head. I am a senior director of technology at a digital services company called AdHoc. I currently and have, for many years in different roles, worked with and for federal and state agencies helping them implement technology solutions to better serve people.
Ryan Koch:What would you say is your personal why? The thing that drives you to get out of bed each morning and do what you do.
Mark Headd:Yeah. I ask myself that often. So for pretty much the entirety of my professional career, I have I've worked either in government or near government or with government. Almost all of that time, has been spent working with technology. I came into government, and I'm gonna age myself here.
Mark Headd:What I what I refer to as the second great wave of technology change came happened, that was the Internet. The first wave being telephones, which happened about fifty years prior to that. And, and the last wave we're going through right now, which we'll talk about more today. But when I started in government, the Internet was just happening, and it was changing, really changing the way government, operate. And so for most of my career, I've been, trying to help government figure out the best way to use technology.
Mark Headd:And it's a really hard problem, but it's extremely rewarding as anyone who works in Civic Tech knows if you can help, if you can help do this work well, you can really impact people's lives. So I I guess to answer your question, that's that's what gets me up in the morning. That's what gets me out of bed.
Ryan Koch:That answer is that maybe a great way to segue us into our topic today, as you kinda mentioned this, like, next phase of technology as you talked about. And, we're gonna talk a lot about, modernization because you have a book that's come out called The SpecOps Method. You know writing a book is something that takes a lot of time, a lot of energy, passion, and you really dig into digital modernization and the urgent need for new approaches and you call for urgent action. Why did you write a book like this one?
Mark Headd:Yeah. And I I I asked myself if I would write another book knowing what I know now about how how much energy and time it takes. I was so naive when I when I started, but I I felt a real urgency to to write this book, and and I and I feel continued urgency to talk about this issue. 2025 was a a real watershed year. I had mentioned previously the the third wave.
Mark Headd:I call it the third wave of technology change. The the telecommunications revolution that brought telephones into people's homes took decades to to to actually happen, and governments had a fairly long time to adjust to it, and it it did change the way they operated. You know, you had telephones in government offices. You have government phone banks. All that happened over many, many decades.
Mark Headd:When I was coming into government, the second wave was happening. This was the Internet wave, and things moved much more quickly. And governments had a really hard time keeping pace with that change. And in many ways, we're still playing catch up. I don't know if you've seen the latest numbers on, the web accessibility audits of federal agencies, but we're still struggling with those sorts of things.
Mark Headd:The way I describe it to people is we're now in third wave of change, the AI age, and the change the rate of change is much more rapid. So to me, the imperative is how do we ensure that governments do what they need to do to prepare for the changes that are already here. But in 2025, some really important things happened. Right? We had a we had a really significant change in the federal workforce.
Mark Headd:About 317,000 people left federal employment in 2025. Now that was that's above the average. If you look at the average over, the last, I don't know, ten years or so, it's about a 150,000 retirements per year, for for many reasons, which we won't get into on this podcast. There are a lot more of those in in 2025. Taken together, if you if you net out the the in the the new employees that came in, about 11% of the federal workforce left in 2025.
Mark Headd:That's a big deal. The other thing that happened in 2025 was AI coding agents really came into their own, and it happened extremely rapidly. I only saw the change in the last six or seven months of 2025, but a a really fundamental change has taken place. I always describe it that way. I always describe it sort of in the present and past tense now because it the change has happened.
Mark Headd:And the debate that's kind of going on in the world of software development is really, in my opinion, just coming to grips with that. You know? We're going through the various stages of acceptance, if you will. So it really it really became a thing that I know, I I described it to someone recently as, I don't know if you've ever seen the movie Close Encounters of a Third Kind, but there's a character in the movie that's played by Richard Dreyfus, and he sees the devil's tower mountain, in in everything he's doing. He's making it out of his mashed potatoes at the dinner table.
Mark Headd:I was seeing this issue that I read about in the book. I was seeing it everywhere. It was in everything I saw, and I just felt this really urgent need to, to write a book about it and and hopefully talk more about it and get people thinking about it. That's why I wrote the book.
Ryan Koch:And your reason why you mentioned that, like around 10% of the workforce retiring, you know, a higher than normal amount but Yeah. A lot a lot of folks. Right? You actually in the book talk about some specific examples of categories of folks leaving that are problematic like for example you mentioned COBOL programmers.
Mark Headd:Yeah.
Ryan Koch:With an average age of 58 and actually 10% of that workforce also on average are each year. Can you talk about the urgency of that situation as it applies to kind of this bigger picture and why it's important for folks to pay attention to it?
Mark Headd:Yeah. The the people that are leaving, if you look at the the demographics of government employees, I'm I'm I'm referring to federal data from the office of personal management, but we can easily and comfortably extrapolate to many state governments. The the the the workforce about 15% of the workforce is in their sixties. Less than 7% is in their twenties. The people that are leaving government, the people that are retiring, even though, you know, even though 2025 was an outlier, that trend is still not very favorable.
Mark Headd:Right? The people that are leaving are the people that have worked there the longest. They have the most institutional knowledge about how systems work, why they work the way they work. They are they have such valuable information that's walking out the door. I talk a lot about this in the book, but what's what's what ends up happening is, in some instances, these folks go back to work for government as contractors.
Mark Headd:Some of them into their seventies, because there simply are no other people that can do the work that they do. So, you know, we've known about this problem for a while. This is not a secret. It's been talked about in the world of Civic Tech, I think, for a good bit, and people have been, uncomfortably aware of it. But what happened in 2025, you know, accelerated the trend, and I think made it more urgent.
Mark Headd:So we are we are we are living through a time when the people who understand the most critical systems in government are walking out the door. And the imperative that I try and, articulate in the book is be before we let that happen or before it happens to a greater degree than it's already happening, we need to take steps to capture the knowledge that they have, to understand the systems that they've maintained for decades, before it's too late.
Ryan Koch:It's it's hard to talk about, staffing without also talking about budgets. They tend to go hand in hand as it were. And one of the things you point out in the book is that 79% of federal IT spending goes to effectively keeping the lights on, you know operations and maintenance. Which really that doesn't leave a lot left over to make new things or to incrementally replace something or to even just make improvements to something that's there other than just keeping it turned on. What role do you see that reality having in creating and then sustaining these problems that we've been talking about so far?
Mark Headd:Yeah. The and I'm not the first I've been saying this for a while, but I'm not the first person to say it, and I won't be the last person to say it. But the the way we budget for technology projects in government is is one of the reasons why we are where we are. And I I do go into this, as you say, a bit in the book to talk about the way these projects are funded. I think mostly it's a reflection of of of an older view of technology projects as more traditional infrastructure.
Mark Headd:You know? There's a there's a there's a point at which you pay a whole bunch of money up front to build a thing, and then you sort of pay a lot less annually over the usable life of that asset. And we know as people with you know, working in the Civic Tech community that that's the wrong way to look at these projects. We need to we need to view them as products, and we need to fund them as products. And we need to have funding to enhance features, and and meet the needs of real users.
Mark Headd:And, you know, the reality is that that's just not the way most governments budget for technology. I think I I think an important thing to understand is that and this goes back to what I was saying about the waves of change. This is the way we've always budgeted. Right? The budget process works in government as it was designed.
Mark Headd:It was designed to work the way it works, And there are very good reasons why it works the way it works, where we have an annual appropriations process. You know, we have accountability, through, audits and, you know, things of this nature. There's a good reason for it existing the way it exists. And I think a a misconception that that some people have, particularly people that are new to government, is that, oh my gosh. This this system's broken.
Mark Headd:It doesn't work. It it it actually does. It works the way it was designed to work. It just doesn't work well in an era when technology is changing by the month. Right?
Mark Headd:It wasn't set up to move that quickly. There was a time, for example, when the the telephone was making its way into everyone's home, and it fostered this different expectation for people about how they're gonna interact with their government. And then eventually, saw, as I said before, you saw telephones turn up in government offices. You saw government phone banks, all of these sorts of things. That took many, many And the budget process and the and the policy making process all move at that pace.
Mark Headd:But starting with the second wave, the Internet wave, and certainly now with the third wave, the AI wave, those those processes do not move quickly enough for for the rate of change that we're seeing.
Ryan Koch:Do you see that, adaption to rate of change as being something that's uniquely government organizations as a problem, or do you see that as it's just big organizations struggle with this kind of thing?
Mark Headd:That's a great question. I I definitely think that there's that other big organizations, large bureaucracies, you know, particularly in regulated industries, will have the same challenges. You know, this is an historic pace of change that we're seeing. So, yes, there will definitely be struggles with other large organizations, and and they have their own bureaucracies to navigate too. They have their own processes that where they're accountable to shareholders and and and other stakeholders like that.
Mark Headd:With government, it's it's a little bit different because, a lot of the things that slow down the budget process or make it I should say, it more deliberative are things that that we want. We want people to have a chance to participate in that process. We want there to be transparency in that process, and we want there to be accountability. And in many times, those things manifest as more time. And for much of our history, that was okay.
Mark Headd:Increasingly, though, as technology becomes such a fundamental part of how government operates and and since it's now changing so rapidly again, I'm not the first person to say this, but it's definitely time for a reexamination of how do we how do we accommodate those very, very important, principles, but also move more quickly and become more agile in how we, we adopt technology.
Ryan Koch:One of the things you talked about before, we were talking more about the kind of the labor situation, was the importance of institutional knowledge. I think that was a term you'd use. And, in the book you make a reference that I'm a fan of, where it's at Marianne Bellotti's writing, I hope I'm pronouncing their last name correctly.
Mark Headd:I believe.
Ryan Koch:But they mention how legacy systems are there and become a legacy system because they managed to work that whole time and maybe even work really well for some or all of that time. How do you suggest ensuring that respect for that institutional knowledge is there as modernization work takes place?
Mark Headd:Yeah. It's a great her book is it's on the bookshelf next to me. Never Far From My Reach. It's a seminal book on legacy system modernization. And it's a it's a very different way to think about legacy systems.
Mark Headd:Right? She she suggests that the reason these systems have been around so long is because they've they're good. They're successful. They achieved their end for a long, long time. Maybe they've outlived their usefulness, but for a long time, they they were very valuable.
Mark Headd:And I think, you know, in addition to it being just a very different way of thinking about these systems, I think it gets to some of the reasons why it's so hard sometimes to replace them because we have as you mentioned, we have this whole body of institutional knowledge that's been built up around them and all sorts of internal business processes that are tied to them. They they're very hard. It's a legacy modernization is probably the most difficult thing to do in government technology. It's extremely hard. And I talk about in the book, I talk about the the track record of projects that have have tried to do it and have failed.
Mark Headd:You know, it's it's a long list. You don't have to look far for examples of where it has gone wrong. So I think, to me, the reason her book was so useful in framing my thinking on on on what I wanted to write was because it really centers on this idea of knowledge being the valuable asset. I think that most of the time when we think about legacy modernization project and you talk to people who are involved in that work, the goal is we wanna turn old software code into new software code. And what I'm proposing in the book is that that is wrong.
Mark Headd:That is the wrong way to think about it. The right way to think about it, in my opinion, is we need to capture the institutional knowledge around that system. We need to understand in detail what the system does and why it does that, specifically, like, the policy directives and the other the the legislation that drives what that system has to do. That is the the and should be the focus of legacy modernization. Code conversion, converting to new code, that's incidental.
Mark Headd:That is incidental to the knowledge capture that should be at the heart of legacy modernization. And I think that's a still a fairly controversial statement. I may take some some some heat for that, but I believe that that's the right way to start thinking about these projects. And her her writing on it is eloquent and and engaging and and definitely worth worth the read for any any listeners.
Ryan Koch:I know I've I've seen this myself in in my day job trying to do some modernization. I imagine you've seen it along your path and many others out there in the audience that you mentioned in the book this second system effect that I think you referred to as being from the mythical man month which is also a fantastic book. But the idea is like as you're doing the work folks who are kind of helping you figure out it should do start to kinda put things they wish the old thing did onto the new thing. And then, oh, like it you know, we're making this new thing. It should go ahead and fix this problem too.
Ryan Koch:What kind of impact can that behavior have on a modernization effort?
Mark Headd:Yeah. I mean, the the most direct consequence of that is, scope creep, is, you know, the accumulation of risk. I talk a lot in the book, and this won't be a surprise to you or to anyone who's ever worked on legacy modernization, but they they typically follow a pattern of, big bang deployments. So the the the plan is that you're gonna build a replacement system, and there will be some point in time when the old system gets shut off and the new system gets turned on. And invariably, that approach does not work.
Mark Headd:I talk at length about an example in the book from, the government of Canada and their Phoenix payroll system, but there are many others, that follow that same pattern. And I think it's because so to your question, Ryan, I think the the reason why that can be so tempting is because I think it relates to the budget process we talked about. I think it there's often a sense of an overarching sense in these projects of this is our only chance to do it. We we get this one shot to to to fix things, so let's fix everything. And, you know, to some degree, I think it goes back to, you know, product thinking.
Mark Headd:Thinking about this as product that's evolved over time based on the feedback you get from real users rather than we're gonna just throw everything in the kitchen sink into this modernization, and all of our problems will be solved. Right? It's very tempting to think that. And I'd I'd be I'd be I'd be overs I'd be I'd be missing something if I didn't say that. I think oftentimes, that sentiment can be a product of, like, top down, like, you must to the team.
Mark Headd:You know, you must fix this. You must fix this. That that's never a recipe for a successful project. But
Ryan Koch:We've talked a bit about the the varying problems that can happen as one embarks on these projects. And you started to talk a bit about maybe something that's a good segue into some solution type thinking, like the as you mentioned, SpecOps. This idea that using the institutional knowledge as you talked about in your prior answer. Your book with that same title spends a lot of time talking about that method. For folks that maybe haven't picked the book up yet, what is it and why should agencies and teams look at working that way?
Mark Headd:Yeah. So as as I said before, I think the the traditional way of thinking about legacy modernization is how can we convert the old software code whether it's COBOL or Java or Assembly, whatever it is, and convert it to a new software stack so that we can be done? And in the usual course of those projects, there are documents, many many, many documents created, you know, system architectures, requirements documents, all sorts of things. But they almost always exist only to serve that transition from old code to new code. So the the title of the book is SpecOps the SpecOps Method, a new way of thinking about legacy system modernization.
Mark Headd:Because what I propose in the book is that instead of focusing on code conversion, updating the code, we should focus on capturing the knowledge that's that's currently embedded in that code. Right now, in these legacy systems, they are the source of truth. Whatever the COBOL says is the source of truth for how the system works. And that has a lot of disadvantages. Right?
Mark Headd:Because it means that if if if you or I are are are not an engineer or a policy expert or a domain expert, we can't inspect that source of truth on our on our own. We have to talk to an engineer, and more specifically, we have to talk to, like, a Cobalt programmer. So the spec in SpecOps comes from something called specification driven development. And I mentioned earlier that this is something that's coming out of the the the rapid development of these AI coding tools. It's a it's an approach to building software that is based on a foundation of a detailed specification.
Mark Headd:So before you build anything, you first describe in detail what it is, how it's supposed to work, and why it's supposed to work that way. And then from there, you can have AI tooling build this thing for you. So in practice, what it what it shakes out as is you spend less time coding because your AI agents are doing that for you, but you spend a whole lot more time clearly articulating what is being built. The the ops in the title comes from a a concept called GitOps, which is a little bit esoteric, and maybe some folks aren't as familiar with that practice. I do talk I do have a chapter in the book where I talk about it.
Mark Headd:In essence, it's just a way to describe the state of a cloud environment. And that documentation usually, people have heard of something called infrastructure as code. Really, that's just a bunch of files that describe the state of a cloud environment, and and that's the source of truth. It's not the actual running environment. That's just an implementation of the truth.
Mark Headd:You want the truth, you go to the documentation. So the really, the book sits at sort of the crossroads of those two ideas that you know? And and when when these changes were happening is that I was working with AI tools myself, and I and I you know, I'm my thinking is obviously informed by the work I've done in the past working on legacy systems. It just occurred to me that this this we've been doing it wrong. We've been thinking about this the wrong way.
Mark Headd:And I have one of those moments where you're like, you know, you're you're you're talking to everyone you know. Every single conversation comes back around with this idea that you've got burning in your head. You know? I found myself at the grocery store, you know, talking to people, talking about this, and, you know, I joke with my wife about it, but, you know, she says, you know, there's probably some guy at the at the produce counter going, hey, man. I'm just looking for the organic bananas.
Mark Headd:I don't know anything about SpecOps. But every conversation kept coming back to this. I felt this real urgency to to to talk about it and to write about it because I myself had worked on some really painful legacy modernization projects. Some of them I talk about in the book, but others I haven't talked about. And I really feel like, you know, it was it was really important for me to to to try and get this idea out there.
Mark Headd:But it's really a function of these new methods of software development, which, you know, some listeners may not be as hands on with, but I'm I'm sure that people are being exposed to some of the the debate about this. But it it really has fundamentally changed in just a very, very short period of time how software gets developed. And there's a whole bunch of debate in the software development community about it, but, you know, my take is we've crossed the Rubicon, and we're not going back. And how we deal with that as people that work with and around technology and government, I think that's there's still some debate about that, but I don't think there's any debate about whether or not the fundamental nature of software development has changed, and that's kinda what underpins the book.
Ryan Koch:As we talk about those specifications, that knowledge that then would get used for modernization effort, kinda using the method you're describing, what would you say are key aspects to make good specs or specifications?
Mark Headd:Yes. That's a great question. So a software specification has to serve two purposes in in the SpecOps Method. It has to be detailed enough in its technology details for an AI coding agent to use it to build something. So, Ryan, if you were to write a software specification for a new podcasting software solution.
Mark Headd:You would need to write enough details in it so that your agent would know, what am I building? What software stack am I using? Am I using any sort of special approaches? Am I interfacing with any other systems? But the reason why I think this approach is is so valuable and so important is because if we do this correctly, it allows people who are not software engineers, who are domain experts, who are policy experts to inspect the source of truth and to verify, yes, that works.
Mark Headd:That's exactly what it's supposed to do. So if you were a, someone who's a domain expert in public benefits, you could read a software specification and and understand whether or not something was, described correctly or incorrectly. And I provide some examples in the book of this. The way it works now, if you're a domain expert in public benefits and you wanna understand how the system calculates based on a specific income type, more often than not, you've gotta go ask an engineer, and the engineer has to then go and inspect the code and say, okay. I see what's happening here.
Mark Headd:It looks to me like this is how the system accommodates that that income type. The the real value of SpecOps is that it brings people who are domain experts and policy experts, the people who know the most about how these programs are supposed to work. It brings them into the conversation about the system and how it's supposed to operate. And that's one of the things that these all these changes in the world of AI coding has enabled. It's enabled a lot of things, and some of them are not good.
Mark Headd:But one of the very good and valuable things is that with this approach, it allows people who are the real experts in understanding how policies and programs are supposed to operate, and it gives them a seat at the table in determining how these systems are built and operate.
Ryan Koch:In the book, one of the things you advocate for, kind of in that AI debate of how do we use it, is, beyond just the use of it that you see people talk about on the Internet that generating new code, right, based on prompts. You talk about using it to read old code. Kinda going on that thing where these tools are very good at like taking lots of text and summarizing and drawing, find finding patterns, that sort of thing. Why do you see that use as a being a particularly compelling use case for this kind of tool?
Mark Headd:So to me, the the it is what I propose, to respond to the trend we talked about, at the top of the discussion, which is the people that know the most about these systems are leaving government in in record numbers. And even though there are many things we can do with AI tools, we can create funny videos. We can create, you know, images. We can do all sorts of things. One of the things I think that will have the most return on investment for governments is to use these tools to better understand their existing code bases, particularly when the number of people who understand those code bases are leaving.
Mark Headd:So I propose in the book to use these tools to generate these specifications that describe how a legacy system works. And while we have access to the people that are still in government to help us verify that, yes, that works the way it's put that that is correct. Or, no, that calculation is wrong, but, it's fixed down the line, in another system or by another, you know, manual process. All of those details that that underpin how these legacy systems work, many of them are captured in the heads of the people that are leaving government. So we can use these AI tools to generate new code, and and and maybe that's the way we build replacement systems.
Mark Headd:It doesn't have to be, but I believe it is a very effective way for governments to better understand the software that they have. And if I could, Ryan, I'll share a a quick story with you. It I haven't told anybody this yet. This is the first time I've ever shared it, but I won't name the state. But I worked with a state not long ago, where I was brought in.
Mark Headd:There was a the the the motor vehicle department was having conversations with the agency that handles drug and alcohol treatment programs. There's an obvious connection there. Right? Because if someone gets a violation for impaired driving, their part of their sentence might be to complete a drug and alcohol program. So there's there's a data interchange point there.
Mark Headd:And, you know, in some side conversations, the folks from the motor vehicle department said, you know, we're having these conversations with this other agency, but we can't get any traction. We don't understand why nothing's being done. And through some investigation, what we came to learn was that there was a an old process that batched up a whole bunch of data from the drug and alcohol treatment agency for the motor vehicle agency, and no one understood how it worked. No one knew who had done it, and they were they were terrified to touch it because that they thought they were thought they were gonna break it. These are the kinds of things that it was written in a in a very old scripting language.
Mark Headd:It was it was using a very integrated approach, and I there was a lot of really, really high anxiety about touching it. So what I'm proposing in the book is this is a perfect opportunity for us to even if we don't change that code, it's a you know, initially, this is a chance for us to use these tools to better understand it. What is it doing? Is is it doing something we're not fully appreciating? How would we propose replacing it?
Mark Headd:These are all questions we can start to ask with these tools that don't necessarily involve generating new code. It's really just a way to document out of these existing systems where and I'll tell you, I'd love to tell you that that's an outlier. I'd love to tell you that that's a very unusual circumstance in government, but it's not. There are many, many legacy systems. Now they're not all huge, you know, payroll systems like the ones I talk about in the book.
Mark Headd:There are tons and tons of small one off legacy systems that are not documented, that are doing critical functions and gluing systems together that we just need to understand better.
Ryan Koch:So I guess done in a semi ideal sense, what you might have someone do is say set up an AI tool of their choice, maybe using like a CLI tool or again or an ID, whatever it is that they prefer. Have it run and with some prompting that's like hey help me document what's going on here and probably much nicer descriptive language. You then would get something that you could have both like engineers look at and go oh does this make sense with like my institutional brain. But like now I'm looking at something going, oh is this correct or not correct? Not like tell me everything you know as the question which is much harder.
Ryan Koch:And then you can also take it and hand it to a program person and go hey like do the rules that it's talking about here in the code make sense or are they correct? Yes. And maybe there's even goofy stuff where they're like, oh well yeah it does that but then like I open the spreadsheet and I fix it and send it back and of course you can choose whether you want to keep that or not as you make a new thing. Is that is that the kind of flow you're advocating for?
Mark Headd:Exactly. It's a it's an iterative process where both engineers and domain experts can can use AI tools to better understand this code. For example, using the scenario I just described, we could have engineers examine this code, whatever code was running this this process, to better understand, oh, I see what's happening. It's pulling down a dataset from the mainframe, and it's batching it up into a ZIP file, and it's placing it on this FTP server, and then another process comes along. Okay.
Mark Headd:I get the details. From a domain perspective, you know, this specification would have enough detail for somebody to come in and say, oh, wait a minute. We we're not allowed to share that data. This this particular field is not shareable because it's there was a there was a law passed two years ago that prevents us from sharing. These are the kinds of conversations that this approach enables.
Mark Headd:It isn't just, I'm a domain expert. I understand what we're legally allowed to share and not share, but I can't look at the source of truth. I can't look at the code and make a determination as to whether or not from a policy perspective that it's doing what it's supposed to be doing. And that is really the big change that has happened, and to me is is is a thing that look. Legacy modernization, there is no magic here.
Mark Headd:This this isn't AI is the magic solution to all of our legacy modernization problems. It's not. I say this in the book, but it's it's basically that what we've seen occur over the past year or so has converted something that was almost impossible to something that's now just difficult. Right? It's still hard work.
Mark Headd:We still got a a lot to learn. But if we can give nonengineers, new domain experts and policy experts, a seat at the table, we will be able to capture the knowledge we need before we see more and more people walk out the door with that knowledge in their heads.
Ryan Koch:Earlier in our conversation, you made mention to the term GitOps, which is someone who's done the kind of tech side of things a lot of my career that hearing you talk about also seeing it in the book perked my interest. In particular you cite the benefits of it having that declarative nature, having things like versioning among its other benefits. How do you see those principles being generalizable to kinda organizational practices for modernization?
Mark Headd:Yeah. I mean, as someone who develops software a lot, I I'm I I I I'm probably more attracted to version control than maybe your normal purse, the guy in the grocery store who I was talking to. But the the idea that these specifications that describe how these important systems work are living documents, Our the you know, version control is important when documents change. If documents never change, we wouldn't need version control. So to me, the the one of the powerful, connectors with with GitOps is that you have these documents that are constantly being refined, and they're constantly being iterated on, not only as we add new features because, you know, perhaps the law changes or our policy changes, and we need to the system needs to work differently.
Mark Headd:We're we're seeing that now play out in in states with h r one and some public benefit systems. But, also, as we learn more about these systems, as we we we can complement our knowledge, We'll always be refining it. These are these are living documents. They should and and everybody that works in in these domains, whether they're engineers or or policy experts or frontline people, you know, people, you know, doing all sorts of things. They all have an ownership stake in that because it represents the system that that supports the work they do.
Mark Headd:So, you know, it's I'm admittedly, I'm probably more enamored. Mean, you and I are probably more enamored with the the idea of GitOps than than maybe a layperson. But the other thing I think it does is that, you know, GitOps is a is a is a mature practice. It's a mature form of DevOps. So the fact that the SpecOps Method I described in the book shares some of the same DNA with with GitOps to me is another validation point of, yes, this is an approach that that works.
Mark Headd:And we haven't gotten into this yet, but it it also sort of gets to the idea that the tools we need to do SpecOps work are not these elaborate, you know, expensive tools. Git is an open source tool that is pretty ubiquitous, and you can get for free. That's another part of the book that I'm really excited about because when I when you think about how we do this work, we don't need to go out and purchase an elaborate, you know, software solution or tool to do it. We we have the tools. You need an AI model, but most governments have access to those things now.
Mark Headd:You need you need a version control system like Git, and you need a just a few other things, and then you can do this work. It is there isn't huge barriers to doing it. So for all those reasons, think the GitOps connection is a is a really powerful and important one.
Ryan Koch:As we're we're talking good practice, another one that the book references is the strangler fig pattern. For folks that are reading the book and kind of learning about this, what is that pattern, and why do you consider it a good practice?
Mark Headd:That's a great question. The strangler fig pattern is, is a is a term that was coined by a gentleman named Martin Fowler, who's a who's a prolific and and very respected, software developer and engineer and writer. And it refers to a type of plant that, grows up around an existing plant. And as it grows, it encases the old plant and eventually the old plant, I guess I guess, dies. So it's a little dark.
Mark Headd:It's little bit of a dark analogy, but, the idea is the the reason why I've it's it's a very common concept in legacy modernization. It's not the only, pattern that that people use, but it's it's it's a very popular one, a very well known one. And if we bring that idea into the world of legacy system modernization, the approach dictates, a phased approach where you incrementally replace a legacy system. So you have some period of time where you have your old system that's still operating, and then you have this new system sort of growing up around it. And then eventually, at the end, the new system encases the old one, and the old one could be decommissioned and turned off.
Mark Headd:It's I think the reason why it it resonates with a lot of people in the world of Civic Tech and specifically in in the communities that have done legacy modernization is because these these these efforts can take a long time. They can take years. And, it it it is a way to accommodate the time it takes to do these, to do these projects. And, for me, it was it was important to to at least touch on this concept in the book and to talk about how SpecOps can be used in in connection with it wasn't it wasn't in conflict with what we consider to be a very, very common way of doing legacy modernization. But the name comes from a, very strange plant.
Ryan Koch:You also spend a lot of time talking about the importance of subject matter experts validating the specs as we've kinda referred to earlier in the conversation. It seems like there is maybe some effort for teams to get to a place where that's possible between say your technical teams and your program teams and other stakeholders. It's almost like you to even get to a place where you can understand how you're talking to each other.
Mark Headd:Yep.
Ryan Koch:How can teams get to that place of whether you call it, like, shared language or a place of understanding?
Mark Headd:Yeah. It's a great I mean, we're now we're getting into how do we do SpecOps successfully. How do we do it well? You know, like any project, it requires, it requires a clear mandate. It requires, good leadership.
Mark Headd:You know, I I don't think it's as, there was a time when I think, you know, just the whole notion of a cross functional team was was unique, but I don't think that's the case anymore. I think it's fair fairly common now. So, I think, you know, I've had many conversations over the past six to eight months with people that don't consider themselves engineers. They consider themselves policy experts and domain experts. And, you know, these folks are often they don't feel empowered or incentivized sometimes to ask questions of the engineers or or or they don't feel like they're given the opportunity to say, hey.
Mark Headd:Wait a minute. Does is that does that work the way it's supposed to work, or how can we validate that? Or you know, these are the kinds of, I think I think, opportunities that SpecOps helps surface to really give those folks a a seat at the table. But, you know, the people that have the most expertise and the most knowledge in those areas are are are are are also some of the people that are leaving government. So I think bringing those folks into these project teams and empowering them to, you know, be the arbiters of correctness is is super important.
Mark Headd:Whether whether you're I would say that whether you're working on an active legacy modernization project, like, we have to replace this system by such and such date, or as we were talking about before, we just wanna understand this work this code better. We just wanna document it as a as a insurance policy or an investment we make for the future. But it it's a very different way of thinking about, a technology project where usually it's engineers who are the arbiters of correctness. And in this model, it's the domain experts and the policy experts, the people that understand how programs work. They're the arbiters of a a a certain aspect of correctness in the system.
Mark Headd:They're they're not going to try and say, hey. You need a a a, you know, high availability or, you know, five nines of availability. I'm gonna I'm gonna weigh in on that. But they are the ones who are gonna say, you have to calculate this benefit this way. Or if if this system works correctly, this application should be approved and this one should be died.
Mark Headd:You know what I'm saying? They get a full they get to see the table now to to to validate those things without having to have an engineer translate for them.
Ryan Koch:I've seen in shops that there can be a difference between like ceremonial signing off, like I kinda look at something just kinda stamp it, versus like a deeper more meaningful review of the thing. As I guess we're kinda continuing on this topic of doing SpecOps well. As your team kinda say pursues that, how can you encourage that deeper review to happen when teams interact?
Mark Headd:Yeah. I mean, that's a that's a great question. I think it's it's probably applicable to non SpecOps teams as well. You know, like any effort in government in the kind of work that we do, you know, getting the team composition right, getting the right folks on a team, getting the right leadership for a team, making sure the remit is clear. Those are all super important.
Mark Headd:I think that, you know, I do think that the the folks so we we talked a little bit about how policy experts and domain experts, you know, now become sort of first class disciplines, and that requires a little bit of a minds mind shift change for them. I've like I said, I've talked to many of them, and they're they're is there are times when they're like, you know, should I be asking this? Is this should I be buttoning on this question here? I I I think it will require some practice. I think it also requires engineers and technical leads to understand that, you know, that there are other people who are now first class participants in this process.
Mark Headd:But there's a there's a whole chapter in the book about getting started. I wanted this to be very practical. Like, how do I actually start to use this? I do think this will require some practice, and I do think that picking out the first project that you tackle with this approach is important. Right?
Mark Headd:It's it it needs to be one that's that's consequential enough so that it doesn't look like it was nothing, but it can't be, the most important system in the government on the first go. But getting that practice in and building up that culture, you know, this this is not dissimilar to what happened when, when DevOps, changed things. Right? There were plenty of old school system administrators who were used to managing the infrastructure from their command lines that did not wanna hear about, you know, continuous integration and automation. They were like, what's that?
Mark Headd:That's what I do. And that required a real culture change in in technology teams and and really changed the way the people that write software participated in those in those efforts because they now became a different level of participant than than they than they had been. I think the same is gonna be true of SpecOps and other similar methods where it will require some practice. It will require some culture change. But the good thing is that we do have we do have a little bit of history to look back on in terms of, like, how did DevOps become the norm?
Mark Headd:Because there was a time when if a developer walked into a, you know, a a meeting and said that we should do DevOps, you know, you had a whole bunch of, you know, people that wrote Perl scripts and were doing sysadmin from the command line. We're like, uh-uh. We're not doing that. So I think we'll get there, but it it does require some practice, which I would I would definitely urge people to look at the getting started section of that, of the book.
Ryan Koch:As you talk about culture change, something you do point out in the book is that at some projects, you end up with this kind of like good news only culture, which Yeah. You can argue a lot of that comes back to like do people feel, you know, psychologically safe to say something is to say bad news, right? To whoever needs to hear it whether it's kind of up a chain or to a particular person who's working on a thing. How should teams try to help foster an environment where folks can feel like I can say that this is not right, rather than just letting it roll? Yeah.
Mark Headd:You know, I think some of that was at least some of it was a function of, the the nature of software development. The the the big change we've seen is that the cost of generating code has come down dramatically and will continue to come down. And when code is really expensive and you find out after that code's been generated, maybe even after it's been deployed, that something doesn't work. It's a big deal. It's expensive.
Mark Headd:It's painful to change that. Maybe maybe as, the cost of writing actual code comes down, that changes. And, you know, we're teams aren't as hesitant to say, hey. There's a there's a mistake in the system, but it's okay because we'll just update the spec, and then we'll generate a new version of the code, and we'll we'll fix it. But I do think some of that was is a relic of it's expensive to generate code or or used to be.
Mark Headd:And, you know, some some of these not some of them. Many of them, all of them are political in the sense that, you know, careers have been won or lost on on projects like these. So there's a there's that whole other dimension to these projects that, you know, makes it sometimes more, advantageous to not share bad news, to not be the bearer of bad news. But, I really hope that, as we continue to see the cost of developing software come down, that there'll be a greater incentive to for for lots of people to step up and say, hey. Wait a minute.
Mark Headd:That that doesn't look right. That looks wrong to me. Let's let's fix that.
Ryan Koch:I I think I hear a connection there where, for example, in, like, DevOps space. Like, we've talked we talked about how, like, using c I d continuous continuous integration systems allows you to effectively make change cheaper, less risky. If you change fast, you can undo mistakes or you can fail forward. As the production of code becomes cheaper, it's kind of a similar effect. Right?
Ryan Koch:If work is incorrect, well it's not that big of a deal to fix it, or to redo it, or to pull it back. So it seems like maybe those concepts connected. It's, not just about organizational risk, but even personal risk in the organization it goes down as it's easier to make change. Would you say that that's kinda there?
Mark Headd:I I would. I mean, I think, I also feel like there's a connection back to our previous conversation on the the the we were talking about trying to fit every change into the system. You know, the the the high cost of developing software, created a lot of, you know, perverse incentives, I think. We we see a lot of those incentives play out over these projects, and they're the cause of a lot of dysfunction. I'm not naive enough to believe that, you know, the changing nature of developing software will create new problems.
Mark Headd:I'm sure it will. But I do feel like you're right that, you know, one of the benefits of DevOps was it lets you iterate more quickly. So the cost of a mistake is, is is reduced because you can fix it. You know, as we move into, and we are, unavoidably moving into a time when AI tooling is used to generate code, we will probably see more of that, where changes is easier and and mistakes are less costly. And maybe that will help folks who are, domain experts and policy experts step up and share when something's not working right.
Ryan Koch:With the last few minutes we have here in our conversation, I wanna ask a couple of kind of feature facing sorts of questions. For one, I kinda wanna speak directly to a group of, like, real folks that are probably out there listening to this. Know, imagine there is an IT manager or a director or other folks in that leadership structure that they're hearing this, and they're like, you know what, yeah, I want to advocate for this approach in my work. What could they cite as ingredients for something like I think in the book you mentioned, like, a ninety day SpecOps pilot as being a virtuous thing to go for. What are, ingredients for that?
Mark Headd:Yeah. I I I I really wanted that chapter of the book to be something people could pick up and start to use. And I I'm actually I'm I'm working with a loose coalition of people who are trying this in in various governments today. So I think, as we said before, I think picking the right pilot project is is really important. You're and this is true you know, this was just as true when people wanted to move to the cloud, they needed a pilot or they wanted to, you know, adopt DevOps or user testing, whatever that was, whatever that that point of change was.
Mark Headd:Somebody had to figure out, well, we I I need to advocate for this to be a regular practice in my organization. So I've gotta pick a project where I can show that it works, and it's gotta be meaty enough so that it will land with the the, you know, leadership or stakeholders. And they'll say, oh, yeah. That that actually I can see the value in But it can't be so big and complex that I fall on my face the first time I do it. So I try and lay out in the book set of conditions for for picking the right project, but it really depends on the environment that someone's in.
Mark Headd:We talked earlier about the tooling. You need your tools. They these are not elaborate, expensive tools. They're probably all already available in whatever agency you're working in. I list out the tools in the chapter.
Mark Headd:And then putting your team together, picking the right kind of people. One very important kind of person that you need is you need a domain expert. You need someone who's gonna be able to so, Ryan, when you point your AI tool at your legacy code base and you generate the first version of a a specification, you're gonna inspect it and say, yeah. This looks right to me. This looks like it makes sense.
Mark Headd:AI tools are very good at generating, correct looking, content. It needs to be validated. You need a domain expert to look at it and say, yes. That's right. That that makes sense.
Mark Headd:That matches this provision of the statute, or no. That that that changed. It wasn't updated here, but it that has definitely changed. We need to fix that. Having those kind of people available, is is is super important.
Mark Headd:But beyond that, you really don't need anything else. You need an AI tool. You need a version control system. You need a way to track track track work like a Trello board or, you know, GitHub issues. You don't need anything really elaborate.
Mark Headd:It's probably already all available at your agency. You just need to pick a good pilot project, and you need to find your domain domain experts, and and you're off and running.
Ryan Koch:Mark, thank you so much for joining us here on Civic Tech Chat. From this conversation, I have no doubt that practitioners and others out there are gonna have nuggets they can bring into their work or just interesting things into their day.
Mark Headd:Ryan, thanks for having me. I really appreciate it.
Ryan Koch:Visit us on the web at civictech.chat or subscribe to us for content updates wherever it is you download your podcasts.