Investing in agility with James Shore

Kevin Montalbo

Greetings listeners, my name is Kevin Montalbo, Content Editor at Toro Cloud. In this episode of cocktails, we talk to one of the foremost thought leaders in the agile software development space as he shares with us the concept of energized work, the evolution of team rooms, applying agile methodology by looking at complexity in large-scale enterprises, and his advice on how organizations can invest in agility.

Joining us for a round of cocktails as always is Toro Cloud’s CEO and Founder, David Brown. How’ve you been David?

David Brown

I’ve been very well. Thanks, Kevin.

Kevin Montalbo

I love the cocktails in the background. Our guest for today is one of the first ten people to sign the Agile Manifesto when it was released to the public in 2001. In 2005, he was an inaugural recipient of the Agile Alliance’s Gordon Pask Award for Contributions to Agile Practice. The first edition of his book, The Art of Agile Development, was released in 2007, and became a much-loved classic. He went on to create other influential works, such as the Agile Fluency Model with Diana Larsen, and is a popular speaker at events around the world.

James Shore, welcome to the podcast! How are things?

James Shore

I’m doing great. Thanks for having me.

Kevin Montalbo

Let’s dive right in. You're currently working on the second edition of your book, the Art of Agile Development. What changes in the industry or discoveries have you made in the past decade that spurred you on to start doing a second edition?

James Shore

Well, the first edition came out in 2007, so that's, what, 13 years now. Boy, that seems like a lot. A lot has changed since then. The book was originally written to be sort of evergreen and I think it succeeded in that way, but nonetheless, I've learned a lot since then. The biggest thing that's happened since 2007 is cloud-based software has really just exploded and the dev ops movement came along. As a consequence of doing that first edition of the book, I actually got together with Diana Larson and we taught a number of courses based on that book. One of the outgrowths of that collaboration was the agile fluency model. So, it just seemed like after 13 years – also my publisher came to me and said, we're getting requests for it to be updated – it seems like it was time.

So, the new editions got a lot more about cloud and dev ops in it, it brings in the agile fluency model, talks a lot more about adoption and how that works. It's not going to come out for another six months or more, fingers crossed. We'll see how that comes. So, I've got a lot of writing still to do, but so far I'm really happy with it. I'm going through and I'm just touching everything on revising absolutely everything.

David Brown

Have you witnessed the agile process change or has it just been that cloud adoption and dev ops has introduced new implementations of the agile process?

James Shore

[Agile] is more of a philosophy or a way of thinking about software development.

First, agile is not really a process. It's more of a philosophy or a way of thinking about software development. Martin Fowler has his description of agile, which he calls the essence of agile development, which I think is really dead on. He says “it's adaptive rather than predictive” and “people-oriented rather than process-oriented”. I don't think that's really changed. Martin actually came up with that characterization before the term agile was even coined. I think it's still true today. I think even today, nearly 20 years after the Agile Manifesto was written, there were a lot of companies who just don't get it. Like they don't get that core idea of adaptability rather than predictability and people orientation over process orientation. So, I would say the core ideas of agile, no, they haven't really changed, but agile is a big tent.

Like, what we do in agile today is not what we did in agile in 2000. We're constantly bringing in new ideas. So, Lean Startup by Eric Reis has come in. There's been a lot more work in the area of how do you do large scale development with Agile? So, these ideas are always coming in, dev ops movement, of course, is taking XP’s ideas of continuous integration, which is another thing that's sort of been misinterpreted over the years and bringing it to cloud development and doing continuous deployment and delivery. So, I would say agile is evolving, because it's always taking in new ideas, but the core ideas haven't changed. And I would say the vast majority of companies that say they're doing agile have yet to adopt those core ideas.

David Brown

Why would that be?

James Shore

Becoming agile as an organisation requires changing habits, culture, and ways of thinking about your work.

It's hard and it's a culture change and it's so much easier as somebody in a large company, who's been tasked to bring in agile, it's so much easier to hire somebody who says they're going to do it and they're going to make it easy. So, there are so many courses that you can buy and pay for that are two day courses where you'll get stamped with the certification afterwards. And some of them are really good courses, but there's only so much you can do in two days. You're going to get stamped with the certification. Then you say, “Tada! We're Agile!” But actually becoming agile as an organization requires changing habits, culture, and ways of thinking about your work. That doesn't happen in two days. It takes an in-depth, thoughtful approach that's at odds with people who have been tasked to do something they don't necessarily understand and want to just pay money for it.

Kevin Montalbo 

Speaking of cultural, you recently published an excerpt of your book, which is on energized work. Now it's quite refreshing to see a book about software development, Agile, that dives deep into energizing and motivating the workforce. Can you tell us more about your thought process behind the section of this book?

James Shore

That's a great question. I think that actually really ties back to what David was saying a moment ago because this idea of energized work, it goes way, way back to the very beginnings of agile and it's not my invention at all. I remember I said that Martin Fowler described the essence of agile as being people-oriented rather than process-oriented. That really shows in this idea of energized work. The idea that, so, prior to Agile, the big approach of the day was these processes. These big heavyweight processes, where the underlying theory was that if we have a well-defined process, it doesn't really matter who we put through that process. The processes are real assets, and we can exchange people as needed, who are notoriously unreliable and tend to do things like want money and quit and stuff like that. So, we're going to rely on our process to be our real asset, and we will just pump people through this process and we'll get really nice, consistent results. They did get consistent results, but I wouldn't call them nice because it turns out that people are really important for software development. We all know that! There was, oh five, six, seven years ago. I don't know there is, I don't know if I should say this on, on the podcast. Maybe you can bleep it out, but there was this movement called Programming Motherfucker, right? So, it's basically that core idea that how we do programming matters and the skill that we have as people matters and Agile made that really central. And one of those ideas from the first edition of Extreme Programming, which was the first, really big, popular agile method was the idea of 40-hour week which says basically you do your best work when you're not worn out.

And then Kent Beck who wrote that, who created Extreme Programming did some work in Switzerland and came to understand that not everybody thinks of work in terms of 40 hour weeks. So, in the second edition of extreme programming, he called it “Energized Work”. For the first edition of our book, we just took that name exactly because we're all about building on what came before. It's funny, you should mention this specific practice because of all the things that all the stuff we've written for the second edition, that practice that one had nearly zero changes. Everything else has had lots of updates, but that one hadn't nearly zero changes.

David Brown

You talk about process-orientated – because organizations from the industrial revolution, that's what they became really, really good at, right, is they became sausage machines and they are good at producing that sausage, whatever that product was that organization, but varying away from their sausage was something small business could do not necessarily enterprise, right? So, when you were talking about the Agile Manifesto and being an Agile organization implies change, being able to adapt to the market and change and be quick at changing and being quick to respond to change.

It's a total mindset change away from being process-oriented, which I think as an industrial revolution concept which worked super well for the last 150 years. So, now we're asking organizations to be fundamentally different. They are really good at making sausages, and now you want them to make pies and sausage rolls and everything else as well, right. Whatever the market fancies at the time. Is that a reasonable analogy?

James Shore

It is a reasonable analogy. I think a lot of early software development was based on this idea. I don't know how true it was, but I'm actually just been working on a section of the book today talking about some of the history of software development. There's this idea of “waterfall” that people say as sort of a straw man, but is how people actually thought how software should be developed in the 20th century. A lot of that was based on this sort of assembly line model of the industrial revolution and of the early 1900s. There's Frederick Taylor who had his scientific management, which was all about seeing workers as kind of lazy, undisciplined people who had to be controlled in order to get good results out of. But then, the manufacturing world actually passed the software world behind, left the software world behind and came up with the idea of Lean Manufacturing, which was introduced by Toyota with the Toyota production system. You've probably heard of Just-in-Time manufacturing. When I was growing up, this is just betraying my age a little bit, but you were to order something on, well there was no online, if you were to order something from a catalog, this was like the internet, except it came on a paper and in the mail you order something from a catalog, everything would say allow six to eight weeks for delivery. And now, of course, you order something online and you get it within a couple of days, or you're pissed off. That's because of these short supply chains and the innovation that's been happening as a result of Lean Manufacturing. Agile is actually taking a lot of the same underlying theory and adapting it to design rather than manufacturing – to software development, rather than manufacturing.

David Brown

The phrase agile is used in a much broader context these days, talking about becoming an agile organization. You don't talk about just software development and an approach to software development. You're talking about a mindset for the organization itself and the ability to be listening to the customer and the market and your employees and business partners and being quick to respond to the demands of those people and the change that they're undergoing as well, everyone's undergoing digital transformation, yourself included. But you have to take into account the transformation that your customers, employees, and business partners are also going through. Right? So, software’s underlying, we talk a lot about that and facilitating it, but hasn't the phrase also been used in a much broader context in terms of the way organizations are thinking of themselves?

James Shore

Yeah. I think it has. I'm not sure what to think about that. You talked about how these ideas are really more, something that's easier for small businesses and medium businesses to accept. And that's certainly, it's something that we saw in the early days of agile. It started out and it was all the early adopters and innovators, generally the smaller companies that really embraced agile, but they had so much success it attracted a lot of attention. You asked me earlier why aren't more people doing agile as for real, as opposed to just using the name, because it's easy to call yourself agile, right? There's no trademark, you can just smack it on the door. You're done. “We're Agile! Congratulations!” That's what I see with a lot of these bigger companies, it's now the term is digital transformation, but it's still the same old stuff. It's: “Let’s take these agile ideas and actually use them to transform our business”. That's really, really hard because it takes cultural change. And that is something that big companies are not optimized for except for a very, very few.

Kevin Montalbo

Speaking of a change. Let's dive into this a little bit more deeper. There's another excerpt that you published recently, and it's about this concept of the team room. Now I understand that the team room can either be a physical or virtual, but with a global pandemic forcing people to work remotely from home, has the approach towards collaboration within this team room? Has it changed from when you initially conceptualized?

James Shore

Oh yeah, no kidding. The pandemic of course has been a real change to the system for everybody. And it's actually impressive how much I think we're all able to weather it. I think in software development, we've got that capability much more easily than more physical occupations. The first edition of the book was really clear. It said, this is something that we know how to do when you're in person, and we don't know how to do when you're remote – but that was 13 years ago. And actually before that first edition was published, I actually created a startup with a friend of mine called “Card Meeting'' with the intention of let's make team room work when you're remote. That startup failed, as they tend to do. But now the same ideas you can see, and I'm not going to claim to be the instigator of these ideas, because it's pretty obvious, but the same ideas of having a collaborative workspace where you can move cards and stickies around as if you were in person is now found in tools like Mural and in Miro.

And they, unlike us, have actually succeeded. We had a product that basically did that, but it was never more than an alpha or beta. They've actually made it work as a real proper production project or product. So, we have the tools now I think to make it possible to interact in the same kinds of ways that we had when we were physically together, this rich communication where you can have multiple people collaborating on the same discussion board, the same whiteboard at the same time which is really, I think, essential for an agile style of collaboration. So, that's something that if the pandemic hadn't come along, I don't know if I would have leaned into that as hard in the book and the second edition of the book as I am. Now, it's front and center. The first edition said, you really don't know how to do remote. We don't know how to do remote well in an agile environment, the second edition I was going to say, well, we do know how to do it. And here's some tools. And I was going to put that off on the side of one chapter somewhere. But now, every single thing I write when I say here's how you do collaboration with agile, they’re side-by-side, here's how you do it in person. And here's how you do it remote. And it's actually not that different because we do have the tools now to make that remote collaboration possible.

Kevin Montalbo

All right, so let's move on. Let's talk about applying agile. Let's talk about applying the agile framework in a large scale enterprise. So, in one of your previous articles, you described how complexity in software development is inevitable as we cannot eliminate it, but we can move it around. So, from the monolith and microservice approach to using nanoservices and a hybrid approach by a mono repository, how do you think large scale enterprises should look at and deal with complexity?

James Shore

That's another great question, because I think complexity, this is Coding Over Cocktails, right? So, we've got an audience of programmers. And I think the essence of programming is managing complexity. That is really what programming is all about. And that's what software design is all about because the computer of course doesn't care. We could write an assembly. We could write in C++, we could write in Java, we can write in Python. It doesn't matter to the computer, under the covers, it's all just byte code, binary, right? The programming is for us humans. And what we're doing when we program is we're managing complexity. Every time you write a function, you're managing complexity. Every time you write a class or a module, you're managing complexity, do you choose to use a functional language versus a procedural language – it’s because you think the functional languages, it makes it easier to manage complexity.

So, this is even more true in a large-scale environment, because the problem is I ran into this when I was a kid, when I was 15 or something. One of my first big programs, D&D character creator, because I'm a geek. And every geek's got to write a D&D program. So, D&D character creator written in Applesoft, basic, which was a lovely programming language that had two character variable names. Well, technically you could write as many characters you wanted the variable name, but only the first two characters counted. It used line numbers. So, it was gotos and gosubs, right? And so I'm programming along, slinging line numbers gosub 10,000 cl string, dollar assignment string, a cl dollar sign, AN, AN percent, all this stuff. And I guess I had gotten called for dinner or something like that, because literally between one moment to the next, I went from all in my head ‘till I had no idea how this thing worked.

And that was because it had outgrown the level of complexity my brain can handle. All software outgrows the level of complexity your brain can handle if it gets big enough. And large-scale software development by definition is too complex for one brain to handle. So how are you going to manage that? And that is where these ideas like microservices come in. But I think they're fundamentally flawed. Microservices aren't fundamentally flawed. They're fine. But as a tool for managing complexity, that people are using it like a silver bullet, right? Because if you can't build a monolith with good management of complexity, there is no way you can build a microservice mesh with a good degree of complexity. All you're doing is you're kicking the can down the road. But if you thought dealing with a monolith that was poorly designed was bad. Just wait until you get a microservice mesh that's poorly designed. Now you've got all the problems with the monolith of not knowing how things are supposed to fit together and being able to link it all together, combined with distributed programming, which is God-awful to begin with. The one saving grace is it is a lot harder to use stuff like globals in a microservice environment. And that is the source of a lot of mistaken coupling. But I think that over the next couple of years, we're going to start to see, microservices have been around long enough that they've hit that magic level where they're no longer new code. They're now legacy code. And that's what sucks. It's not the language, it's not the design. It's the fact that it's old and you didn't write it. We're getting to the point now that we're going to start seeing some major reports of major problems with microservices, just because the people who couldn't design monoliths also didn't change their design skills when they went to microservices.

David Brown

So, what is the process to manage the complexity? What do you recommend to manage that process?

James Shore

I don't have a simple recommendation. It is fundamentally about design and in a large scale system, when you're thinking about which teams you have, what you're really thinking about is what is the design of our software? Because of course, there's Conway's Law that says the design of the software reflects the design of the organization, and vice versa. So, when you're thinking about how you want to set up your teams, you need to also think about how you want to set up your software. And that means minimizing coupling between teams, having clear interfaces between teams. Being really crisp about how teams depend on each other, so that when this team makes a change that affects this team, this team can be aware of it. And if you've got a poorly designed microservice mesh where you don't know that stuff, and this team makes change, and it affects all of these other teams – now you've got a problem, especially if they don't know it.

David Brown 

It always comes back to people, right? Whenever we talk about this stuff, it's always coming back to the people.

James Shore

That's right.

David Brown

Team organization, how you structure the stakeholders, whether you get them into the same room together, whether you make them as one team, whether you make the transparency communication between teams, that all comes back down to people.

James Shore

Yeah, absolutely. And one thing I would add to that is that people think organizational leaders think about the decision of what teams to create as a management decision, but because of Conway's law and because of this management complexity, if you have interdependent teams, if you have teams that depend on each other to ship a product, you don't have a management problem, you have a software design problem. And you've got to put people who are experienced in software design at the head of deciding how those teams are broken up. For example, it's really, really easy and common to say, well, we're going to have our user experience, our user interface, front end people here, and our backend people here, but what have you just done? Everything that the software does now involves these two teams communicating. It would be so much better if we said, well, no, we're going to take this one thing that we have, and we're gonna split it into admin with front-end and back-end and log in with front end and backend, and then, and a profile with front end and backend.

If you do that now they're still depending on each other, but you don't have every single thing they do depending on every other team. So that's the same thing. You know, when you think about your front and backend coupling, don't create teams that are coupled in the same way that those are. Make your teams cross-functional and that's just the beginning. There's a great book, “Team Topologies” that I have not read all the way through, but discusses this sort of stuff in detail that I have a really good feeling about and has come very highly recommended.

David Brown

Kevin, our next guest guest, Ruth Malan.

Kevin Montalbo

Yeah. We're going to check her out. All right. Okay. So let's circle back to your book because we're talking of large scale enterprises here where you advocate the idea that organizations need to buy into the underlying agile philosophy by investing in agility. So, as we're down to the last month of 2020, so we're recording this December, our listeners might want to have an idea of what they can prioritize as they start the new year. So what's the best advice apart from buying your book, of course that you can give them when it comes to investing in agility?

James Shore

A decision to become agile is not a decision to take lightly.

Well they should check out my book at least until it is published, I'm doing an open review. So, you can actually see everything I'm working on for free and give your feedback on the mailing list because I'm an agile guy. I take feedback and I act on that and I iterate. So, it's all available online right now. If you go to jamesshore.com/s/aoad2 for Art of Agile Development Second Edition. So, you can see this stuff right now. And there was a chapter called choose invest in agility that has these ideas in it. I think all those investments in that chapter are important, but if I had to narrow it down to just a few, I would start with a buy-in at a decision to become agile is not a decision to take lightly.

It's a decision that requires people to change their behaviors and their habits. And that's the team members, but it's also managers and stakeholders. And you need those people to go into it, knowing what they're getting into. So, start with the buy-in. And then from there, remember Martin Fowler's essence, it's people-oriented rather than process-oriented and it's adaptive rather than predictive. So, it's going to take the people time to learn. So make sure to account for the fact that your productivity is going to dip where you're going to learn, and it's going to cause some chaos and you should manage that chaos well, especially in a large organization. Put your people together on dedicated cross-functional teams that don't have a lot of dependencies on other teams as we were talking about a moment ago. And when you're thinking about how you do governance in your organization, do you assume a predictive environment? Do you assume that you're going to say that this is a project that's going to take a year and it's gonna ship this thing at the end. It's going to take this long and cost this much. If you do, you don't want Agile, you want a waterfall style process. Decision to use agile is a decision to say, we think adapting is more important than predicting. If it's not, don't use Agile, it's not going to do any good. It's not meant for that. And there's more, but I'll, I'll stop there.

Kevin Montalbo

Yeah. We'll give them the chance to read on your book or even the excerpt that you're publishing on your Twitter feed.

David Brown

Sure. Thanks so much for your time today. Our listeners can obviously follow you at jamesshore.com. And your Twitter handle is?

James Shore

It's "@jameshore," pretty easy.

David Brown

Awesome. Thanks so much. And good luck with the rest of your book. We'll certainly be following the excerpts as you publish them.

James Shore

Yeah, my pleasure. Thanks for having me on and cheers.

Kevin Montalbo

Thank you very much, James Shore, for being with us. To our listeners, what did you think of this episode? Are you taking on any agile development methodologies for your organization? Let us know in the comments section from the podcast platform you’re listening to. Also, please visit our website at www.torocloud.com for a transcript of this episode, as well our blogs and our products. We’re also on social media, Facebook, LinkedIn, YouTube, Twitter, and Instagram. Talk to us there, we’ll listen, just look for Toro Cloud.

Again, thank you very much for listening to us today. This has been James Shore, David Brown, and Kevin Montalbo at your service for Coding Over Cocktails.