"What I learned from deploying thousands of APIs" with Matthew Reinbold

Kevin Montalbo

Good day, ladies and gentlemen of the internet! My name is Kevin Montalbo. And of course, joining us from Australia is Toro Cloud CEO and Founder David Brown. Hey, David, what's going on?

David Brown

Good morning, Kevin. Beautiful day here in Sydney.

Kevin Montalbo

That's fantastic. And now let me introduce our guest. Our guest for today is a platform thinker, digital transformer, API design architect, people manager, conference speaker, developer advocate, and writer. Since his first web projects at IBM in ‘99, he has built and maintained software governance teams at multiple Fortune 500 companies. His passion for building successful software developer cultures result in the continuous delivery of business value, even at scale.

He is currently the Director, API & Event Streaming Platform Services at Capital One. And he writes about the transformative nature of the API industry at https://tinyletter.com/NetAPI Notes/archive. He also maintains a curated list of in-person API community gatherings at https://webapi.events/.

Joining us for a round of Cocktails is Mr. Matthew Reinbold. Hey Matthew, great to have you on the podcast!

Matthew Reinbold

Great to be here! Thank you for that intro.

Kevin Montalbo

All right, so let's dive right in. Can you share with us what you do at Capital One as director for the API and event-streaming platform services? That's kind of a mouthful.

Matthew Reinbold

It is, it is. And it takes a while to describe at parties, so you think I'd be well versed into explaining this. But when I joined Capital One 5 years ago, it was to help develop their newly formed API lifecycle management process, basically helping developers have success and grow their design practice over time. That turned into managing a team and becoming a platform and turning it into the thing that we have today, which is several thousand APIs registered, doing approximately a half a billion responses a day. It's almost entirely for our internal processes and our internal development. And it's all about making sure that the APIs that we develop are consistent, they're cohesive, that they add business value and don't add to the overall complexity of modern software development at a large enterprise.

Because of that success, we moved into different areas. You mentioned event streaming. We've had some work to do there, but my passion at the moment, what I'm really interested in looking at is how we can take the success that we had in the API governance side and spread it around, help make digital transformation be a turnkey for the rest of the organization. Whether that's around data, whether that's around events, whether it's around a host of a number of other things, how do we take that success and make sure that we're having success in other areas? And that's a hard thing to do. Regardless of your source that you're looking at, you might have 70 to 90% of enterprise digital transformation efforts fail, so there's a lot, a lot of work to be done there.

David Brown

That's really interesting. I'd love to dive more into that and your passion, where you think that's going to take you. Before we get there, can you give us an overview of API governance. What is it and why is it important?

Matthew Reinbold

Yeah, absolutely. So I come at API governance a little differently. Many people immediately jump to rules and police enforcement on a platform. I stepped back, and I asked, "What are we actually trying to do with a governance program?" And to me, it's about how we relate to each other. So imagine a road system, a street system. The fastest way to go from point A to point B is to drive as fast as possible, ignore all traffic lights, ignore traffic signs and take the corners wide, do all these crazy things if you are the only one on the road.

But the fact of the matter is in large enterprises -- in medium-sized companies at this point -- we're not the only team that's trying to get from point A to point B. The fact of the matter is, there's a lot of people that need to go from where they are to where they need to be. And we need to make sure that the means by which they are getting there is done so safely and can accommodate the most people possible. So for me, governance is how do we design the systems? How do we design the networks so that we get the most people travelling to the most number of places in the safest way possible? Yes, that might mean rules, but rules are a tactic. It might be an education. It might mean a variety of other different things. But I'm trying to, in my work, expand the definition of governance and get people to see beyond the very top-down command and control type of delivery systems that we oftentimes see in enterprise environments.

David Brown

I understand. So, that’s a nice metaphor there with the single car on the road there. So basically, we're talking about the discoverability, standardization, making sure that APIs are reusable, secure according to standards and there's some sort of oversight on how this is all done. But getting into more specifics, what are the factors that you need to consider to enable discoverability and standardization and security? What are the sort of things are you defining in your governance?

Matthew Reinbold

Those are absolutely meaningful outcomes to a governance program. But for those starting out, it really is about how do you get to the point where you decide that discoverability is important? Who's in the room? What's the mechanism by which you are deciding that you're going to pursue those shared goals? So, at some point, somebody decided, "Hey, discoverability would be a good thing." In some cases, registration might be a really good thing. Every time we turn over another rock, we find another microservice that poses a risk for our ingress and egress. We don't know what one hand is doing. We don't know if we're actually violating some regulatory compliance mechanism.

So in these particular cases, governance starts at deciding whose responsibility it is to move the company forward. Who decides what the rules are? How do we communicate those things out? How do we create virtuous feedback cycles so that the people that are adhering to these statements can feedback into the system and we can co-evolve these things over time and make sure that we are staying true to what we need to be delivering? And that we don't end up publishing, for example, a style guide that is out of date the minute that we happen to publish it and then subsequently isn't used. How do we avoid systems that try and lock enterprise environments in stone and that we can't evolve because we had an idea of what we should be doing at this moment in time, but we aren't able to react to new frameworks, new trends, new languages that come around the bend tomorrow?

David Brown

That's an interesting concept because you're talking about defining guidelines and standards. So, how do you avoid limiting innovation within an organization when you're applying API governance?

Matthew Reinbold

How do we create a repeatable safe process so that we can continue to evolve?

It goes back to that process that I'm talking about. Rather than focusing on having the perfect API Style Guide, for example, the emphasis needs to be, “How do we create a repeatable safe process so that we can continue to evolve?” Because we're going to acknowledge straight-up we're not going to achieve perfection. Like, perfection is the opposite of being done. And so if our goal is to have the perfect set of rules for all time, we are going to fail. It's not possible. So, the emphasis needs to instead be on continual improvement, like that's how we operate our sprint cycles and our Agile teams.

That should also be how we think about governance. We take this first step, we look and see what the ramifications of that step were. We look and see whether it's achieving the goals, and then we take another step, and another step, and another step. And so this kind of iterative, incremental type of governance is really what I try to espouse. So, when it comes to the rules, yes, eventually you will get to the point where you have something in your organization that says, "Okay, maybe this is our standard error object. For the sake of our cooperation and our communication, we are going to adopt a standard error object so that we can leverage our experience with one API across our entire portfolio." That's really good. But before you get to that stage, you need to have that underpinning, "How do we get there?" Who are the people in the room? Are we doing a straight-up "majority rules" type of system where if people think this is a good idea, it passes? Are there other mechanisms by which we can figure this stuff out? How do we communicate it out? How do we grandfather in existing systems that may not conform to this? All of those are the type of questions that need to be answered by governance. And then once you get those rules in place, then you get to management and enforcement.

David Brown

As you say, once you got those, you've collaborated with stakeholders, established some sort of process that is going to work for everybody and foster innovation and not to be too restrictive, you then want to start applying the models and the processes. Are there any sort of toolsets, processes, models that people can adopt to apply these strategies?

Matthew Reinbold

Yeah. So, so much of what I do and what my team does is around communication. And even when it comes to API design and API governance, it's how do we communicate with each other? We found over time that "Jobs to be Done" is a wonderfully clarifying model to help teams get very crisp and precise on what they need to build and what they need to capture in that API definition that we subsequently used as a means to communicate across teams. We also use Event Storming quite a bit. Domain Driven Design is wonderful, but it is a bit heavy for a lot of teams.

David Brown

What do you mean by that?

Matthew Reinbold

What do I mean by that? So on the surface, Domain Driven Design is really straightforward to understand. It's using the language spoken as a means of identifying boundaries between different abstractions. But to follow that process through to the letter, you start getting bogged down in a lot of specialist terminology and a lot of specialist facilitation that requires experience and knowledge. It's not something that any team can just pick up in an afternoon and have success with. And this is something where we've used techniques like Event Storming to get more of that rapid, Agile approach to defining some big principles, aligning teams and getting agreement on what they need to be building, and then subsequently capturing the most salient points in an API description that can be useful and long lasting to the rest of the organization.

David Brown

Run us through events storming. How does that work? What is it?

Matthew Reinbold

So, it's a technique that came about for exactly the reasons that I described with Domain Driven Design. It needed to be lightweight. It uses the idea of physical Post-It notes, which we've had to experiment a little bit in this Covid time to virtualize. But the idea of identifying from the beginning of a process to the end, what are all the incremental little things that need to happen along the way? And by visualizing it, by putting it on these little stickies, what you're forcing the stakeholders in a room to do is to come to agreement. And regardless of the size of the effort, I guarantee what happens when you get these stakeholders in a room is you end up uncovering assumptions. You end up uncovering aspects of language that were taken for granted.

So, somebody will put a sticky note up and they’ll say, "Okay, in this step, we call this system and we get back this magic token," and somebody else would go "That's not how I thought that worked." And then you have a discussion. So, it's a forcing mechanism to get those stakeholders to recognize not only all the pieces but come to a common language. And once you have that common language, now you can start slicing and dicing and saying, "Okay, this needs to be a Microservice", "This needs to be an API", "This can safely be abstracted away, and we never have to think about it again." So on and so forth. But it's a forcing mechanism to get people to surface those unknown assumptions and make them explicit so that we can actually model things correctly.

The worst thing that can happen during the course of API design is unsurfaced assumptions. Because that's when you get the API design that goes out the door and that client uses it the first time and says, "That doesn't do what I needed to do." And now you have a versioning problem.

David Brown

Yeah, got it. Before we move on to your thoughts on how you can apply this to digital transformation in the future, with API governance, we mostly think of it within an organization. But now we're starting to look at regulatory governance such as open banking standards. Do you see any more regulatory compliance according to APIs and their standards?

Matthew Reinbold

I do. I think we're probably long overdue for a number of privacy and security and regulatory approaches to the way data is handled in big companies. Here in the United States, we have something called the CCPA or the California Consumer Privacy Act. Because California is so large and such a significant percentage of our economic activity, the rest of the nation typically ends up having to do what they pass. Worldwide, there's things like GDPR that impose certain behaviors around data and facilitate certain actions around data, and I do think that it is going to be increasingly vital that API designers are incorporating aspects of this in their designs.

Now, we're not going to be able to have every API designer knowledgeable on every rule in every regulation worldwide. That's just simply not going to happen. But that is something where API governance can play a role. You can have a group who's already well-versed in marrying the enterprise expectations and the enterprise needs with the team's understanding of their use-case and the team's understanding of their particular output. And together, through collaboration, you end up with something that is both enterprise-safe and desired, and fulfills the use case that the team set out to build. So, you end up with something that's greater than the sum of its parts when those two groups can work together. But it's going to be vital that the API governance aspect is able to be aware of these things, work with the proper groups within their organizations, and account for those in the subsequent designs that they're consulting on.

David Brown

Okay. You've obviously had enormous experience over the last five years building several thousand APIs. Many people would have thought that you've already nailed digital transformation. But you started out this conversation saying you now want to apply these learnings and principles to other digital transformation-related projects in your organization. So, let's start out by talking about what is digital transformation to you and how do you see it evolving? And how do you see these learnings that you've obtained applying to those?

Matthew Reinbold

Digital transformation is really about behavior change within an organisation.

So, digital transformation has become one of those words, almost like "Microservices" where it means a lot of different things to a lot of different people. To me, digital transformation is all about "Can you use digital technologies to create new and novel experiences and services for customers, like just doing what you already did before?" It may be semantics, but I call that digitization. Digital transformation is creating something new at the end of the day.

And so, as we look at our internal organizations and the increasing pace in which new technology is causing marketplace disruption and marketplace challenges, it's absolutely incumbent on organizations to be able to react to these things, if not lead. And in order to do that, it's vital that they understand the various components of digital transformation. Whether that's having the correct operational platform underneath, whether it's being able to incorporate feedback in a rapid manner, it's allowing industries of all stripes and shapes to behave with the cycle time of the software company. And that's that's where we're at. Like 20 years ago, 30 years ago, you could safely have an industrial company that does the same thing day in and day out. But today, all companies are software companies, and therefore all companies have to have the type of speed inherent with a software driven life cycle, and that presents a lot of challenges. How do you get the information from Bucket A to Bucket B safely but at speed? And that's a challenge.

So, as enterprise leaders are looking across the organizations and identifying the opportunities for rapid change and rapid modification of processes and controls and approaches, it's incumbent that the governance that is applied is not a gate, but it's an aspect of how they empower people to do things better. As I've gone down this route, I've become increasingly aware of how much digital transformation is really about behavior change within an organization. It's not enough to simply do what we did yesterday using new tools. It's about changing how we behave when given certain problems or certain opportunities. And behavior change for people is really, really hard.

David Brown

Yeah, most people are resistant to change, right? That's a culture thing.

Matthew Reinbold

Governance applied is not a gate, but an aspect of how to empower people to do things better.

Yeah, and so your API governance needs to be as versed in how to do organizational transformation, how to do organizational behavior change as it is with the bits and bolts of API nuance, and that's a real challenge. So, many of the people that are put into positions of governance within an organization come from a technology background or even sometimes product, or sometimes risk. But the governance programs that I've seen successfully rolled out typically are led by people that understand this "human" element. They’re as well versed in things like negotiation, conflict resolution, even communication, as they are, the particular technology that they might have done five, ten years ago.

So, it's going to be a challenge for a lot of folks, it's not a set of skills that I think are practiced. But increasingly, as we need to incorporate more digital transformation in order to be successful in our businesses, individuals that can cross these disciplines and have these types of skill sets I think are going to be increasingly in demand.

David Brown

Huge demand. I mean, how do you change culture? Like, what would be your approach to foster innovation with an organization to develop new digital products and transform the organization? You said you have to approach that before you get to the governance aspect, right? So where do you start on that?

Matthew Reinbold

That's a big topic. The first, briefly is, understand the landscape. Before you can tell people where to go, you have to understand where you are. And so that's why it's so difficult for people like consultants to be plopped into a company and expect them to have success because they're coming in with their own experiences, their own assumptions. And they may not understand the lay of the land. So, having a deep understanding of how decisions get made, who has authority both formal and informal, how does power manifest, and what are the incentive structures that people respond to. Those are not universal things. They do vary company by company, and therefore it's absolutely vital to understand that.

Second thing, once you start trying to make momentum happen, it is always vital to shrink the change that you're trying to accomplish. I've seen way too many efforts, declare some grandiose, "burn the boats" type of initiative like, "Everybody, for all time, is going to do this thing and only this thing". And as you might imagine, the amount of pushback to something like that is as absolutely proportional to the size of the change that is being asked for. It might be necessary, but in order to get traction, you have to build positive momentum. And the easiest way to do that is to shrink the change being asked for.

When I go into a company or I advise the company that's just getting started with API governance, they have a whole suite of things that they're trying to accomplish. And I say, "Well, what's the least controversial thing that you could mandate right now?" Like, does everybody have a health route? And they might look at each other and after some debate, "Oh, you know, maybe we do need a health route, but that's so insignificant." And I said, "Exactly, start with the uncontroversial stuff. Ratify your process, whatever the means is - forgetting that thing accepted and communicated and monitored and policed - whatever that tiny, tiny thing is, have it be uncontroversial because you're still figuring out how all of this works. Who are the people at the table? How is this going to play out?" Once you get that decided, you will better be able to take on the bigger rocks, the harder things.

If you start with those hard things while you're still trying to figure out what the relationships are and how to work with the people, it's not going to be pretty. So, shrink the change. After shrinking the change, you want to script the critical moves, so you have a big initiative. It probably sounds great at the 50,000 feet view, but for the person working in the trenches, they don't understand how they get from where they are to where they need to be. You have to script the critical moves. People are probably not pushing back out of malice there, probably pushing back because they don't know what the connecting steps are. So, script the critical moves. And finally, communicate, communicate, communicate, communicate.

I've seen way too many well-meaning efforts that do really hard work to figure out a complex problem and nobody ever hears about it. And when I follow up with those engineers or architects and I say "What happened to that?". They said, "Well, we posted a page to our wiki," and it's like you just expect people to stumble upon that? You just expect them to know that it's there? Even when you think you've told it 1000 times and you're blue in the face and you are so sick of the message, I guarantee there's probably people that have not heard it yet and that are not on board, and they just have to hear it probably hear it several times.

David Brown

It's amazing, time and time again, when we're talking about digital transformation efforts, it comes down to people. As a technology company providing solutions, you love to talk about the technology, but actually what we end up talking about is culture and people.

Matthew Reinbold

Absolutely, yeah.

David Brown

Matthew, it's been a pleasure to have you on the podcast. Amazing experience, and it sounds like you have many more amazing experiences to come at Capital One. If you have some social channels which people can follow, care to share those?

Matthew Reinbold

Yeah, absolutely. I think probably the nexus for everything is my website, which is unimaginatively named MatthewReinbold.com. Just look at the show notes if you need a spelling for that, but it's MatthewReinbold.com. From there, you'll find links to Twitter. I'm fairly active on Twitter. You also find links to the newsletter that Kevin mentioned, so on and so forth. But if you go to my website, it's my name dot com.

David Brown

Excellent. Thank you.