The year 2020 saw an unprecedented and transformative shift for the IT industry. The lockdowns forced us all to shutter our offices and work from home, which called for IT departments everywhere to rise to the occasion and use technology to become more agile, responsive, and resilient.
As we reach the end of the year, it becomes vital for us to look at how digital transformation is currently being implemented within organizations, and what businesses are doing to improve their IT infrastructure in 2021 and beyond.
Kevin Montalbo: Our guest for today describes APIs as the “ultimate do-over solution for enterprise IT” and in this round of cocktails, we discuss how organizations can establish a formal API program. We also take a look at both top-down and bottom-up organizational approaches towards digital transformation, and discuss the ALIGN, DEFINE, and DESIGN approach to implementing APIs.
Joining me today is TORO Cloud’s CEO and Founder, David Brown, good morning David!
Our guest is a highly experienced microservice architecture consultant with more than 20 years of experience in architecting, building, and deploying software products for Fortune 500 companies, as well as early stage startups. He is also the founder of LaunchAny, a consulting firm that specializes in digital transformation, API design, development, and cloud infrastructure. He is the author of the book “A Practical Approach to API Design: From Principles to Practice.”
You've previously discussed that when an organization undergoes digital transformation, they need to create a formal API program. What should be included in a formal API program?
James Higginbotham: First, it's important to recognize that APIs are really the ultimate do-over for an enterprise, whether they're producing a product or whether it's just for their IT department. It's helping the organizations to start thinking about the digital capabilities that they bring to their workforce, their partners, their customers. What I like to do is I like to look at an API program from eight core disciplines as part of that digital transformation process. The first is their program strategy: why it exists, and what they're doing with APIs. Second is the process and governance: how are they operating? What kind of checks do they have in place and workflows? The third is their portfolio management, how their APIs take the shape of the organization and vice versa. How are their APIs incorporated into the overall portfolio?
What kind of API styles are they offering? Rest, graphQL, RPC maybe even some async API, like web hooks and so on. And how does that all together, help people get things done. We need to look at discovering the adoption side. Is the API documented well and is it discoverable? Are we making sure that people can find the API as they need as part of that transformation program? Fifth, are they onboarding easily? Are they onboarding developers easily by that? How easy is it for a developer to get up and running with an API or how difficult have they made it, perhaps? We focus also on design and delivery, how they are designed, how we deliver the API through the standard SDLC, the software development life cycle. We look at the management analytics aspects, how they are tracking the APIs, their usage, what ones are really being used heavily and which ones aren't being used at all. Finally, we encourage product ownership of APIs. So how do we look at our APIs overall as products so that we don't just release them once and then forget about them and treat them more like one-off projects. But rather we treat them like products that we continue to release and improve, mature, and grow them over time.
David Brown: That last point kind of comes full circle back to where you started, right? In terms of determining from an organizational perspective, why are we doing this as part of a digital transformation strategy, but we may be trying to create new digital products out of our data or services or processes. Let's talk about the sort of organizational requirements from the perspective of people, rather than the technology in terms of getting everyone on board as to what the strategy is here and, and we're doing it and getting everyone everyone's buying it, what's the process there do you recommend?
JH: So I see a few things. One is making sure that all the executives are in alignment. Some digital transformation projects come from the executives down, but the other kind starts off from the ground up, where there's a need and a recognition, but we haven't fully experienced the full buy-in of the executives. So, being able to hone the message, understand and show, through execution, the ability to transform the organization in some particular area, and then reincorporating that messaging throughout.
The other thing is training. Making sure that teams are trained up on what this means. So, we'll oftentimes fall back into the patterns that we always have, the way we build things. We won't initially look for APIs. We're going to look at data stores. What's the data store that has the information I need and who do I need to talk to, to get access to it? And we don't look for those APIs. We don't look at developer portals.
One of the other things that we see is having that executive buy-in and having developer portals shifts the conversation quite a bit. I've been in organizations where they've had all these different APIs and they weren't gaining adoption. They would come to me and say, well, we've got some API started, but no one's using them. How do we help with that? And so it's evangelizing that throughout the organization.
DB: It’s about discoverability. That developer portal you're talking about is about giving some accessibility to the API so that people within the organization can discover what's available.
JH: It is. There are multiple personas that need to look at a portal. We often focus on the developer, where's the reference documentation that I need to write the code to integrate with this API. That's super important. But one of the factors that is not always thought about is the evangelistic capabilities that the portal gives. One of the clients that I had, they went from a few APIs and a developer portal that was behind a login screen inside their API management layer that only about five people had access to, to a portal where VPs that had never written a line of code before were running around, showing the portal and being able to demonstrate not just the reference documentation, but what are the capabilities that the APIs are delivering to the organization and to the marketplace.
And they were actually evangelizing all over the organization. So every time they would travel to one of their other, office locations across North America and even overseas, they would be showing the portal and being able to see, see what we're doing today. Oh, look, actually, there's a new capability that's been released that I didn't even see last time I pulled this up and they become evangelists and help spread that word. And that is absolutely instrumental to a digital transformation effort. Otherwise it kind of stays inside of IT and gets locked inside of it and siloed and it doesn't break out of that.
DB: For those that are not familiar with developer portals, what kind of things would you expect to see in a developer portal?
JH: Some of the primary things we see is the API documentation, how to use an API. We oftentimes have documentation on how to get started or how to register your applications so that you can access the API, but really developer portals can take on a life of their own and they really should be treated like a product like anything else. And that means understanding who's going to be visiting the portal and bringing content to bear that helps people use that portal effectively. And that means helping people understand what exists, you know, what kinds of things already exist there and keeping it in a very high level form of discussion or content rather than something that's very technical so that the non-technical people in the team can go to the portal and reference it and see what kind of APIs do we have, or “how do I use an API?”, “how do I get started?” They also will oftentimes have documentation for style guides that tell people kind of here's how to design your API so that they all look consistent. There'll be referenced documentation on how to actually use them. They'll oftentimes have examples as well, and even reference applications you can download or view or interact with. And documentation that you can interact with as well to try out the API. So it should be, you know, kind of multifaceted, not just focused on, on the few developers in your organization that may be needing to use it.
DB: Speaking of users and giving accessibility to the APIs, there's obviously a challenge associated with permissions and access to data because IT's natural tendency is to restrict access, right? So to provide the minimum level of permissions. And so if you're creating an API, you might tend to want to restrict the number of people that are allowed to access that API. But when you're talking about a large dispersed organization, how do you balance that challenge of providing only that access that is required from a security and privacy point of view to mass adoption and digital transformation and agile organization that can discover these APIs and respond quickly?
JH: That it is a very big challenge and a lot of the organizations I work with are dealing with, with anything from PII data to regulatory requirements that restrict what they're able to do. So they often do a few different things. One is they categorize or classify their APIs as to what kind of PI data, if any, it may be offering. And so they'll have a base level of acceptance that if someone onboards with an API one, they're going to do it in a sandbox environment, not production. And they're working with an API that doesn't have those kinds of rigid requirements. They don't offer any kind of that concerning data. The other thing that they do is they oftentimes partition their APIs. So they may partition them into different portfolios. Some of those portfolios would be private and would be only used for internal purposes. And furthermore, they may be restricted inside of certain data centers that have certain types of compliance and auditability the others may not. And they'll use those kinds of techniques to try to keep the organization as open as possible, and then have the kinds of typical approval processes you would expect for the more restrictive APIs. That way they're encouraging APIs to be built and deployed. They're reviewed for the types of data that they're exposing and then they're categorized and partitioned accordingly.
DB: There's something else I want to dive into as well. So something you mentioned earlier about API adoption occurring from both a bottom-up and a top-down demand approach. We often think of digital transformation strategies as being driven by board level decisions or executive decisions, operational efficiencies from management level or new products from management level, but in actual fact, a lot of it is demand driven, right? So the bottom-up approach where you just have developers trying to solve problems on a day-to-day basis, right? So, where is digital transformation realistically occurring in an organization? Because we think of it being applied as a strategic move and decision being driven by top level management – is that actually what's occurring or is it just people solving problems and using APIs to solve problems?
JH: It's definitely a bit of both. We see the need for the executive buy-in and the executive vision and execution from the top down to really drive the budgets necessary to take a large organization – say 10,000, 15,000 developers, 2,000-3000 scrum teams and try to shift them over to a new direction and transform themselves that takes kind of that top down and complete buy-in and large budgets, but they don't often start that way from what I've seen. I sometimes see organizations that, you'll see things like solution architects and mid-level managers that really understand the value of reusing APIs.
Many of those folks that are doing that saw the last couple of decades with our service-oriented architectures and SOAP and things like that, and they saw how you could either build services or in this case, APIs that were reusable and that we could leverage time and again, and then they also sold the one-off integrations that came up from all the different e-commerce solutions and supply chain solutions that required us to rewrite and rewrite new and different integrations every time. So, I think there's kind of a combination. And in the case of kind of the bottom up or mid-up sort of approach, they'll oftentimes take something that has already been budgeted and is in-flight. And they'll start to use that to demonstrate the values that can exist and the approaches and the shift in mindset that can exist. And then they take that upward to get a future follow-up and buy-in from the executives to get the larger budgets they need, which means they have to be a little scrappy in the beginning and until they can get everything officially adopted and at a larger scale.
DB: They're much more likely to get buy-in but at being able to demonstrate something, right?
JH: Absolutely. A demo is worth a thousand words. If you can take something as sort of talking about it and PowerPoint sliding someone to death, and you can come up with a vision, have a team to be able to execute that either in a skunkworks way or with some sort of buy-in, but in a very scoped and focused way for something you're already going to deliver that can help kind of overcome some of those concerns and demonstrate value early on.
DB: Particularly if you can render some pretty charts.
JH: Absolutely. If you can go and say, well, look, we built this API and this was actually fundamental in this feature for our mobile app and we're running millions of API requests per day through this, or per hour through this, or whatever it is that wow, that goes heaps into really being able to transform somebody as mind and say, well, if we can do it here, we can do it in the rest of the organization.
DB: Totally. Let's talk about the technology considerations from a developer's perspective. You've written about some of the essential considerations for developers when creating API specifications, run us through those.
JH: A lot of what I like to do is encourage developers to slow down a little bit. We are always eager and I'm a developer as well to kind of get in and start writing code. While code is really absolutely fun and exciting to build and see it all come together, it's oftentimes important for us to step back a bit. I encourage development teams to slow down and align, define, and design their API.
Aligning, meaning making sure that we understand what the APIs need to do, the outcomes it needs to produce, making sure that we're breaking down those assumptions, that every line of our code is going to have wrapped into it. Once we've aligned on what it needs to do, then we can spend some time defining what the API is actually going to look like. The operations that it's going to deliver, perhaps its name, its scope, its purpose, kind of at a high level. It's what I call the API model. It doesn't get into the specifics of how it's going to work with the protocol details, but it is going to get into the specifics of what it needs to do. Finally, the design phase helps us take that model and apply it either as a rest API, GRPC, graphQL, whatever the case is.
Maybe it's a combination where we use REST to make modifications and retrieve individual resources and then we use GraphQL for really robust query language support. So, align, define, and design helps developers to kind of go from what we need to build, to how to build it in a very simple and approachable way that's rapid fire to get us into the code as quick as possible, but to ensure that we're not having to throw a lot of that code away and scramble at the last minute to try to make a change because we misunderstood something that we could have understood from the beginning.
DB: Is there some sort of formal procedure they can follow in terms of getting requirements, liaising with stakeholders? Do you recommend some sort of formal workbook or procedure to log and document the requirements and to execute or mock the solution? What do you suggest?
JH: I do. I've been developing a process for some time around this kind of align, define, design approach. I've documented in a book that I have out currently. It’s a practical approach to API design as the title of it. You can find it at theapidesignbook.com and it sort of lays out a process of how to go from user stories or job stories into kind of what your API needs to do, the activities and the steps, and kind of helps you break that down. It's a collaborative approach. You want to get subject matter experts in the room, your domain experts, and pull it all together and start to work together. Even if they don't understand HTP some of your subject matter experts aren't that technical to that extreme or to that depth, that's okay.
To work through the process and to figure out what you need to deliver, and then to arrive at that design, that you need to deliver to the marketplace to meet what's needed. Now, it assumes that you're never going to nail that design right off the bat, it assumes that you're probably gonna have to make some changes, but it helps you hone-in on that design early. Then, you can use mocking or prototyping styles to turn your design into some sort of a small little mock version that you can use to both test out the design and to accelerate the development testing, and delivery of that API since front end developers can use that mock to start tying it into their front end code while the backend developers may be building the full API implementation, integrating with legacy systems, multiple data sources and so on. That's what I really recommend. It's really approachable, really lightweight. It's really driven and just kind of breaking things down into steps and then turning those into operations for our APIs.
DB: There was an important point there you mentioned as well, don't expect to get it right the first time. It is an iterative process, right? You're going to keep going through the process to improve and you can update APIs all the time.
JH: Absolutely. I like to remind people, APIs are forever. Once you have your first production integration, it's gonna be very hard to encourage that integration, that team that integrated your API to make a code change. So, you have to learn to design evolvable APIs so you can add to them, but you can't make fundamental changes. That doesn't mean however, that we can't go through a design process where we establish a life cycle for that design. We can allow people to integrate with it early, and are willing to absorb some of those changes that may need to be made as you discover more things. It's about setting expectations and having maybe early adopters be willing to take on a little bit of that extra burden to get access to your API first. Then once it solidifies it, it's forever.
You can add to it, you can improve it, but you're not gonna be able to make some fundamental changes. So, going through a design process helps you kind of get there quicker and then put it in front of people and allow them to try working with either the mock or the eventual evolving implementation and make those changes while you can, before everything is locked into place. You're only going to be able to add to it, but not make some naming changes or something that you decide you don't like down the road.
DB: When and how do you decide on the implementation, the choice between REST and GraphQL and GRPC?
JH: I like to do it in that align, define, design process. The define process helps us understand what the API needs to do. It's at that point where we start to transition to design that we can start asking some questions, who's going to use it? What are my developers capable of or comfortable with? What kind of performance needs do we have? If we need high-performance, GRPC is usually really good selection because it's taking advantage of HTTP/2 and some of the improvements there and some of the compression and other things that exist underneath the covers. You can have more persistent connections with GRPC so that when you make a new connection or when you make a new request, that connection already exists if you want, and you can have bi-directional communication and those kinds of things.
So it's, during that time, we have to determine what's the best way to do it? Who's going to use it, how are they going to use it? A lot of discussions I get into, particularly with developers that are very strict about, well, “This is the way I do things and this is the only way I do things”. They'll say, “Well, which one's best?”. My answer is the best API style is the one that's best for your consumer, not for the provider. The provider's job is to make the consumer's job as easy as possible. So, we have to put ourselves in the mindset of who's going to use the API and figure out what API style they might need. The other thing is to think about ‘and’ not ‘versus.’ In the improvisational world, we are taught the “Yes, And” approach to improv. The idea that we take something that somebody threw at us or said to us or something, and we add to it. And with the API style selection, it's the same thing. I find a lot of organizations still use REST, but they're using REST and GRPC, or they're exploring GRPC and graphQL. They're also looking at async API styles, like web hooks and server-sent events and web sockets to build a “push things back and forth” so that it's not just strictly request response. It's during that kind of a define-design transition where we start to ask ourselves, okay, how are people going to use this? Now that we know what it needs to do, how are they going to use it and what styles are right for these operations that we're going to bring.
DB: There's a lot of technical decisions you've just mentioned there, right? Design styles and implementation options. If you're new to API as a developer, where do you start?
JH: I recommend, first of all, starting with REST but really start with HTTP. Get really good at HTTP. Understand it. Now you don't have to have, like some of my friends do have memorized RFCs, practically. But, understand what HTTP can do because I oftentimes see teams building APIs or struggling to consume APIs, struggle with just using HTTP. We use it every day in our browser, but we sometimes forget about all of the amazing things built into the protocol, from caching and entity tags and detecting when something has changed and all kinds of things that are actually built into HTTP. Once you understand that, then you can get a little bit more advanced and you can start seeing, well, “how can I optimize my queries, maybe graphQL will do that for me”.
“How can I optimize my interactions or get bi-directional communication or avoid all the boilerplate code? Well, GRPC can help me with that”. So, getting a good understanding of some HTTP basics and what it can do will also save you time in trying to reimplement what's already in HTTP, inside your API, thereby making it longer to get to market and perhaps more difficult to consume by someone else down the road. So just starting with HTTP fundamentals is a key factor if you're going to look for anything to start. And then REST based APIs kind of come with their own sort of myths and misunderstandings. So, just understanding how to use HTTP, how to use the HTTP methods, the response codes, serializing things in JSON, maybe looking at some different media types or starting with something like JSON API, that's kind of there and opinionated and ready to go can help jumpstart that process so that you don't have to take it all in at once. You can kind of do it a little bit at a time.
DB: The tool sets for rest are also readily available and reasonably mature now.
JH: They are. That's the one thing that I really find organizations challenged with as they sometimes want to adopt things like GRPC or graphQL and while tools for those worlds are emerging, there’s just not as many of them. So, the default tends to be rest and HTTP. So any tool you pick up can make a get request or post request, whatever you need, but to formulate the right standard request response formats for GRPC or for graphQL, it’s a lot more difficult. And there are tools coming out to make it easier to make it better, but if you're just starting out and you want to be able to leverage some of those tools to get you going faster, to keep you from having to sit at the command line and give you kind of a graphical interface to almost like an IDE to work with an API, then the REST is a really great way to go to get started.
DB: But even as a software company, we took the same approach. REST is where we got started. We've totally nailed REST, but graphQL and GRPC are well under development. We'll come out with tool sets for those which are equally accessible to developers as our REST tool set. As I'm sure all of the other software publishers in this space are also doing so the tool sets will evolve. But REST, they are already there.
Great! I'm guessing there was also a good opportunity for you there to plug your own educational material and book for the developers to learn this stuff. So, tell us about your books. I think you also have one under development as well.
JH: I do. I mentioned earlier theAPIdesignbook.com will take you over to Leanpub and Keith Casey and I put together that book about five or six years ago, and it has a lot of this process that I mentioned. So, it's there. That was used to build our API training courses that we've delivered to thousands and thousands of people all over the world. We go in and we start from “What is an API” and “How does HTTP work” all the way up to consuming APIs and designing them as part of our training. If you get the book, then you have kind of the basics of what we deliver in training. We've done that onsite and we also do it virtually but I'm also working on a new book. Von Vernon, who's known well in the domain-driven design space has a signature series from Addison Wesley and we're working on an API design book that really takes kind of what Keith and I started.
It blends in some of the domain-driven design techniques that are out there as well as some other insights that have gained by helping teams, design APIs and train them how to scale design processes as part of digital transformation. We're taking that kind of align, define, design process and expanded a little bit and talking about how to do things like event storming and jobs stories to how to design evolvable APIs and how to go from a model to rest, RPC, graphQL documentation, and testing strategies, and how to scale a manager API program, all in one book. So, I'm currently putting that together now, and it should be out sometime spring of 2021. Really excited about that as well.
DB: That's exciting. Big project. Out of interest, how long does it take you to write a book like that?
JH: So, it took Keith and I quite a few months, but we did a lean approach to the first book we released. So, we would release a few chapters at a time. And the sooner you bought the book, the lower the price was, and then we just bumped the price up with each subsequent release. So, if you got in early and supported us, then you got a little bit of a discount. That's how we did the first one. And we kind of spread that over a couple of years because we would be out on the road training quite often. And so it takes a little while to build it. So, it took a couple of years really to build that one. This new book, I'm really hoping to have wrapped up in the next couple of months, and then of course with the editorial review and all the other steps to go to print that'll bring it sometime in probably spring of next year.
So, it takes a little while. Even with as many times as I've taught the material, it takes a while to get it into print form and really get it in a way that everybody can understand it with great numbers of examples to demonstrate different success and failure paths through the process of growing your API program and helping to design them and then scale that process out.
DB: Brilliant, looking forward to seeing it and maybe we can get you back when it's out. And we can talk about some of the concepts which you mentioned in there.
JH: That'd be great.
DB: And James, how can the listeners follow you on social media?
JH: So I'm on Twitter at LaunchAny. You can also go to launchany.com and also if you go to APIdeveloperweekly.com, you can sign up for a weekly hand curated newsletter. So I pull together a lot of really interesting articles every week that cover API details, kind of the business side, the technical side, as well as some tangentially related technology news as well.
DB: Awesome. James, thank you so much for your time. It's been a pleasure having you on the show and we look forward to getting you back when that new book is published and gaining your insights.
JH: All right. Thanks a lot. Look forward to it.
KM: All right. That's a wrap for this episode. Thank you very much, James Higginbotham for being with us. To our listeners, what did you think of this episode? Is your organization taking a bottom-up or top-down approach to digital transformation? How are you implementing your API strategy? Let us know in the comments from the podcast platform you're listening to. Also please visit our website at www.torocloud.com for our blogs and our products. You can also find us on social media, Facebook, LinkedIn, YouTube, Twitter, and Instagram. Talk to us there because we listen, just look for TORO Cloud. Again, thank you very much for listening to us today. This has been James Higginbotham, David Brown and Kevin Montalbo at your service for Coding Over Cocktails.