Designing APIs with Swagger and OpenAPI with Joshua S. Ponelat and Lukas L. Rosenstock

Kevin Montalbo

Welcome to episode 63 of the Coding Over Cocktails podcast. My name is Kevin Montalbo, and joining us from Sydney, Australia is Toro Cloud CEO and founder, David Brown. Hey, David! 

David Brown

Good day, Kevin! How are you?

Kevin Montalbo

I'm great! All right, I'm very excited to say that we have two guests today joining us for a round of cocktails. Our first guest is a senior architect working with the Swagger/OpenAPI team at SmartBear, who aims to reduce API friction and help teams build better tools. He is joined by our second guest, who is a consultant providing services around APIs, such as backend development, technical writing, content production, and API design consulting. 

Together, they are the co-authors of the upcoming book, Designing APIs with Swagger and OpenAPI, courtesy of Manning Publications. Stick around until the end of the video to learn more about the book and how you can get your hands on a free copy. Ladies and gentlemen, I give you Josh Ponelat and Lukas Rosenstock. Hi, guys! Welcome to the show!

Joshua Ponelat 

Hey, Kevin!

Lukas Rosenstack

Hello! Good to be here.

David Brown

All right, thanks for joining us!

Kevin Montalbo

Yeah, so let's first start with Josh. Josh, the title of the book is Designing APIs with Swagger and OpenAPI. So, what is the relationship between Swagger and OpenAPI?

Joshua Ponelat

Swagger was the beginning. Yeah, that was everything. It was the tooling. It was the idea of describing HTTP APIs. And then at some point later on in its lifecycle, in order to keep it open, the heart of Swagger, the specification, the standard got donated to the Linux foundation and as part of that transition, that got renamed OpenAPI. So, this is why we have these two buzzwords today. They are still separate entities, but they started out as one. And so, we often see them used synonymously with each other.

David Brown

And so when you refer to Swagger in the title, are you talking about the specification or are you talking about tooling? Or is it just because some people get confused between Swagger and OpenAPI? Because I often hear a lot of people are referring to OpenAPI as Swagger, and they're referring to the specification.

Joshua Ponelat

Exactly. So, I mean, that's kind of why we went with the title. We wanted to catch-all. We wanted to show that both were here. But to be more technical, we're referring to a certain set of tools that fall under the Swagger umbrella. So, when we're designing APIs, we use those tools within the book to design OpenAPI or to describe them.

David Brown

Yeah, right. Lukas, I understand it you joined Josh a little bit later as a co-author on this book. Tell us a bit about your background and what led you to co-authoring the book with Josh?

Lukas Rosenstock 

So, as Kevin said in the introduction, I was working as a freelancer doing work around APIs and it was kind of funny because I had just started a new website for myself. And it was one of the first messages that came in through the contact form. It was a message from Manning the publisher. And they said, “Oh, they, they are currently on the search for co-authors for a book.” And yeah, so I was asked to join it and I thought about it a bit because obviously, a book is a big project because I was already doing writing work before, but I had never written a book. So I thought, “Yeah, this is a great experience. So, I would like to join the book” and yeah. It was a good experience writing this book together with Josh. 

Joshua Ponelat 

I'm really glad Lukas joined in there.

David Brown

Yeah, because it's a huge undertaking. How did you guys go about breaking up the content and chapters between you?

Joshua Ponelat 

So, at this point I'd written about a third of the book and I was burning out. This book writing is — my goodness! Never write a book. Lessons learned. And then so thereafter, the structure of the book, we sort of sat down. We looked at, you know, how can we break this down to something that we use in software development? How can we break this down to sprints or certain milestones? 

And so, we divided the last two sections of the books into three parts based on that. And we had this cool little rhythm of, like each section would be a sprint. And so based on that, we mixed and matched a few chapters and some of the brainstorming sessions around, like the dummy projects and all that sort of stuff.

Lukas Rosenstock

I found it really interesting to work on this, the structure of the book. And to be honest, we kind of restructured it even a few times in the process because the biggest challenge at the beginning was to find a coherent narrative and to decide together, “Okay, what do we want to tell in the book? And how do we create a structure?” We both know what we want to say and that we both can write parts of the book that we can work on individually, but the parts still fit together at the end and you could tell it like a story which we want to tell.

David Brown

The story about APIs. Well, let's get started on that story. So, it's obviously about OpenAPI, the specification and the tooling associated with that. Many of our listeners will have actually heard about OpenAPI and spoken many several times before, but let's start at the beginning. What is OpenAPI? What are the advantages of using an API description format such as OpenAPI? I'll open the floor to either of you. So, either of you jump in and answer the question as you wish.

Joshua Ponelat

So, I have this recipe analogy when it comes to OpenAPI. So, if you have a restaurant, we can view that as the service, right? You have this food you want to get out the door and then in order to get that food out the door, if you're the customer, you look at a menu and say, “Ooh, I'd like the burger with some fries.” So, what OpenAPI does is it makes those menus something you can hold in your hand and something that you can focus on without actually having to think about the restaurant or some bespoke format. It's a standard way of writing menus for services i.e. APIs.

David Brown

Now this is a difficult question to answer objectively, given that you work for Swagger, but OpenAPI or Swagger, the original version of OpenAPI wasn't the only game in town when it came to the API description formats, right? There were a bunch of others. So, what makes OpenAPI the specification of choice? Why has it evolved to be the standard for describing restful APIs?

Joshua Ponelat

So, OpenAPI. Today, it's simply because it's the agreed upon standard, right? For better or worse. And that's what makes it the choice today. What led to it becoming the standard for description formats? I would think it was its focus on tooling as opposed to completeness and power. A lot of the other HTTP API definition formats are actually easier to use. They're more powerful when it comes to composition of your documents. But because of that power, tooling wasn't able to keep up, and OpenAPI managed to restrict the space heavily. And in some cases, controversially, like with JSON schema being a flavour or a subset of it, but it allowed tooling to be able to do useful things with it. 

I believe that was one of its features leading up to it. But as if we were to compare it today, like if RAML or API Blueprint or any of these other formats were the standard, that would be it. No question. It's really the one that everyone's using. That's the one I want to use. Whether it's good or bad is a little more subjective.

David Brown

It's interesting because some of those other formats were also driven by commercial enterprises, which were pushing the format for their own platforms as well. So, I wonder if that also played a part, in that Swagger was being a little bit more “the Switzerland” whilst providing good tooling to facilitate, whether it's deployed on this iPaaS or that iPaaS, whatever the platform of choice was. 

Early on in the book, there's a section that goes on to discuss OpenAPI and when not to use it. So it's interesting that you described that it's not necessarily a description format which should be used in all use cases. Can you tell us a little bit about that, when and when not to use OpenAPI?

Lukas Rosenstock

So, I think there are different kinds of APIs that we need to look at. So, there's what we usually call REST APIs, even though that term could also be controversial because not everything that we call REST is actually REST by the definition.

David Brown 

That's another program. What we call REST is not really restful.

Lukas Rosenstock

Yeah, so we have these kinds of APIs that have different endpoints with different requests and responses. And if you have this kind of API, then OpenAPI is perfect, but then there are other types of APIs. There are APIs which are based on query languages, like GraphQL and there are APIs based on RPC or Remote Procedure Card, so there are different ways to design APIs. And really, if you have any of these other formats, then OpenAPI is probably not the best format to describe that API.

Joshua Ponelat

Yeah. I mean, that's it right? It's when HTTP is used as a transport rather than the API itself. It backs a lot of these other technologies and sometimes when, to expand on even further, things like hypermedia, when the semantics are richer than what HTTP may offer, there's better ways of describing that. Hopefully I'm not too versed in that, but OpenAPI isn't too great when it comes to hypermedium.

David Brown

Interesting. We've covered a lot of the benefits here of taking an API-first design approach. We've talked about it extensively on this podcast, but you mentioned in the book that whilst taking an API-first approach for API design is a good approach and is valuable, it does come with trade-offs. Can you run us through the trade-offs you're referring to?

Joshua Ponelat

Okay, so the trade offs are… it's a process. And unless you get a lot of buying into that process, you risk having your team members get frustrated or aggravated with this idea that we need to think about the API ahead of time. And I think that a half-baked process might be worse than a no-process-at-all, thinking pragmatically here. 

And then in those cases, where either your scope is your teams are too large to adopt this full scale, you want to think in small steps to get to an API design first. And that may start with simply documenting the world around you, simply taking your services and doing a code-first approach, which is very incremental, right? You can add a little sprinkle, little annotations along your Java code, and you'll be able to generate these OpenAPI definitions, which could lead to being a gateway drug into, “Ooh, I kind of like the benefits I'm getting out of it!” 

But slamming on design-first without thinking about these process changes and getting the buy and showing the problems it's solving, I think that could lead to more hurt than benefit. And it's something I think we should go in with open eyes.

David Brown

Lukas, you're nodding your head there. So, you're well in agreement here. 

Lukas Rosenstock:

Yeah.

Joshua Ponelat

Yeah. I mean, with all this said, API design-first, that's where we want to be, right? That's the destination, that's definitely a good thing, but depending on the maturity and where you are with your teams, that might not be the first thing you tackle.

Lukas Rosenstock

You could think of API design-first as the North Star, as your goal. And even if you can't fully implement that process in your organisation, it still helps to set that as the goal and then move incrementally.

You could think of API design-first as the North Star, as your goal. And even if you can't fully implement that process in your organisation, it still helps to set that as the goal and then move incrementally as Josh said.

David Brown 

That makes a lot of sense. Yeah, we often talk about these things like you should go head long into it because it's best practice, but maybe it should be, like you say, that North Star instead, in a lot of use cases and get proof of concept first.  Okay. So, you mentioned, Josh, JSON Schema briefly before, and I think there's some confusion between the role of JSON Schema and OpenAPI. In fact, we had Postman on the program not that long ago and in their survey, they asked their participants to choose the way they describe APIs and JSON Schema was up against OpenAPI as a way of describing APIs. But some people may not realise JSON Schema is still used within OpenAPI. And I think that distinction is pretty unclear. Can you run us through how that works?

Joshua Ponelat

So, JSON Schema, it describes data for those who, I'm assuming a lot of folks on this channel may have heard of it in some form or fashion, and it predates OpenAPI. Where I like to find the distinction is that we are mostly interested in data. And when it comes to describing the services, all our focus tends to be, “I just want to know the shape of this data coming in and going out, being returned to me, what I need to push up.” And so, we can think of it that way. In fact, we could get far just by describing those things as JSON Schema. 

However, where OpenAPI comes in, it's connecting these JSON schemas together for HTTP requests and responses. So, we're saying, “Well, okay, this message, which I've called FUBAR, I need to send it against this endpoint with this method, and maybe these query parameters.” Now, yes, you could just have JSON schema and you have to somehow, out of context, describe which API endpoints it comes to based on whatever you got going. Or you can combine the two together, and again, OpenAPI encapsulates to wrap around JSON schema. JSON schema is really the tasty part, right? And OpenAPI connects these dots together to give us a cohesive API.

David Brown

OpenAPI supports a subset of JSON schema, right? So, what does that mean? And what is the subset?

Joshua Ponelat

So it did support a subset. So we've got three major versions of OpenAPI today. One which we might call Swagger 2.0 and get slapped around a bit, which we can rename to OpenAPI 2.0. Then we got 3 and we're currently on 3.1. So, 2.0 and 3.0, they're the ones that have this subset of JSON schema. So they've limited certain features of JSON schema. Again, as previously mentioned, I think it was to make tooling better and easier to maintain and build. But that was controversial because we'd want to reuse those schemas in different tooling independently. Those JSON schemas? There's a whole tooling ecosystem around that. So, why can't we just shuffle them in and out? So, when it came to 3.1, the specification team decided, “Okay, we're gonna add full on support for JSON schema.”

So there's no subset of it. Now they have added a few bits on top to make it easier to describe XML shapes, the little discriminator flag and one or two bits and bobs on top of JSON schema, but that's on top, right? So they support everything in JSON Schema 2020-12, which is JSON Schema's weird draft versioning system that's always confusing the hell out of me. 

So yeah, 3.0, there's a subset. And you have to maybe get a little converter just to tweak it a bit. It's not major, it's things like type, might be boolean. And in JSON Schema you can have multiple types, so you can have boolean or null, which was the big one there. But 3.1 does have full JSON Schema support. But because of that, tooling is really struggling to catch up and we're not seeing as much adoption of it as we'd like to see, but that'll definitely change. 

David Brown

The more technical question now — I was reading some of the excerpts from your book and I found the section on inheritance and polymorphism in domain models. Quite interesting. Can you run us through how they apply to OpenAPI?

Lukas Rosenstock

Yeah, sure. So, I would like to introduce the example that we used in the book, because I think that makes it easier to understand. So, in the second or third part of our book, we present an application called “Pet Sitter” and the idea is it's a marketplace, it's basically like an Airbnb or Uber for pet sitting. And in their first version, they just make it an application to take dogs for a walk, but then later they want to expand it to more pets. 

And that's where these inheritance and polymorphism come in because in the domain model, they have jobs and they have users and they have dogs and later, the question is, “Okay, should we add a different model to our domain model? Just add a new concept?” and obviously, like any concept in a domain model, it turns into an adjacent schema later. So, will we have a schema for a dog, will we have a schema for a cat, will we have a schema for whatever else you have there? 

And then if you do that, you kind of lose that information. Like, a dog is a pet, a cat is a pet and we may want to express a job. A job has a pet and that pet can either be a dog or cat. So, we have these kinds of advanced scenarios in a domain model where we want to show, “Okay, this concept is a subtype of another.” And whoever has done object oriented programming knows about class inheritance. We have similar concepts in JSON Schema. And so we have composition keywords where we can create a schema and say, “Okay, this schema actually inherits fields from another schema,” or is a composition of a different schema. And yeah, that's the gist of it.

David Brown

And the polymorphism?

Lukas Rosenstock

The idea of polymorphism is that you can always replace a subtype with its supertypes. So that means, in the example that I mentioned, whenever we have a dog and cat, we can have all these other pets, we have the supertype “pet.” So, whenever we use a dog or a cat, we know that this is a pattern. We can use the “pet” supertype for it.

Joshua Ponelat

I think when it comes down to it, at least for me, there's a lot of these modelling processes that are fantastic, but I just wanna write a model once somewhere and be able to reuse it as much as possible with tweaks in and out. JSON Schema still struggles a lot with that. We're not near what RAML and tools like that can do. But it does have them, right? And learning how to use them keeps our things more consistent.

David Brown

How should developers manage breaking change in an API? I know you cover this in your book. So, clearly we have versioning and that sort of stuff, but in practice, I see a lot of bad practises emerging. We're consuming a lot of APIs as an integration-platform-as-a-service ourselves. So, really what practises should developers follow to manage that stuff?

Joshua Ponelat

Never break your API. So that’s it summed up in one phrase. I mean, we can ask the question, “How do we never break our API?” but before we do, we can weigh the costs of what it's going to cost to break your consumers. What's the cost involved with that? Which is different, right? So public APIs are open-ended. We have no idea mission critical systems could be built on this. Internally, we may have a better idea. So we could think, Bob over there, he's the only one. And he uses this for like a little reporting thing, minimal cost, versus the cost of working around it because we're actually not breaking it. These aren't breaking changes, we're mitigating so that we don't break them.

So when we add versioning or anything like this, we're not breaking our API, we're adding something. And then it becomes a question of how do we reduce the cost of working around this potentially breaking change. I like to think of it more as mutable changes. We're changing something that you're using to make it a different way of using that same thing, which is from, say, adding a completely independent feature. But yeah, they're not breaking changes in my mind. Right? We're mitigating them.

And then maybe to answer your question, which is like, “Okay, now that Josh has gone through all this hoopla of he doesn't like breaking changes, what do we do?” Well, we keep things open ended. That's that's most helpful, right? A little trick is never have arrays as your top level item, have an object. This is predominantly for JSON, obviously, as your messaging format so that you can keep adding fields to it. That's one way of keeping something open ended where if you had a field called “items” and it's a bad list of things, and you don't want that to be the list anymore, you can change that to maybe another field with “items 2.” 

Yeah. Not the greatest example, but at least it's additive and you can then supersede one with the other. So, you can say, “Well, if ‘items 2’ exists, ignore ‘items.’” That means my consumers are targeting the right source and we can override it, but while still keeping those other consumers happy and ticking over.

Lukas Rosenstock

With API design-first, because you have an actual design phase where you think about designing your API, you're trying to make it consistent.

I would like to add that going through an API design-first approach really, really helps prevent breaking to changes because when you go code-first and you just start writing something, it's very easy to write something that's not backwards compatible if you need to change things later. But with API design-first, because you have an actual design phase where you think about designing your API, you're trying to make it consistent. And that's like the chance you have to really think forward. Okay, how can we design our schemas in a way that we can easily extend them without breaking stuff? So we should really use that design phase to think ahead and see how we can build an API that we can use for a long time and change without breaking it.

David Brown

And in terms of versioning your API going from major version one to version two, it's okay to introduce breaking changes?

Joshua Ponelat

So, versioning is the last resort. That's like, you've tried everything under the sun and you've just given up, this API is awful, it was built in the stone-age. Rich Hickey, idol of mine, right? Clojure, in his “Simple Made Easy,” one of his great talks, he talks about this idea of versioning, and it's really just a new thing. So, version one is “foo” and version two is “bar.” Now we may still want to hold on to the assets, like the name of the domain and other various things to call it version two, but effectively it's a new API. So anything goes right? You're designing an API from scratch. You're not trying to worry about previous consumers with it. So yeah, you can break everything, do what you need to, but it is a last resort, that's a bomb of like, “Okay, no, we're done now, new API entirely.”

David Brown

If that is the bomb approach that someone's taking, what should be the transition strategy? I heard some vendors, Salesforce is one that came to mind, still supporting some 30 versions of their API. So what is the migration strategy? When do you turn off the old API? How do you do that process?

Joshua Ponelat 

That's true. I mean, poor Salesforce, 30. Wow. That's rough. So, it comes back down to that equation at the beginning, what is the cost to your consumers? And if you can figure out, if you can start to quantify that cost of removing an API, which is effectively breaking them, or, you know, breaking parts of it in order to merge it with another one, whatever that may be. And that comes from metrics that comes from having a dialogue with your consumers, that's where the energy should be, right? Figuring out that cost. Because once you've got that quantified, now it becomes a decision you can make and transitioning is blind until you have that to say, well, “Yeah, we'll shut it off,” or, “We will shut off parts of it.” It ends up being the same thing. You're blindly taking action whereas you should be focusing in on how can we make this a quantifiable decision to move over these consumers or to completely remove the API, whether it's a version or an entire API.

Lukas Rosenstock

And you need to be really transparent with your API consumers, you need to clearly communicate what your roadmap is. And you need to plan ahead, like how many versions you can support? Because the worst thing they could probably do as an API provider is give users the impression that you will support this thing forever and then give them a shutdown notice of 30 days or something. So, it's really about communicating with the customer and setting clear expectations on how and when you will deprecate the API and then give them some time to move off to a new version. 

David Brown

The final chapter of your book is titled “ The API Pre-release Checklist.” So, can you share with us some of the essential characteristics that an API should have before it is released? 

Joshua Ponelat 

If this is the first time your API is getting public, you're in a position to release as small as necessary.

I think this comes as a great segue because it's monitoring, it's metrics, it's security. I think a mistake people may make is releasing too much of their API and getting stuck into that. If this is the first time your API is getting public, you're in a position to release as small as necessary, so that you can focus on making that as robust as possible with your metrics, figuring out how much stuff is going in and out there, adding your identifiers if you can, try to figure out where your traffic's coming from. 

Security's becoming more and more important today. And after-the-fact security is more expensive than before-the-fact. So if you can double down on that, if you have the resources, but yeah, release small amounts of your API. I think that's the key here. And then you can focus on how you're going to grow that without having major impacts to the rest of your API. Of course testing and being robust all come from this part. So the checklist is there to think about these things and decide how much to invest in, making them more robust. But yeah, if your API's small, then you have less to worry about across the board.

Lukas Rosenstock 

Yeah, Josh previously talked about the cost of changes and the cost of breaking changes and why we want to avoid them. And, I mean, the math is obvious that the more consumers your API has, the more difficult it is to change it because the cost is higher because there are more consumers relying on a certain behaviour. You can change things more easily if you just use it internally. And if it's just like two people, and then if you expand to, let's say a limited group of like a few external consumers, it's still pretty reasonable to make changes. But once you've opened up the floodgates and people use that, it's really hard. And then you need to think about all these maintenance questions, and you need to get your roadmap. You need to get your developer experience right.

And it's also about things like documentation and providing support because yeah, if it's just one API consumer, one API provider, they can talk to each other, but that obviously doesn't scale. So at some point, you need to make sure that the documentation you have for your API is really good so that people can figure out their problems and solve them. We have one chapter where we talk about error handling and how important it is that APIs return error messages that API consumers can actually make sense of and debug their integrations without having to ask a support person, which again, doesn't scale. 

So, it's really about getting all these small things right that help people work with the API. And again, the more people use it, the more expensive it becomes to change, the more investing in developer experience pays off in the end. So, it's really important to get to a certain level of these things before you can release an API to the public.

Joshua Ponelat 

People using your API, whether it's internally or small beta testing, that gets even more feedback than your design will cause.

And we want those problems, right? So we want tons and tons and tons of people using the API. What we're also saying is get those feedbacks looped in. And API design-first helps with that, too. But of course, people using your API, whether it's internally or small beta testing, that gets even more feedback than your design will cause. Once the tires hit the road, things always change.

David Brown 

So much good advice, not only on OpenAPI, but API design as well, best practises. When can we expect the final published version of the book?

Joshua Ponelat

So, the book itself is written and we've already done the proofing stage, it's going through the indexing and then the layout stages. Yeah. So, I think we're on the order of a month or two, but I don't actually know. And we can get back to you on that for sure.

David Brown 

Yeah. Okay. I did read it was targeting around April, that sounds what your publisher is sort of promoting at least anyway,

Joshua Ponelat 

I need to check. Yeah, you've got better info than we do.

David Brown 

Where can our listeners follow you and keep up to date with the publication of this book?

Joshua Ponelat

So I think Manning is probably the best port of call for the book. With the title, obviously, Designing APIs with Swagger and OpenAPI. We are building the website, DesigningAPIs.com as a sort of housing platform for the book. It's still in its early stages, but we'll be growing that out as well. 

Lukas Rosenstock 

And of course, you can follow us. For example, I'm pretty active on Twitter and I post updates regularly both about the book, but also I generally share a few stories about APIs, interesting stuff federated. So yeah, I would like to keep in touch over Twitter.

David Brown 

And your handle, Lukas?

Lukas Rosenstock

Yeah, the handle is just my name “Lukas Rosenstock.”

Joshua Ponelat

I mean, so if you like maps and weird stuff, “@jponelat,” otherwise just follow the Swagger API Twitter handle, where it's more API-focused.

David Brown

Josh and Lukas, thank you so much for your time today!

Joshua Ponelat

Thank you, David! Thanks, Kevin!

Lukas Rosenstock

Thanks for having us!