The Design of Web APIs with Arnaud Lauret

Kevin Montalbo

Welcome to episode 46 of the Coding over Cocktails podcast. My name is Kevin Montalbo. Joining us from Sydney, Australia is Toro Cloud CEO and founder, David Brown. What's up, David?

David Brown

Good day, Kevin!

Kevin Montalbo

Our guest for today is a Senior API architect at Nataxis based in Paris, France. His daily job involves helping people understand what APIs are, why they matter and how to do them. Known as the API Handyman, he authored the book, “The Design of Web APIs”, which teaches readers how to gather requirements, balance business and technical goals, and how to adopt a consumer-first mindset while teaching effective practices using numerous interesting examples. I'm very happy to announce as well that we're giving away a copy of the book to one lucky listener. So, thanks to our friends at Manning Publications. Stick around until the end of the show to find out more about our giveaway. Without further ado, joining us for a round of cocktails is Arnaud Lauret. Hi, Arnaud! Welcome to the show.

Arnaud Lauret

Hi! Thank you.

Kevin Montalbo

So let's dive right in. In your daily job at Nataxis, you regularly help people and consult on API design. What are the main challenges that organisations are facing when designing APIs?

Arnaud Lauret

There are actually many changes. We could talk about it for days. So I think we don't have days actually for this, but let's pick one of the topics I really like, which is actually APIs are application programming interfaces. But nowadays they are far more than that, because they're not just meant to connect pieces of software together. That's their primary intent obviously, but now as an organisation, a team or business, you need a company, a group –  people misunderstand that APIs are a new way of creating business, a new way of making your company work better. And so, actually, the main challenge regarding API design is to make people understand that they have to design their APIs, because until a few years ago, most companies let the IT people alone [to] design APIs, because they were just technical connectors.

But now when we design them, we need to gather the IT people, people with business knowledge, people who understand what developer experience means, to ensure the creation of APIs that do something that is useful for people outside your organisation. Can you team your business unit or your group, and ensure that it can be used in various use-cases? Because, as an example, if people start to project for, maybe, reworking their website or mobile application, they can be tempted to create an API that is really tight, too close to the user interface that was created.

Instead of doing that, they should rethink, “Okay, yes, we want to do a mobile application or website, but what are the functions we want to put in them from a business perspective? Do we want to create bank accounts to wire transfer?” Yes, I'm coming from the banking industry. And you have to think of that first and afterwards, you will use them in your mobile application or website, but you will also be able to provide these features to partners or to fulfill other use cases.

So that's really, for me, the main challenge it's changing mindsets for the IT people to understand that they have to do more. And sometimes you just have to explain it to them and they get the thing real quickly. But sometimes you really have to work a lot with people because, sometimes, developers tend to be too focused on really technical issues. And you have to talk to business to make them understand that you have to invest in APIs.

David Brown

It's an interesting perspective because you're right. APIs came from system to system integration primarily initially. Now we've brought people into the picture and stakeholders and business users, and end users, and departments, and all these requirements and different devices and the like. And they [still] need to liaise with those technical people who are going to be implementing the API. And there's going to be a whole bunch of communication going on. And I guess, technical people aren't known for their communication skills. So, they prefer to sit in front of a computer coding away. I can imagine, you brought up this when we asked you the question, the problems, the challenges that organisations face designing API. This is the one you said is a particular favorite of yours. I imagine introducing that “people aspect” to the API design process just makes it that much more complicated.

Now in your book, “The Design of Web APIs”, you've talked about designing a “predictable API”, and it should be “consistent, adaptable, and discoverable”. I'd like to dive more into that consistency aspect. What do you mean by making it consistent?

Arnaud Lauret

So, let's take an example. If on a washing machine you see you have a button on the control panel with an icon, which is a triangle going to the right. Do you know what it means?

David Brown

Fast-Forward. Play. (laughter)

Arnaud Lauret

With consistent design, you are ensuring that people will understand quickly how to use your API.

People are able to understand that it means, okay, it STARTS what this device should do. We know that because we have seen this button elsewhere. We have seen this button, especially on media devices or, people who are as old as me, on Walkman, cassette players, on whatever.

But this is actually a consistent design. It's a design that looks like something you already have seen. And with consistent design, you are ensuring that people will understand quickly how to use your API, what it does and so on, because you are replicating something else.

David Brown

What are the attributes of a consistent design then? What are you replicating?

Arnaud Lauret

Yeah. So actually there are four levels of consistency. At minimum, your API should be consistent with itself. It means that if you decided that a customer ID was a string, name, account number – you should ensure that everywhere in your API –  where you are –  you need to retrieve that information or you need to provide it, it is named and typed that way.

If you change names, people will be lost. People won't be able to make the connection between the various operations. At another level, your API should be consistent inside your, let's say your, organisation. It can be your team. It can be a business unit, it can be a group. All your APIs constructing your API surface must share common features like how you handle security, how you manage errors, how you represent versioning, everything you can. That way, once you have learned to use the first API in your organisation, when you shift to the next one, even if the business is completely different, at least you have a current base, you don't have to relearn everything.

Next level is trying to create APIs that are consistent with your business domain. For example, in the banking industry, there is a standard, which is called ISO 20,022. It's a terrible and ugly standard from my designer's perspective. But, it's a standard that you should care about if you want to create APIs that will be used by corporate banking software. Because this software actually understands pretty well ISO 20,022. And that can be standards in travel, in health. So, always take a look at that, to ensure that what you are building conforms to the standards, that way your API will be easier to use.

And the last level is trying to make APIs like every other people around the world are doing them.

David Brown

What does that mean? Different how?

Arnaud Lauret

There are two ways of seeing that. First, there are standards, but are not too many related. For example, if you want to represent countries or currencies, there are ISO codes for that, that everyone uses. So if you use them, your API will be interoperable. And, if you are doing RESTAPIs, you should follow the HTTP protocol, behavior, and semantics.

So if you are using the gate HTTP method, don't dare to use it to delete something, for example because it's against the protocol. And there are also common practices. For example, if you want to return data in a list, there are various ways of doing that, but there are ways that are more common than others.

For example, in my company, we decided to “return list” as an object containing a property named “items”, containing the list. Either this list contains bananas or carrots or whatever. There are also common patterns regarding the definition of your RESTful stuff. So, try to see how people actually do them and replicate that. Because if you do like some of our API's, especially famous ones, people who have used them when they come and use your API, they will feel just like home.

David Brown

I guess that principle applies regardless of the type of API, whether it be RESTful or GraphQL, you can still make them consistent with each other in terms of those principles you just outlined.

Arnaud Lauret

Yep. Totally!

David Brown

So ensuring your API is discoverable, which is one of those three meanings of a predictable API, is it as simple as ensuring you have some good document, human readable documentation, an API description format such as open API for machine-readable formats, and there may be a Postman collection for your HTTP client. Is that what you're talking about discoverability or is it more than that?

Arnaud Lauret

Yeah there's actually far more than that. Because, first, very simply, okay, you made some documentation, you created a Postman collection – but if nobody knows how to find it... if nobody knows your API actually exists, that document is worth nothing.

You need to have a place – some people put into portal developer, or an API registry or whatever. You need to have a place where people can find all of your APIs. And so you will put in there your documentation, your OpenAPI files, and your Postman collection regarding documentation, reference documentation, which is usually based on OpenAPI files. Depending on your context, they may not be enough. For internal API, there may be enough. But when you provide APIs to the outside world, you need use-case documentation that actually explains what you can do with the API, because the reference documentation is a kind of list of ingredients. But you need the recipe to actually use those ingredients.

And on top of that, you need a very short and business focused description of your API that will help people to understand how this API can solve that problem. So, that's the usual way of how people think about discoverability. But there is something else that you need to think about It's when you are actually using the API, you can make it discoverable, in a sense of, “Okay, I have seen the documentation. I know that my first request could be to read a list of customers and I do a “get-slash-customers”.

But if inside the response, you provide information to explain, “Okay, now you have a list of customers. You can read a customer, delete a customer or whatever. Maybe you create them [inaudible] if you provide inside the response, the links to other operations, like you would do in a website that make your API accurate is visible. Because some people don't like to read documentation, but just want to start to use the API. And so if you give them directly, what are the next possible operations, it could be interesting.

David Brown

Even the response itself, which was actually the original definition of a RESTful API. Right?

Arnaud Lauret

Exactly. And even if you don't want to do RESTful, making APIs consistent makes them discoverable because, okay. If I see a get-slash-users, I can guess, but I can read a user by sending a get-slash-user-slash-user-ID. I can guess, but I can probably delete the user by trying to send a delete-slash-user-slash-user-ID. Because this [API design] is consistent with common practices and this design also actually respects the HTTP protocol. So, making API discoverable means making them easy to find, but also when you use them, you need to provide, by any means, information of how people can use it, how people may find over operation and over features. 

David Brown

Hmm. Interesting. There's some confusion, I think sometimes, in terms of the role of an API gateway in the whole serving of an API. We're going to dive into the API gateway and the implementation of it. So, first of all, can you just define what is the role of the API gateway?

Arnaud Lauret

In my opinion, an API gateway is only supposed to expose API in a secure way. That means it's a kind of proxy, but it's a little bit more [of a] rounded proxy. It may handle high security levels such as, okay... “As an API gateway, I will ensure that only this application will be able to consume this API”. I may also say, “Okay, this application can consume this API, but it can only use a subset of all the other operations”.

An API gateway, middle supervisor features such as throttling. We say, “Okay, you can use the API, but you can only do, let's say, 10 calls per minute”. That is for me, the role of the API gateway.

If you put something else inside it, sometimes it can be interesting to do that, but most of the time you are, for me, working on the domain of what should be inside the implementation.

David Brown

And this is what I wanted to get to. So now, I know you have some thoughts on this matter in terms of where the boundaries should lie between the API gateway and the implementation. So, cause I know a lot of API gateways allow you to do transformations and caching and perhaps throughout the lookups and all those sorts of various things. And the lines are starting to blur between the gateway and the implementation of the service being proxied. So, can we dive into that? Where should those boundaries lie? And what are the reasons for that?

Arnaud Lauret

In my opinion, what is tied to business rules must not be in the gateway.

Yeah. In my opinion, what is tied to business rules must not be in the gateway. So, if you need, for a practical purpose say, “Okay, we are returning that JSON, but it could be interesting to return them as CSV.” Okay? It's just the technical transformation without business involved. Maybe it could be put in a gateway. 

But if you are starting to say, “Okay, now if the user is of type-admin, we will do this or do that and call this service and call two operations instead of one, take some data, halve them, [inaudible],” that's business rules. And that must not be in the API gateway. It must be inside the implementation. So for me, the separation is really… “Okay. If it's a high level of security, if it's a general concern, just technical concerns and so on, it can work”. But the implementation must be the owner of all [of] what is tied to the business domain, the business rule, the business intelligence.

David Brown

What is the reason for that? And what is the danger if you have logic in both places?

Arnaud Lauret

From what I've seen, remember what happened with ESBs? I've worked in companies having ESBs. We have built a lot of business logic in there and it was really a total mess because it was a proprietary technology needing really specific skills to develop in that. And when we wanted to get rid of that, it was really, really, really hard. But if we had stuck to, say, “Okay, we are creating our application.” They are there, the whole business logic and so on. And on top of that, we have maybe a security layer doing all that stuff. If you want to change it, we can.

So, based on my experience, developing business logic inside gateways or ESBs for that matter, requests are actually usually specific skills. You need to understand the product, how it works and how you can develop on it. Sometimes, you don't have all the features you would have using a regular programming language such as Java, .NET or whatever. You don't have unit tests. You can’t put in place CICD easily and so on. You can’t test in isolation. And so that's why I prefer to keep the business logic outside of the API gateway.

David Brown

Yeah, because like you said, it is like an ESB. The API gateway is almost mimicking that ESB-type environment where you have smart pipes as opposed to the smart endpoints and dumb pipes, right? So, I totally get it as well as you've mentioned the development process you have around the deployment process to the application logic as well. It makes a lot of sense. You mentioned the security… obviously that proxying concept and the authentication and security is lying in the API gateway. But will you have circumstances where there is some level of security in the service itself, the business logic as well? So, where should the boundaries lie between the two?

Arnaud Lauret

Yeah, so definitely putting an API gateway in front of your implementations, whatever it is; a good old monolith or microservices or whatever, [for] the API gateway, we only provide a high level of security. As I said, it will ensure that only registered consumers can use the API in some sort of way. So let's say I grant access to my “users” API to some consumer. This API is allowed to use my “users” API. And inside this API, the consumer can use all of the functions. Let's say, “create users”, “delete users”, and so. If there is an end user using this application, let's say, Arnaud, me. And I trigger an API call, but it's supposed to delete a user.

We get where I will say, “Okay, who is making this call?” It's the application that is allowed to use this API in this specific operation. So, the call is sent then to the implementation. The implementation needs to ensure that the call comes from a known gateway, obviously. And it needs to ensure that, actually, the person - me - who was requesting to delete user 1234 is actually allowed to do that. It means that user 1234 is someone within my team. So, I can delete this user because this person is no longer in my team. But if it's not the case, the implementation will say, “No, Arnaud cannot delete this specific user.” You can delete other users, but not this one because it needs a data parameter. That's what I call application level security.

And most of the time, people actually don't really get it at the first time because I think, okay, we have an API gateway, it will handle all of the security, but no. Because the API gateway doesn't know the business logic. The API gateway does not know who is allowed to delete a specific user or who is allowed to read a specific user or not. This is the job of the implementation. And that's really important because being able to access data that you are not allowed to is actually the API Security 101 failure we see every two or three weeks. 

For instance, I don't remember the name of the application, but there was an application on iOS, I think, that allowed users to record phone calls. And so the phone calls were stored somewhere. To get the phone calls all you had to do was send in a request basically telling, “Okay, give me the phone calls for this phone number,” Whatever, okay? And if hackers look at what it's doing in the application, when they see that, what is their first move? They say, “Okay, what happens if I put the phone number of someone else who may be using the service?” And obviously in that case, they were able to retrieve a conversation because the implementation was not checking that, “Okay, give me the conversation for this phone number.” But this phone number does not match with the user we are going to actually connect it to.

David Brown

That example you just got there, I mean, it's a record-level permission, but it's like an account-level ID. So, I can imagine there are some sort of database, with some sort of account ID and they're associated with some sort of token being passed on by the API gateway [that] are allowed to read and write to these records in our database. But I'm imagining, particularly in the finance industry, in which you work in a lot, it gets a lot more finely grained than that. And it would get quite complex when you're down to the record-level of who's allowed to read and write to records. Are there other frameworks around like, security, top frameworks around to do this sort of stuff? Or do you really have to spin this stuff yourself?

Arnaud Lauret

Nope. As far as I know, no. For example, in my company, when a consumer is actually making an API call on the behalf of someone, first the API gateway [will] delegate the authentication of the end user to an identity provider using SAML. And then we get to know who is the user, what are his roles regarding this application. But it's only a fraction of what is actually needed to handle application level security, but at least the gateway transmits all of this information to the implementation inside the token, the JWT token. And that gives some hints for the implementation to do its job. But most of the time, the implementation needs to know the users are stored in this table and we know how to make the joins we built over the table to ensure that this user is actually allowed to do that.

And that really depends on the business you are doing - what your implementation is, how it works and so-and-so. Actually, there is no magic framework that will handle all the possibilities. Well, there are frameworks that can handle role-based security and so on. But sometimes you have to go deeper than that. For example, let's say that a bank counsellor tries to make a wire transfer on the behalf of the owner of the bank account. And this person can be allowed to make wire transfers from a hundred euros or dollars to a thousand dollars, but not beyond. And that is actually application-level security. And that cannot be handled by a kind of generic role-based access mechanism. You have to actually write your own custom one.

David Brown

I was hoping you'd say, “Yes! Go here and you can buy one of those from the shelf,” but sadly not. But it's interesting, like a lot of the stuff we're talking about in terms of design and stuff, which is still very topical today. I mean, we've been working with APIs for several years now and people are still struggling with a lot of these concepts. I'm guessing at some point in time, APIs themselves will just become hardware, their endpoints of mechanisms and all this stuff, which in these foundation days will seem old hat. What are you seeing in the space? Like, we have REST and GraphQL and AsyncAPI, GRPC, some architectures trending more than others. What do you see happening in the API space?

Arnaud Lauret

If the adapted solution is doing a good old REST API, we will do a good old REST API.

Actually, yes, there are new or not so new technologies, like GRPC has been there for a long time. GraphQL and stuff have been there also for a few years. And I think there will always be new ways of making software, talking to each other. The important [thing] for me, and especially when people say continuously, “Okay we don't want to do a REST API. We want to do fancy new ways of doing API because it's brand new and we want to do brand new things,” I say, “Okay but why are you picking this particular technology?” What problem are you trying to solve that can be solved in the end? Can it be actually solved with this particular technology? And as an architect, I try to moderate people saying, “Okay, it's not because it's brand new, but it is actually the solution. It's not because you like it, but you need to use it always.”

So regarding all these different ways of doing APIs, “What is the problem you are trying to solve”? And we must choose the adapted solution. If the adapted solution is doing a good old REST API, we will do a good old REST API. And that has some advantages because people are used to them. People can use them quickly and they are easier to implement. But if we need to do AsyncAPI because you are... inside our company, we want to use Kafka for actual really good reasons. We do that. And if we need to, let's say, create a backend for a frontend, for a mobile application, and we need to optimise network calls and so on, maybe you can do GraphQL. 

But the problem with GraphQL is it can be really tricky. You can do crazy things with it, and especially, you can just destroy your system because a consumer can send really complex requests. And if you don't have mechanisms that prevent them from doing that, they will send us those requests and your system will just go down, or you will have to pay 300% more than what you expected on your AWS or a Google cloud provider. 

So yeah, there are new ways of doing things and there always will be, but in my opinion - and actually it's not really an opinion for me, it's a fact - choose the adapted tool to your context. And on top of that, all the design principles, if you have learned actual design principles and not just the technology, you will be able to apply those principles on REST, on GraphQL, on GRPC or AsyncAPI. So it makes your API - whatever they are - consistent, usable and so on. And everything would be okay, but just be sure to choose the right technology for what you wanted, not just because it's fun, it's new and, “Oh, Netflix is doing that.” Yeah, but you are not Netflix.

David Brown

Sensible advice, Arnaud, thank you very much for that. You make it all sound very accessible and easy to understand. How can our listeners learn more about what you're writing about and discover things like “The Design of Web APIs” which is a book you've published with Manning?

Arnaud Lauret

They can come to my blog. Since March, I started to write more frequently, a post a week. So I learned more from experience, my failures, my thoughts and so people can learn from that. They can follow me on Twitter, my handle is “@apihandyman.” And obviously on Twitter and on my blog, people can find the link to buy my book. But actually they read it online on the Manning website. You can read it freely. I think it's five minutes a day. You can actually read it so you can see what I’m actually speaking of. So you can test it before buying it. And actually if you like the book or if you don't like it, it lacks that feedback from readers because I may work on a new version of the book if time permits. So, having feedback will be interesting.

David Brown

I did that myself. Actually, I went to Manning’s website and I was reading sections of the book and it starts to obfuscate the text after you've been reading it for a while. It's of course, quite cool. 

Arnaud Lauret

But the figures, you can still read the figures afterwards. So sometimes they are interesting just by themselves.

David Brown

And of course your blog is at APIhandyman.io as well. Thank you for joining us today.