How Toro Cloud killed the API connector

Kevin Montalbo

Hello listeners, welcome to 2021 and our very first interview episode of the year. You’re listening to Coding Over Cocktails by Toro Cloud. My name is Kevin Montalbo.

After listening to this podcast, make sure to visit our website at www.torocloud.com for after discussion notes, and blog posts related to this topic. Joining us today for Cocktails is Toro Cloud’s CEO and Founder David Brown.

Hey David! Good to have you on.

David Brown

Hi, Kevin!

Kevin Montalbo

All right. And joining us as well is Toro Cloud's Product Architect, Vivien Jovet, all the way from France. Hi Vivien.

Vivien Jovet

Hi, Kevin.

Kevin Montalbo

All right, let's begin this podcast. Vivien, can you tell us how connectors work in the context of Enterprise Application Integration platforms? What are these so-called connectors?

Vivien Jovet

Yeah, so a connector is a piece of code, a piece of software that wraps the necessary logic to communicate, to make request to an external API. So, we often call them a SDK or client repository, and basically they contain methods that allow you to make request to the operations of an API.

Kevin Montalbo

All right. What can you say are the disadvantages of connectors when we're trying to integrate these with APIs?

Vivien Jovet

[Connectors take] a time-consuming process to create, update. And of course it's prone to error.

Yeah. So, one of the main disadvantages of connectors is that they get out of date really often. The reason why is that APIs publishers make changes to their APIs and they don't necessarily warn you about the breaking changes or what exactly are the changes they are making. And you will only discover after you connect, those don't work when you get the errors. So, when you rely on an integration platform for your connectors, you actually realize first that they would have the connector for your API that they are monitoring, the APIs for changes and things like that, and that they will actually update the connector quickly so that it works with the latest version of the API. And the reason why it's not good is that, often the connectors are maintained by developers by hand. So, it's a time-consuming process to create, update. And of course it's prone to error. So, you can imagine that for an integration platform to maintain hundreds of API, it's really not scalable that way.

Kevin Montalbo

Do you have that experience, David? Are you able to observe that kind of experience from the industry?

David Brown

A connectorless approach means that you bypass this software, which sits in between the integration platform and the API.

Yeah, look, I mean, that's why we developed Martini as a connectorless integration platform, because to be honest, we started out to the same as pretty much every other integration platform out there and that we built connectors and you'd largely build these connectors by hand like Vivien was saying. So, pretty much every integration platform in the market today still uses connectors. The problem with them as Vivien explained is it's a very laborious process and we found that ourselves. So, you know, we got up to a couple of hundred connectors and just keeping pace with the number of APIs available and all the changes to those APIs just meant that it wasn't scalable for us. We had to find a better way of doing it. So, a connectorless approach means that you bypass this software, which sits in between the integration platform and the API and the integration platform connects directly to the API instead.

Kevin Montalbo

So, now that we've established that we needed to kill the connector because it's not scalable, what should take their place, David?

David Brown

The issue is not all API providers provide a schema.

Well, at the time we were building these connectors, API schemas were evolving and API schemas, there were a number of competing formats for those schemas as well. But those schema wars were largely resolved as we were trying to find a better solution. And I asked Swagger and OpenAPI, the successor became the winner of those wars and everyone has sort of consolidated for RESTful APIs on the OpenAPI specification. So, that meant that if an API provider or publisher provided an OpenAPI schema and OpenAPI schema is a machine readable format. It's human readable as well, but a machine can read the schema and interpret it, run through all the operations, the security required by the API and the like, and machine generate the services that are required to interface with that API so that the integration platform is now speaking the same language as the connector with the services to interface to that API.

So, it bypasses the concept of the connector completely. The issue of course, is, not all API providers provide a schema. Whilst we encourage API publishers to make their APIs discoverable by providing an API schema in any format, whether it be OpenAPI. And in some cases it's beneficial to provide it in multiple formats, including Postman. Many don't and many will, in the very least, provide some basic web based documentation. And so, what we did is we realized quickly that if we're going to rely on schemas to provide a connectorless experience to consume an API, then we also needed a way so that end users could build an OpenAPI schema if one didn't already exist. And so, we built a tool which was able to basically scrape web-based documentation and it builds those OpenAPI schemas for you.

It's literally just a case of pointing and clicking at the documentation. And API documentation tends to be templated with consistent CSS styles and classes. And so, the crawler leverages that concept and we'll crawl the documentation and build out of that, a compliant OpenAPI schema. And so, as a result, we were able to do that ourselves when we launched this concept. We're able to go through from 200 of the connectors, which were a nightmare to maintain to 2000. OpenAPI schemas are in our marketplace overnight. And even if in that marketplace, there isn't an OpenAPI schema to consume, you can always create one yourself. Of course, with Martini, you can create, you can consume any ad hoc operation without a schema as well. The advantage of a schema is that every single operation from that publisher is available to you automatically.

So, if you're looking at Salesforce, for example, you might want to have a getContact operation and updateContact operation, various operations for different sorts of services you want to perform. An API schema will generate the hundreds of operations that are available in that API, machine generate them for you automatically. If you just need the getContact operation, you can do that ad hoc, but just running through a wizard to build a service to consume that one operation, you don't need the API schema. API schemas just make it extremely convenient to consume an API and all of the operations with it. Now, what the other advantage here is when the API changes, because once the schema is defined and particularly, if you've mapped it to the documentation, you can just rerun that mapping and build the schema again. Or if the publisher provides one, they provide you with a new version of the schema and you can machine generate those services within seconds. And you're now ready to consume the new API as well. So, it's a totally different landscape.

Kevin Montalbo

Yeah. So you've already given us an overview of how we can be able to consume an API using an API schema, but Vivien, can you walk us through the process technically of how you can do this with Martini? How can we consume APIs using these API schemas?

Vivien Jovet

Yeah. So there's two main ways to consume an API with Martini.

The first is to use our wizard. So, we have a wizard to consume an API. It's very simple. You start the wizard, it’s going to be asking you for, basically the specification. So, it's going to be either a file or URL in any format. So, like David said, we support many API, specifically an API format like Swagger, OpenAPI. So, you fill up the wizard page, take finish, and then it's going to be generating all the services that are mapping to the operations of the API. And then you're done, you can already start using the API. Then if you're looking for an API, but you don't yet have the file or the URL for it, you can look in the Toro Marketplace. So, from Martini, you can browse the marketplace for the API, just select it, install it. And then it's going to do the same process, generate the services for you.

And then you can start using it already. And the third solution, like they mentioned, is to use the docs to OpenAPI tool that we developed. So, it's a Chrome extension that you can use. You basically open the documentation website, you start looking at what the recommendation is made, where are the operations defined, the response, the request. You might be choosing the tool, it’s just a simple click on the element. And then it's going to make it for you. Then you run it, it's going to generate the API schema based on that. And then you can either download it, use it in the wizard or either publish it to the marketplace, to contribute to it.

Kevin Montalbo

All right.

David Brown

It's worth noting that the Chrome extension, which we've made freely available in the Chrome store is available to anyone. You don't need to use it with Martini. So, if you're an API publisher and you haven't provided a schema to your consumers of your API, download the Chrome extension and just map your own documentation and build a schema from it, there's a whole bunch of stuff you can do. Once you've got the OpenAPI schema, you can build machine generated services and the like, and then start doing API first design using that API schema and the like, and using tool sets, which is of course, including Martini. But in the very least you can use the tool to create a schema and provide it to your end users so that your API becomes easier to consume.

Kevin Montalbo

All right. So, Martini is just one part of the puzzle in the entire Toro Cloud suite, entire Toro Cloud platform. And the last time we spoke we didn't have this Toro Cloud platform yet, but now in 2021, where we're starting the year strong with a Toro Cloud platform. So, I want to talk about the talk about the Toro Cloud platform. David, what is the Toro Cloud platform?

David Brown

Well, it's an entire ecosystem to facilitate digital transformation. So yeah, digital transformation, whilst a cultural change, also typically requires technology. And there are certain projects which revolve around digital transformation and in particular, data and the management of data, monetizing data, and the flow of data. And so you typically use certain technology solutions to facilitate all this. And to become this new agile organization, which can respond quickly to the market and develop and consume digital services. And so when you're wanting to do that, when you want to become an agile organization, you have to change your IT processes, and you have to change the way you build applications, the way you consume applications, and importantly, creating an ecosystem out of your applications so that your end end-users be it your employees, your business partners, or your customers, get a cohesive, seamless experience between them as well.

And so when you're embarking on those types of projects, you're typically looking at the very least, an integration platform. And so that's where we started. Martini is an integration platform as a service. And it does stuff like application integration, getting all of your applications talking to each other, your CRM system talking with your accounting or ERP system, talking with your auto management system and inventory systems, even your line of business applications or custom applications you've written in house and or databases. So, the application integration part of the puzzle, we have solved now for a few years with Martini. It also does things like APIs management, creation of APIs and the deployment of APIs. And so, we facilitate an API first approach to application integration and development. And this is a new way of services orientated architecture, thinking about the way people are going to consume your services before you start building.

And so, by designing the interface first, by designing your API, you're enabling those stakeholders that are going to be consuming that service to have a say, in the way it's designed and what they want to see to be consumed out of it. They also, as a result of that process, have some sort of ownership because they were involved in the process and probably much more likely to use it. And so, we have a process by which you can do API first design. You can create these APIs and you can publish them. You can secure them, and then you can start creating digital products and services out of them. You can monetize them. So, for example, we facilitate monetization of PI APIs, and that means a monetization of data. That means unleashing data, which is locked up in data silos, be it databases or legacy systems or whatever your proprietary processes are.APIs enable you to create digital products out of those processes or data that you may have. And that's evolving your business into a new digital business. You know, dealing with the customers and employees and other stakeholders digitally. You're creating a cohesive experience for them by having this seamless flow of data between your systems. But then the next thing, the final piece of the puzzle, which we solved more recently was a full blown application development. So, what we started with Martini enabled you to build a low-code microservices and with a particular emphasis on integration. So, these were a low-code in the respect of their use like Lego block type mentality, where you drag and drop Lego blocks to build out a service. And often in case you can build a service with zero code at all, but inside, we enable you to write code, even when you need it, enabling full enterprise class functionality per service, but you basically end up with a service with far fewer lines of code than you otherwise would have to.

So, maybe a service might consist of two or three lines or six or seven lines instead of several hundred lines of code. So, they become much more manageable. They'd become microservices, and they would have API interfaces by default, they have RESTful interfaces by default, like a microservice would have a known interface. And so we already had this backend which enabled you to build these backend services with these RESTful APIs. And what we were missing was the front end to be able to complete the application development life cycle of building the end user experience. And so more recently we deployed an application called Bellini, which facilitates low code application development of particularly, the user front end. It has a unique take on user application development. So, it is also a drag and drop user IDE. As part of that, you drag and drop components onto a canvas and you bind those components to an API.

So, those components may be for example, a form. And you want to populate that form with contact details from your Salesforce database and the ability to edit that contact record, and then post it back to the Salesforce database, for example. So you're able to drag a form onto your canvas and then bind that component, that form component to the getContact operation of the Salesforce API. And then when you click the button of that form, it's binding that button operation to the updateContact record operation of the Salesforce API. So, this mentality of everything being API centric, we are a low code API centric, digital transformation platform, which enables you to build applications, integrate applications, create APIs, transform data, manage data, integrate data, and automate workflows and build you know either synchronous or asynchronous workflows, which may require human intervention or just occur as they're invoked. And so, all of this toolset completes product tool set, which encompasses a lot of projects associated with digital transformation. And so, we call Toro Cloud, a digital transformation platform.

Kevin Montalbo

All right. That's a wrap for this episode of Coding Over Cocktails. Thank you very much, Vivien, for joining us.

Vivien Jovet

Thank you for having me.

Kevin Montalbo

All right. And thank you as well, David we'll join up again soon in the next episode of Coding Over Cocktails.

David Brown

Always a pleasure, Kevin.

Kevin Montalbo

Alright! To our listeners, what did you think of this podcast episode? Any thoughts going connectorless with APIs? Let us know in the comments section from the podcast platform you’re listening to. Also, please visit our website at www.torocloud.com for a transcript of this episode, as well our blogs and our products. We’re also on social media, Facebook, LinkedIn, YouTube, Twitter, and Instagram. Talk to us there, we’ll listen, just look for Toro Cloud.

Again, thank you very much for listening to us today. This has been Vivien Jovet, David Brown, and Kevin Montalbo at your service for Coding Over Cocktails.