Process Automation is defined as “a centrepiece of digitalisation efforts” – where workflow engines are used as “a vital building block in modern architectures.”
In this episode of Cocktails, we talk to the author of Practical Process Automation: Orchestration and Integration in Microservices and Cloud Native Architectures, where we explore the relationship of automation and workflow engines, the tools and technologies to successfully implement process automation, as well as the relevance of isolation and boundaries in automation.
We also dive deep into Robotic Process Automation, how organisations are finding success with it, the risks involved, and why he considers it as a "short-term painkiller".
Welcome to Episode 37 of the Coding Over Cocktails podcast. My name is Kevin Montalbo. Joining us from Sydney, Australia is Toro Cloud CEO and founder David Brown. Hey David!
All right. And let me introduce our guest for this episode.Throughout his career in software development, our guest has helped in automating highly scalable core workflows at global companies including T-Mobile, Lufthansa and Zalando. He has also contributed to various open source workflow engines.
He is the co-founder and chief technologist of Camunda, an open source software company reinventing workflow automation. He also co-authored "Real-Life BPMN," a popular book about workflow modeling and automation. His latest book, Practical Process Automation, came out just last March of this year. And we’ll be talking about it today.
Ladies and gentlemen, joining us for a round of Cocktails is Bernd Ruecker. Hi Bernd, welcome to Coding Over Cocktails!
Hi, Kevin. Thanks for having me.
All right. Okay, pleasure. So, I guess we should start by defining process automation. So, maybe you can give us a nice definition of process automation.
I'm not sure if I could give a nice definition. I mean, process automation is a pretty broad topic in general. It means you want to automate typically the control of some kind of process, or I would at least limit it to some kind of business process in this conversation. But this is still a very big field, so we can use very different means to automate processes. It can be something like RPA. We’ll probably talk about that later on, robotic process automation. That can be something very low code. It can be software development, and there are a lot of areas in software development to automate processes. Very different ways.
So in general, it's very important because we need to automate much more nowadays for a lot of reasons and [for] digital transformation to be more agile, to be more efficient, to be just more compliant with certain processes and so on and so forth. There are a lot of reasons for automating. But the ways we can do that, there are a lot of them and that also makes the topics overloaded and sometimes so complicated to talk about.
And I guess that's why you came up with the title Practical Process Automation? Is it because there are so many of them?
Actually, not. So, let me try to clarify that. So, I do that in the foreword of the book. So, I try to sort out the market a little bit and I focus on one, let's say, segment of the process automation market, which is what I call “developer friendly” automation. And I could even narrow that down to “developer-friendly” process automation by using a technology like a [inaudible].
That's kind of what I'm looking at in the book. I tried to make that clear in the subtitle of the book. And that's what I'm looking at. I called it “practical'' because it's basically rooted in all the real-life experiences I had with a lot of companies, customers over the last almost 15 years. And that's why I call it practical. I'm not so much into just thinking about theory and concepts. I always want to know, “Okay, what are the customers doing? What are the real life problems they stumble upon everyday?” And that's why I call it practical.
And, do you go through case studies in the book? Real life use cases?
In a way, yes. I mean, you probably know that so very often you have an idea. You can't openly talk about the most interesting problems or the failure stories, which are normally the ones you can learn from most. So, it's normally not easy to disclose use cases. So, what I do in the book, I related to, let's say, relatively real life cases, like I’m looking at order fulfillment or customer onboarding or certain processes, certain scenarios we see with every customer and I leave in the learnings I did with the customers very concretely without naming them, without making it a concrete case study.
But I think that works relatively well. So far I have gotten good feedback for that. And then I also tried to be consistent in the example, not to switch it in every chapter, because that's normally a bit annoying as a reader.
What makes a process complex? Is it the steps within a process and the decision-making within those steps? Is it transaction volume? Is it infrastructure and technology?
Do I look at one or two steps only? Then I'm not really looking at what I call process automation.
It's typically a mixture of all of them. First of all, it's kind of the number of steps. I find that one of the most important things to look at first is, “Do I look at, like, one or two steps only?” Then I'm not really looking at what I call process automation. That's more of task automation. I want to automate a single task in a certain scenario. But if I have a lot of tasks in a sequence and probably go left or right under certain circumstances, then this is the process automation.
And then I can look at, like, the number of steps which normally relates to how many, for example, different IT systems. I have to integrate how many different teams, I have to send tasks to how many teams I have to interact to to basically specify the process or roll it out. I might have to look at the size of the organisation, the technical environment I integrate with, of course, the volume I want to run over. It's a very different thing. If I run one process a day, or if I run a million per second or whatever. So, these are all aspects that count into the complexity of a process.
When we talk about, say for example, API design and implementation, we have a methodology called API first where we get stakeholders in a room and we collaborate on the design and iterate through the design with those stakeholders before only implementation. Do you recommend a sort of similar process with UML diagrams and that sort of thing? So, with the stakeholders to define the process upfront before any implementation?
Yeah. So, if we look at process automation and workflow engines, there's something called BPMN, which I find super interesting Business Process Modeling Notation. It's an ISO standard for these tiny processes. So, I wouldn't go for UML. You could probably do that, but BPMN is much more advanced in how we can express processes. And one of the big features of BPMN is that you can execute these models directly. There's no weird, translational code generation, it's simply defined however a flow engine has to execute that model.
And that gives you a lot of power also for the early phases, because normally of course, you start with very often, a drawing on the whiteboard, discussing certain requirements, probably drawing a first model, but then you can go into exploration very quickly. You can deploy that on a real engine and then probably you don't automate all the service tasks you want to automate later on, but you do what's called a user task.
So, you open the app form for the user instead in the first step. So, you can click through a process model and say, “Okay, this now opens a form” that would normally not happen because now it's automatically whatever is sent to Salesforce and so on and so forth. But it allows you to discuss certain requirements with business stakeholders, which normally don't think in abstract models. They think in forms and screens and tasks, and what happens next. And this allows you to get into that, that integration of understanding the process, automating it, step-by-step, improving it very quickly. And that's actually a super powerful concept. Have we seen that working quite often?
Yeah, I understand. And what about the developers themselves? Do they ever require any specific skill set to implement process automation?
I mean, it's kind of the struggle I also have in the book. So, if you look at that topic specifically, it depends very much on the tooling you use. And in the book, I try to be as tool-independent as possible. I mean, I co-founded a process automation vendor, so you can't always be a hundred percent independent. But I try to, and for this question, it's the same problem.
So, you have to look at the specific toolings and there's a big variety of these toolings. So, if I look at workflow engines and what I call “developer friendly” workflow engines, then the idea is to bring this methodology and that technology into the normal world of the developer. So, it means that it can still write code in the programming language. You normally write it, you can write test cases as you normally write it.
You can use continuous integration, continuous deployment, like you normally do. You put it in your version control and the workflow engine is not an alien in that regard. And then if you succeed with that, and there are a couple of tools which are really good at that, then it's not much you have to learn as a developer. You basically have to probably learn the language, like BPMN. You of course have to understand what's the workflow and, “What does it do for me? What's the API?” in order to use it.
Just as an example, I'm doing these trainings for O'Reilly also as a complement to the book, it's a three-hour training. So, we don't have much time and we do a live lab and the attendees automate the first process, including a service task, writing some code in roughly 20 to 25 minutes. And if they have coding experiences, it's not a big deal. The big complexity is normally then the next step, like, “Hey, what's the relay API? How, how do I integrate that? What are all the parameters? How do I do the data flow and process?” That's not so much a technical problem of, “How do I use a workflow engine?” It's more like if you have a complex problem to solve, it's complex.
Well, you dedicated an entire chapter of your book to autonomy, boundaries and isolation. Tell us about these concepts and how they influence how we automate.
So, maybe I can give you a bit of the background story, why this ended up as a complete chapter in the book, and actually the next chapter is also rooted in the same story. So, if we look back in history, like, let's say 10 years back, there were these ideas of BPM and so on. That was pretty hard, like 10, 15 years back and BPM for business process management and SOA for service oriented architecture.
And the idea was that we built services in the SOA layer, and then we basically plucked them together in a process on the BPM layer. That was kind of the architecture. They always saw the pictures you saw. It was always compared to Lego. You have that Lego building bricks as the services, and then you just build the process. And that didn't work too well in a lot of companies for a couple of reasons.
But the big reasons were basically all the same or okay, the tooling was not really good at the time. It was totally not developer friendly. It was very, very unhandy, but that was the one side. But the other side, architecture-wise, was that you distribute it in a way of ownership. So, normally if you implement some business requirement, you have to adjust the process and you have to adjust at least one service. And that meant you have to walk around asking different teams, different people, “Hey, can we coordinate on the deployment? How can we get that feature live?” And then normally, BPM was organized like, “Hey, there's this BPM team. Let's go to the BPM folks. They have to help us with that.”
And this didn't really work well. And a lot of these learnings also led to these microservices ideas, where you looked into, “Hey, let's build that one business capability. It does something meaningful for me.”
And then you have that one team caring about all aspects. And having this BPM layer doesn't doesn't fit in there because you distribute the business logic into the microservice process and divert a lot of discussions about that. And I had hundreds of them with different people, especially three or four years back. And this led me to write all the thoughts down in a couple of talks and then in the book and describing how you can bring that together, how that can go hand in hand. It's actually very simple, but it's important to understand that if you implement a business process, if you automate a business process, that's typically a business capability.
I mean, you do order fulfillment. So, that's business capability. You do a new bank account opening, that’s a business capability. So, you implement that as a microservice, if you think in microservices, at least. And then that uses certain process automation, technologies like a workflow, for example, it's an implementation detail. So, on an architectural side, the difference would be, it's not a layer on top. It's kind of within the microservices box. And that sounds like a small change, but it's actually a huge change in terms of how you think about that.
And this is how I described the chapter, going a lot into these questions. For example, isolation means now, the microservice service team can fully decide how to automate that process. They might hard-code it, they might use workflow engine A, workflow engine B or whatever. It's basically their decision. The important thing is from the outside, you don't see that. You have an API where they say, “Hey, kick off the order fulfillment,” for example, and then this works. And that's an important mind shift we saw happening over the last years and it actually works very well.
What triggered this mind shift? Was it microservices themselves and these domain specific boundaries of microservices which kicked off that thought process?
If I look at it from the process automation perspective, yes, that was definitely the microservice movement. I mean, you could then ask what triggered the microservices movement, because that was kind of the initial thing. And I think that a lot of these problems, we have SOA and not distributed ownership, right? And not distributing or not assigning that to the right teams and having an organization that doesn't really fit to how we implement software. And that was kind of the initial trigger.
I understand. Let's talk about robotic process automation. You mentioned it at the beginning. And I think there is some confusion about the boundaries between workflows and process automation, business process automation, and now robotic process automation. Is it just another acronym that the technology industry loves to, you know, create? Is it something that we've been doing all the time or is it something different?
RPA’s an interesting topic. You have to love and hate it at the same time.
RPA’s an interesting topic. You have to love and hate it at the same time, I think. So, for the name, for example, we can probably come back to that later. I think the name is a big problem. It's not very accurate to what it does. And that makes it more like a hype thing.
But if you look at what RPA does, it basically means that you use some technology to steer user interfaces automatically. And this allows you to automate things by clicking around like the computer clicks around themselves. And this is very powerful in a way that you can easily automate things you did manually before. And if you do these kinds of products, normally what you get is a very instant return. I save money pretty quickly, and that makes it so great, actually. That makes it a big success.
And we see a lot of customers applying it in the first products, seeing the return and then start to invest more and more. So, that's the, let's say, the good side of RPA. And there are a lot of good things around RPA. Especially if you, for example, have a legacy system that doesn't provide APIs, RPA can help because then it can integrate that in automation.
But there's also a big risk. And I think that's not transparent enough for a lot of people or they just don't have it in front of their head. Because these integrations on your eyes, they're typically very brittle. That's the one problem we see happening. So, if you just update a browser version, sometimes things break. If you update operators, if versions update. So,they all normally need a lot of maintenance on a very fine-grained level.
That's the one thing probably, but what I find is actually the biggest issue with RPA, and now I come back to the name and sorry for doing such a long sentence around that, but it should say “robotic automation.” People think about processes and this is not what RPA does. The idea of RPA is, from my perspective, task automation. They automate a single task, like, “Hey, enter this customer in this system.” And then it's still in a way a process because you have multiple steps to do that. “Hey, go to this view. Go there, click there, do this.”
But it's not really a process in what I call a business process. So, it's not multi-step. And if you then start to apply RPA for these real business processes, then it normally gets a mess because you start to mix granularities of clicking around in the UI and like the bigger steps in a business process.
And yeah, this gets really, really problematic over time. We have a super story which you can refer to which I love. With [inaudible] telecom, for example. And there's a whole recording of talks and slides available. So, you can dive into that.
And they also started with RPA with a small use case. A task automation for basically a line diagnosis. A technician does that. He had to call in before now he could use the phone and the bot basically steers UI instead of human. And that was a big success. And then they started to move into RPA and they ended up with, I think, 3000 bots. They had actually saved a lot of money with that. So, it was kind of a big success, but then they ended up doing bigger processes and they understood that this is not maintainable anymore, that they can't really understand the processes.
People think about processes but this is not what RPA does.
They got a lot of spaghetti integration. They had a lot of issues with that. So, then they did something which I think is the right thing to do. They separated the bot layer, the task automation from the process automation layer. So they added a workflow engine which basically orchestrated certain bots. And then they reprocessed on that process orchestration layer. And then they called out for the bots whenever their tasks turned for them.
And that was actually a big success. And now they have this architecture for all bot orchestrations. And I find that important aspect of RPA. This also then allows us to replace bots with real API calls whenever possible, because the business process itself doesn't really change. What's in that one box that's changed, you no longer use a bot, but an API. And they do that. I think it was RBS who called RPA technical debt. So, everybody has technical debt because they basically say it's brittle. So, we have to think about basically the strategy to remove it later on, and then you can easily do that. That was a long answer.
It’s interesting that you find it as technical debt. I mean, most people would be just starting to embrace RPA. And they're so far down the track, but now we’re considering this technical debt, can you elaborate on that a bit further?
Yeah. I think, as I said, it's actually a little bit of both sides and I know that technical debt, the term for that, always leads to discussions, but the idea is really okay. You can easily apply it. You can do it, but it's not a really long term solution, especially because of the brittleness of these spots. If you have a real API integration, that's much more stable. It’s much easier to test that. Again, all the good software development practices. We know we can easily apply to API integration, but not to a bot. And that makes it least risky.
It’s almost like a work around that. We're doing UI manipulation or we're scraping a PDF file or a website, because there is no API available. If there was, we would make a direct API and do it properly.
And even if the API is available and that's very often the bigger problem, you don't have the right people to do a proper integration. You don't have software developers, or you do have them, but they have other priorities. And that's what we see a lot.
And I think this is the biggest risk we have there. I mean, on the good side, you can automate something. Even if you don't have these people at hand, if you use RPA, that's cool. But in the long run, that gets very risky. But for example, [inaudible] telecom does that very often, because I also asked them, “Hey, if you do all these RPA stuff, doesn't IT hate you?” It's like, they probably get all the problems with that. And they said, “No, we even fostered our communication because now, if we want to do an integration, we talk to each other and they say, ‘Okay, we can do that. There's an API, no problem, but we don't have time.’ Now we can do that in a year from now, why don't you do an RPA bot in between?”
So they propose that as a temporary solution. And if you apply that for this kind of thinking I think then it's fine. You just have to manage that risk. And of course the risk is that you don't replace it in a year.
It’s what you referred to as a “short-term painkiller”, RPA being a short term painkiller. Full time solution before we can get to where we want to be with this task, which is a proper API implementation.
Bernd Ruecker: Exactly. And painkillers have the risk that you don't feel the pain anymore.
Yeah. Interesting. You've been doing this for, you say 15 years or so, and you've obviously seen a bunch of evolution in the vendors and the process and microservices and the way you architect business processes, like you're saying through autonomy boundaries in isolation. How do you see it continuing to evolve? Where are we at now and where are we moving towards in the future?
So, a couple of things, again, if you look at the higher level of process automation, I think we see these different categories of tools, not “melting together”, I wouldn't say that, but getting closer together. So, we have RPA, they also extended their execution capabilities for automating processes. There's process mining. That's a pretty hot topic as well currently.
So, if you have implemented processes, which are not virtual and process automation tools, we still get some visibility and understanding of these processes. And we have these first called CPA to digital process automation. That's kind of the category of these for clients and where we are also located. And we extend our intelligence as well as the integration to RPA. So there, these are the big three categories and somewhere in the middle, there are some use cases melting.
And I think this all has just started. So, we’re seeing a lot of adoption of these tools. So I would assume for the next five to 10 years, we simply see people doing it. So there, I don't expect too many big disruptive ideas here, to be honest. I mean, we had a couple of them, but overall, we simply have to do it. That's my feeling.
Of course, on the level down there, there is a lot of innovation and disruption happening. So for example, most of the tools are now also provided as a cloud service. That's a big change. I know you have cloud services for providing process automation, providing the analyzers. So it's much easier to apply them. And we can do it on a very different scale. So, we have use cases now where we look into payments for example, or trading. So, high volume, low latency use cases, which we never could tackle with this kind of technology 10 years back. So, you can apply to much more use cases nowadays. So, there's a lot of this kind of innovation happening. So, my expectation is that we simply also see it much more over the next five to 10 years.
Yeah. I'm guessing technology is going to push the boundaries whilst the process itself doesn't change. Things like edge computing and the internet of things, devices and stuff, being incorporated within business processes and workflows is going to make sure that the tool sets and vendors have to be able to incorporate these technologies within that, within their own state to accommodate a business process.
But the underlying business process itself really is not changing significantly. As you said, it's really now about that digital transformation that everyone's going through and for the next five to 10 years is just going to go through the motions of starting to digitize it.
Yeah, I would agree. I can double down probably on what you said with the IoT stuff. So, what I'm also seeing is a lot of use cases emerging in that sense, like having a lot of events, getting insights out of the events and that leads to a lot of like automated processes where you have to react on stuff. So, there are a lot of things happening, but despite that, it's exactly like you said, we simply have to digitize a lot of things now.
And very, very interesting to talk to you. Where can our listeners follow you on social media and stay in touch with what you're writing and doing?
So, the easiest is always either follow me on Twitter or on LinkedIn. I mean, that's our typical source, not a big surprise. It’s “berndruecker,” so if don't know how to write that, probably make a hint i in the podcast, but it's B-E-R-N-D-R-U-E-C-K-E-R. And that handle is typically used everywhere. Github, LinkedIn, Twitter and so on, so forth.
Great. And your book Practical Process Automation as well at Amazon.com and on O'Reilly.
If you go to the homepage, maybe that's easier to remember. It's Processautomationbook.com. You can even get, currently, a free PDF copy that’s sponsored for the next, I don't remember exactly, six months or so.
Brend, thanks very much for your time today. It was very interesting to have you on the program.