Software Architectures Gone Wild

Hello, everyone. My name is Jamie-Maria Schauven and I'm the cofounder of Deity. Today, I'm going to talk about software architectures and it's going to be wild so wear some sunglasses.

I really want to first, before I start, give some credit to the artwork. It's by James Booker and it's called Random Galaxy Art and it's really amazing. If you want to know more, talk with me or follow us. My Twitter can be found in the left corner. Please tweet and please ask me any question that you want.

Okay, let's start. So first of all, it's 2021. And while we were saving the world in 1944 by actually going to battle, today, we're doing that by staying home and staying on our phone. And most of us are really doing that. And we see that in the Internet traffic. The Internet traffic is growing like crazy. Here is the Internet usage, which was happening in the first part of 2020 until April. As you can see, it's doubling and it's been crazy. And not only the Internet usage. We have the same in the e-commerce world. We were all spending a total online of 82.5 billion in May 2020, which was a 77 percent growth year to year. If you can see in the chart on the right corner, that is only until April. There was 27 percent of all the retail was happening online in the US. And as you can also see, the growth was massive. We've been doing this and they say that the growth has been almost six years. What was projected has been happening right now.

To survive and basically to survive our website, as many websites are struggling with this amount of traffic and this amount of orders coming in, we're going to need some fires on our website. What do I mean with the fires principle? Well, actually, if you want to survive in these times and if you want to make sure that your website is ready for future growth, you're going to need a software system that is flexible so you can build whatever your business needs now and in the future. You're also going to need to have a system that is integrable so you can integrate with any data source out there, no matter where it comes from. And also, you're going to need to be reliable. Your website needs to make sure that you can handle all the traffic as soon as you're growing and as soon as you're moving forward. Obviously, you also need to be extensible.

The markets are changing very fast and you need to be able to adapt new features, new technologies, new marketplaces in a very soon way. And you need to be able that your software can extend without adding any complexity. And last but not least, you need to be scalable so you can scale your business unlimited as soon as your traffic is hitting or as soon as you're growing.

Unfortunately, the flexibility of most websites is really, really bad. It's more like vodka than it is like yoga. And that all has to do with the architectures. So how are sites built? Software architectures are basically the beginning of how you build your system, and you're making a blueprint of each element that needs to be in your site and in the system as a whole. So not just do I need shipping and do I need pricing and do I need an image, but also how these correlate with each other, which system has to communicate with what in the back and in the front end.

Basically, we have three main architectures. We have the monolithic architecture. We have the decoupled architecture and we have the service oriented architecture. I'm going to start with the monolithic architecture. Well, the monolithic architecture is the old and gold one in the monolith. Everything is tightly coupled. The UI layer, the data layer, the processes, and everything is very interwoven. And it looks like this. On my slides? Yeah. So in the monolithic architecture, as you can see, the presentation layer is very stuck together to the business logic and the data interface in the layer design. You find these different layers, but yet they're still very close together. If something happens on the presentation side, your business logic is going to struggle. And if something happens in the business logic, your presentation layer is going to struggle. Also, when you want to change something to your website, you're going to have to change it really inside the layer. So inside the logic, everything is there. Search, images, pricing, stock. Everything is inside that logic. If you want to change your pricing, if you want to change the stock service, you're going to have to break it open and rebuild inside of it. As you can imagine, if you grow, it's going to be a very big mass. It's going to be very, very complex as soon as you add more features to that. So there's a few pros of monoliths. Basically, a monolith is very, very good as the application is packaged and deployed as one.

It's simple to develop. It's simple to deploy. It's simple to test. It's simple to maintain. But only if you keep this application very simple.

Some cons of the monolith, and I'm not sure why there's another slight skip, but very monoliths can fail big time. And what can happen? For example, this is an example of what happened at Magento. We have an admin user. And when the role of this admin user was just doing something, so this person would log in or change a product, etc. The whole cache was going to rebuild. So imagine you are just a business owner and you're just trying to login. Your whole website could literally go down because of this monolith, because of this structure. But this business logic layer is very, very strongly coupled with the presentation layer.

Some cons of the monolith. This brings me to the cons of the monolith. It's obviously that the monoliths are very hard to extend and maintain. Monoliths are not flexible and monoliths are not reliable. And still, most of the modern internet is unfortunately a monolith. So to skip this, you have to go wild. And we call this go wild and you have to go headless. That's happening now. A very big hype at the moment. Everybody's going headless. What does it mean? It means that the front-end is decoupled from the back-end and connects via APIs. The key here literally is to take apart the front-end and decouple it from the back-end. It looks something like this, where the presentation layer is not connected anymore to the business logic. In this way, the presentation layer can work as a standalone application and many times being built as a JavaScript application. That renders the output of this API into HTML. The data you get from the business logic, so the stock data, the images, and everything you get from there. So the only thing you have to do is make sure that the data is available from the business logic.

However, your business logic is still very tightly coupled together. Some pros of going headless... The front-ends are not depending anymore on back-end processes. We have performance advantages on the front-end. We have a high scalability and reliability on your front-end as you can scale and be reliable on your front-end as a whole. And you can be very flexible and integratable on your front-end. And developers can work in separate teams. You have your front-end team and you have your back-end teams.

However, you still need to make sure that everything is accessible via API. Some cons of going headless, therefore, are really data coming via the API can be very, very slow. And you need to expose all data via API. Existing backends are not ready. When you have, for example, Magento or Shopify, they're pretty good. But there's still a lot of custom work that needs to be done to make sure that your data is being exposed to the front-end. And starting from scratch, starting this whole layer from scratch is going to take a long time. Not only do you have to build your front-end team, but you also have to make sure all the data is there and you have to think of all the SEO things. As SEO, I'm not sure why this slide is skipping all the time, but SEO issues with single page applications are there, and especially when you're using JavaScript front-ends.

So some of the major platforms are happily are going headless. We have BigCommerce, we have Magento, we have Shopify and also WordPress and other applications are really preparing for this headless movement. They make sure that all the content and commerce is available via API. So you can add your API layer to that and you can talk to that layer and build your own front-end there. So there's a lot of big improvements going on to move towards this headless structure opposed to this just monolithic structure. But what about PWA? Many times when I talk about headless, people ask me about PWA. So PWA is the marriage between a native app and a responsive website. You can function offline, have push notifications, etc., etc. As we all know, 53% of users will abandon a page when it takes more than three seconds to load. And that's, obviously, very weird when I put it into a real-life situation where I see that when I'm just waiting at the cash register in 3.2 seconds, I am gone. Unfortunately, the average mobile web page load on 3G is 19 seconds. And it's not much better on 4G when we have 14 seconds. If we know that 3.2 seconds is leaving our customer, we're still having this kind of results that I think is really, really bad.

So PWA can really help with this. We have Tinder who've been using it. They make sure that their website was 7.22 seconds faster than their application when using PWA, and therefore, 90% smaller, and they had more messages, more swipes, and longer sessions. Also, Uber tried it, and they were able to load within three seconds on just 2G networks. With that, they were opening a formula market as they could deliver their services in a 3G network within three seconds. Another cool thing you can do with Progressive WAP app is send push notifications, and we can have this kind of stuff coming from websites very soon.

So a Progressive WAP app, what is it? Well, according to our friends at Google, a Progressive WAP app is progressive, responsive, connectivity-independent, app-like, fresh, safe, discoverable, re-engageable, installable, and linkable. These are all feature kinds. I mean, these are all things that describe a behavior, but not a technology. What exactly is a Progressive WAP app? Well, a Progressive WAP app, basically, technically speaking, are three things. It's a service worker, it's a WAP manifest, and it's HTTPS. So literally, we can add a PWA to anything. It doesn't have to do anything with the architecture, and when I look at that in an overview, that's how it looks like, and I don't know why the next slide, why it's not loading. And then it's like this. In the monolithic architecture, the PWA can exist, but also in the headless architecture, the PWA can exist. So the PWA is fully independent from what is happening in your architecture. Unfortunately, the PWA does not give you fire. It's not necessarily flexible. It can be more reliable, but it doesn't integrate with everything. You have to add it to the layer, but it does give you a little bit more reliability, accessibility, not really, and scalability, not really. So PWA is not a solution for these websites that really need to grow more.

So to do that, we need to go to the next level. And for that, we call it the service-oriented architecture. And what we do here is that we bring these features of a headless architecture where your presentation layer is separate. We also bring this to the backend. So we also decoupling the backend. And so that will bring this flexibility, accessibility, scalability, and reliability to both the frontend and the backend. How does it look like? Looks something like this. In the service-oriented architecture, as you can see, we have a very strong middleware. And then we have presentation layers on the top can be one, can be two, can be 10.

And we have business services. And all these business services are not stuck together anymore. They're all separate. They communicate directly with the middleware with their own database without disturbing the rest. The secret of this SOA is obviously this middleware. The middleware connects all the dots between the backend and the frontend, but also transfers the data between different backend services. You don't have to integrate them anymore with each other.

Different from a microservice architecture, as you can see here, where the microservices directly communicate with the UI and with each other. In the SOA architecture, we do that through the middleware. So in the traditional SOA, we use a service orchestrator and an ETL tool. But luckily, we don't have to do that anymore. We have GraphQL now, which is perfectly and can help us do this kind of work without building something very complex.

So, if I move forward, then I'm asking what is the pros of going SOA? We have manageable services. We have separate deployment languages and databases. It's very scalable and reliable. It's flexible and integrable, very extendable without adding any complexity, as you can extend your backends, but also switch your backends and your frontends without having to break down the system. It is a very fast adoption of new features and technology. And the cool thing is, you can use existing services. There's a lot of cool things out there for search, for images, for stock, for orders. You don't have to build all the services by yourself anymore. SOA therefore has Viyas. It has the flexibility, the integrability, the reliability, the extensibility, and the scalability.

Obviously, there are also some cons of going SOA. We have a complex to build from scratch. It needs an architecture for your blueprint, and it's more initial setup work. But eventually, it's very highly future-proof.

But what if we build a light SOA? What if we can make sure that we have a light version that everybody can start with? And that's basically what we did in a project. And what we did is, we made sure that we looked at, what do we need to build this light version? We need a middleware, we need a data integrator, a way to integrate this data, and we need some basic services. And I don't know why the slides are skipping again, but we need these three things to build the light version here. And that's what we did in our company.

So we built a very strong middleware, which can integrate, as you can see in the bottom, with all different industry partners. So we don't build any services ourselves. No payment, we have used BigCommerce for commerce, we use SAP for stock data, WordPress for content, Algolia, etc. You can really make a puzzle and use the services that are out there. You can even make your own services and use what you've built. And then obviously there's a team there on the front end which we use a progressive web platform.

So I'm gonna show you an example of how that works. So here on the left side, I have a demo shop where I'm using BigCommerce and Algolia. There was no direct integration between the two that doesn't even exist. Then on the right side is my backend of BigCommerce where I'm gonna change the price of a product. In the top right, you can see what I'm gonna do. I'm gonna go for my PWA team, ask my server for a search result, who's asking Algolia. Then I'm updating data in BigCommerce which is done sending this information to the server which is immediately sending this to the other server, BigCommerce or Algolia. So I'm gonna search for a clock and the clock is 89 euros. And I'm going to change the price to 99. I'm gonna save it and I don't need to rebuild any index. I don't even have to go to Algolia. Back to the front end, which is a single page application and I just reperform the search and the price has been changed. So the data's going through different back ends to the front end really, really fast. And the results are we have 300 milliseconds processing time and the time to market to build such things or only two to three months. And we can have unlimited flexibility to add any service there to build whatever your business needs. So back, we have three main architectures.

We have a monolith, we have a decoupled architecture and we have a service oriented architecture. Which one is the best? Well, that's basically a hard question because it really depends on what you're trying to build. For a simple block, a monolithic architecture might be perfect. For a web shop, B2C, presentation area with less complex business logic, that's fine. But if you have growth ambitions and you wanna go multi-X, multi languages, multi storefronts, multi back ends, multi stock locations, name it, whatever you want to do. Your multi-X web shop you're really gonna need to have a service oriented architecture to move forward in the future. If you're using headless architectures for that, you're gonna be stuck very soon as your business logic is gonna be overloaded.

To decide what your business needs is really one of the hardest tasks out there. Really take your time to do this and focus on your business and on your ambitions and not what you need today. Many times I see people building of what they need today and not think about what they need tomorrow. That's actually a shame because if you're putting in the time and the work to do it right now, why not do it good? Immediately, you're not gonna build up any technical depth.

That was my presentation. If you want to read more about this architectures, about these differences, I wrote a blog on dev.to about this. You can follow the link here or scan the QR code and you will find it. If you have any questions, please ask me. I'm ready in the Q&A for you and I'm happy to answer every single one question of you. Thank you very much.

Before we go to the audience questions for Jamie, of course, we're going to look at the results from her poll question. So if you don't, if you didn't see the question or you forgot, I'll remind you. The question Jamie asked for you is, the architecture I'm currently using for my recent project is, and the winner is Microsoft's architecture with 39% of the votes. Well, that's a lot. If you haven't voted yet, you can still do it, of course. And Jamie will be able to do something with these results.

So Jamie, hey, thanks for joining us. Thank you. So what do you think? Is this something you were expecting? Like one third, 40% Microsoft's architecture? Not really actually. I also see a very high on monolithic. So that is something I was expecting as there's many projects still are monolithic. But I think it also depends on what type of projects people are doing. I think if it would be very E-commerce, for example, very E-commerce oriented, then that would be much more monolith. If it's more like enterprise oriented, so more enterprise people, probably more on Microsoft of microservice architecture or service oriented way. Sorry, not Microsoft, microservice. We all got you. It's a honest mistake. Don't worry. Yeah. And also, I guess it's the target audience of our conference today, right? So we have DevOps engineers that like to make this architecture possible. So yeah, I'd say it would be fun to do this question again at like a JavaScript or a React View or Angular conference. Then the results would be a lot different, I guess. I think there would be a lot of decoupled architectures. Yeah, well, we'll only know if you apply for another conference from Git Nation, Jamie. And you can ask the same question again. Cool. If you will have me, I will do it again. Awesome.

So we're going to go to the audience questions and let me get them up. So Jamie, when you encounter a big, a big monolith at a client, what would your approach be to break down this monolith into little chunks, a bit smaller pieces, and eventually into a microservices app? So what's a good approach for splitting up a monolith? So for me, that is basically part of why we got into this business. Lots of people were struggling with this same question and they were like, hey, do I need to start all over again? In many cases, that answer was yes. We did a lot of big projects where actually the answer was yes, but we wanted to do it in a better way so we don't have to restart and re-platform and rebuild everything from scratch. So that is why we came up with what we're doing now. So that let's say a light version, a first starting point of the service-oriented architecture. So what we do is we use the current back ends or the ERP systems or whatever they have at the moment, that when we connect to a middleware and we start building the new front end. Then from there we use this triangular pattern. So then we take out all features one by one. So we start, for example, first with auto management if it's e-commerce or with blog, if it's not, if it's like text or whatever it should be. So we really, based on this triangular pattern, we then take out all the features. So then you're already live from day one. So you already have the advantages. If you add new features or if you want to have more scalability, you already have the advantages from day one and then we start breaking down all the pieces until at the end you have as many pieces as you need. So add a middleware to the current system, add a front-end and from there take out all the back-end features.

Well then, would you say just get it over with, get your architecture done with like a big feature or start with something with low business value or of course, everything in your website has some business value, but lower risk than say the checkout function in e-commerce? I don't think the check out function is the lower risk. I think that is one of the highest risk because that's obviously the critical path that you need to do. But I would definitely say, so what we often see is where people are struggling with. So we sometimes see, people are trying to get an ERP system in their current enterprise or try to make their current back-end into something that it's not, warehouse management system, something like that. So we mainly start where what is your main priority? What is your pain point? We also have some people who, for example, start with checkout because they need different currencies or different payment methods which are not available in the current system, or we start with projects that are actually wanted to add new features which don't exist. For example, the example I used in the talk, when you have BigCommerce and you would like to add Algolia to that, then that is your first starting point. You connect to BigCommerce, you add Algolia as it doesn't exist the connection directly. So what to start with really depends on project, depends on where they're struggling with, if it's a new feature, if it's something that they're struggling with. So I would definitely get out the pain points first before moving on to make the whole system more comfortable.

All right, awesome. Thanks. I have a question from Ronny. It says good book, courses and links to start reading and getting into this architectural paradigm. So what are some good resources for getting into this? So there's many things online about software architecture, guys. I actually wrote one blog about, that's when I shared at the end. So that's from my talk, but there is also from Martin Fowler. I'm not sure if I explained the name right. Martinfowler.com slash architecture really has a very good software architecture guide. I can share the link later in the DevOps talk.

I mean, architecture changes, right? So I mean, the architecture itself doesn't really change, but the way we apply it, the way we use it is changing. I mean, things that in the beginning, you know, when we had, if you purely look at the service oriented architecture quite a few years ago, it's very different than what we're doing now. We really have very, you know, things like enterprise service buses and very complex things, ETL tooling, but now we can just do that with GraphQL. So these more complex architectures become more accessible because of new tooling. So really some boops with all this new stuff and I haven't found yet. Obviously Twitter is a very good source, but there's not many people talking about it yet, which I think is a shame because there's some really, really cool stuff out there. And really cool ideas to talk with. And I think we should have more talks and conferences about this subject and have more people actually try it and dive in. I know it's a bit difficult. Many people are struggling, the project managers don't really switch architecture that easily, but I think it's really good to do and to open the conversation on how can it drastically change your approach for the future. Yeah, of course, getting a buy-in for changing architecture is really hard because it's usually a big investment, right? So it's really a big effort to try something new. So I can imagine that it's a big thing, and writing a book on architecture when some things are changing this rapidly in our field, so it's hard to... Well, invest like two or three years of your life to write a book, and then you're done and it's already legacy. You've written a legacy book when it comes out.

Next question is from Jonathan. Key differences with facade better? It seems like facade could act as the middleware that you talked about, and route requests to the appropriate surface. Well, I think that pretty often the combination that you're doing is different services in the backend, so a single request, as what you do with this, can be handled solely by this single service. So I don't think facade would be able to fulfill all your needs. There's a lot of different things going on from different services. Yeah, so I think that could be, but I would do that more in the headless approach rather than really bring it in as a middleware.

It's a single request one way. What we do in the front end middleware is really much more, let's say a backend of the front end, before it goes to the real backend. And that is also like a smart middleware. So it's not just transporting data and transforming the data, but also really be able to catch it as soon as you need it. When there's changes in real life data, we can use it for things like AI, which is not now possible. We see it now happening in commerce. People want to introduce AI technology. They cannot do that because their backends are traditional backends and frontends of the commerce is very slow. With a middleware like this, which goes different directions, which can be very fast responding between backend systems and frontend systems, we can actually use it. If you use just a single request service, I don't think that's gonna work. But to be honest, I haven't really tried it or dived into that. But from the top of my mind, that would not be the direction I would go.

Awesome. Next question is from Mike. How to organize the QA environment when you have so many backend databases? So if you want to do like a full QA, so obviously every team or every service, you need to be tested by itself. But if you do the full stuff, you want to test everything. Mocking the responses could be one of the potential solutions, but that required to set up the whole set of accounts for every service and every backend.

I think that, you know, we do QA, so we test certain things that we expect at certain points. Obviously we cannot, we also have to QA, for example, what would be the response time of one of the services or do I get the expected response time of one of the services? But as soon as you find the buck, it's pretty easy to figure out where it's happening because you know, on the front end, the data is fed up, right? It's coming from one source, so you know very well where it's coming from. In the more monolithic tradition, I've seen things like happening on checkout or on cards, which are related to something with the search or something, it's very hard to figure it out. So, you know, it has its pros, it has its cons. I mean, obviously it's a little bit more complex because it's not as easy to test as a monolith. On the other hand, as soon as you know where it's going on, it's very easy to find it. Yeah, you have to test all the single solutions and obviously test them as a whole and marking the data, marking the responses could be one of the solutions you could use. And we often use.

Okay, great. Thanks. So we have time for one quick question with a quick answer. And the question is from Cesar. When would you not go for microservice architecture over a conventional monolith? Well, it depends obviously on the goal. So there's not one thing which is better than another thing. It really depends on what are you trying to build? I would definitely not go for microservice or a service-oriented architecture when I'm building a block to show my family how my child is growing or how my dog is doing. I mean, definitely not something I would do. Also when you're testing quick things, if you're like more in the startup phase, you want to just see, you know, build some landing pages and just try things out. This would not be first direction I would go. I would go for the service-oriented way as soon as I have bigger ambitions. And we call that the multi-X stores. So the multi-X stores are very good for what we do for the microservice as a service-oriented way. If you have multiple backends, multiple systems, multiple languages on front-end, multiple currencies, multiple goals you want to achieve from your one system. So we call that the multi-X. That's the direction to go. If you're not a multi-X and you don't have a lot of it, you don't have scalability issues, like you just have your parents watching how your child is growing, then I would definitely not go for microservice approach.

Awesome. Well, like I said, so we have time for live Q&A, but for everyone that wants to continue the conversation with Jamie, Jamie is going to be going to her speaker room on SpatialChat. Find the link below in the timetable. Jamie, thanks a lot for joining us and well, having a laugh. Thank you very much.