Canaries in the CloudFront

Rate this content

* intro to DB application architecture (gateway + backend, multi-module react app + component library, etc)
* Deep dive into CI/CD pipeline to do component library build + publish, app deploys pre canary (branch, dev, staging, prod), Lambda@Edge role
* Side  note - multi module applications, enabling the right level of modularity, and B2B SaaS vs B2C
* First canary problem - run full FE regression against stable (release candidate) backend early and often - before FE release candidates are cut
* Second canary problem - run pointed FE regression even earlier - before MRs are even merged
* Third canary question - what does this look like in production? Once again, B2B SaaS vs B2C
* End result - Canaries in the CloudFront! 

12 min
06 Jun, 2023

Video Summary and Transcription

Today's Talk covers designing frontend CI/CD systems, integrating CDNs, and the impact of business type on frontend teams. The speaker shares their experience with CDN integration and its impact on their CI/CD strategy. They discuss canary releases and the challenges of pushing canary deploys to production. Additionally, they highlight the importance of integration testing for shift left and detecting issues before merging to the develop branch.

Available in Español

1. Introduction to Canaries in the Cloud Front

Short description:

Today, I will talk about designing frontend CI CD systems for faster and less risky development. I will also discuss the importance of considering your content delivery network in your application architecture. Lastly, I will explore how your business type and target market can impact your frontend teams' shift left approach.

Hi, everybody. Thanks for joining. My name's Vijay Ramesh. I'm a VP of application development at Demandbase. And today I wanted to talk to you a little bit about something I like to call canaries in the cloud front.

Before we jump in, we only got about 10 minutes today and that's not a lot of time. But in a way it is. There's a lot of content out there. There's a lot of things to learn, a lot of things to dig into and not that much time in any given day. So let me cut to the chase first and talk a little bit about what am I going to learn if you spend the next nine and a half minutes here listening to my talk.

So number one, I want to talk about what it means to design your frontend CI CD systems in a way that enable your teams to move faster with less risk. And as a part of this, what does it mean to think about your content delivery network as an integral part of your application architecture? And then finally, I want to touch a bit on how the type of business that you run, the type of product you make, the type of customers and market that you serve, how that can help determine what shift left could mean for your frontend teams.

2. Introduction to CDN Integration and CI/CD Strategy

Short description:

I'm a VP of application development at Demandbase with over 20 years of experience. We introduced a CDN to our architecture, leveraging CloudFront for edge caching. This impacted our CI/CD strategy as we needed to invalidate the cache with each new build. Our CI/CD system includes dev and staging environments, Selenium integration tests, and GitLab for scans, security checks, and unit tests. We follow a weekly release branch process, deploying to staging for testing before going to production. To enable easier branch review, we built a GitLab pipeline that runs checks, scans, and tests, and pushes the application to the staging origin. We also introduced Lambda at Edge for branch rewriting. This integration with GitLab allows designers and PMs to easily review branch changes.

And just briefly, like I said, so I'm a VP of application development at Demandbase. Over 20 years now, full stack web app type experience, big focus on building rich applications atop of big data and ML platforms, and today at Demandbase I lead three of our application engineering groups.

So before we start, let's actually go maybe one step from before we start, and an architecture diagram like this might be familiar to people who have been doing this for a long time, right? So think in the pre-CDN days, maybe even in the pre-single-page app days, it would be common to have some monolithic system like a Rails app, that app is going to go and be fronted by some sort of load balancer, and requests from a web client are going to hit the load balancer and go back to the backend, and the backend is going to serve up its own frontend. So let's go maybe one step forward here and introduce a CDN, right? So at Demandbase we're using AWS, so we use CloudFront here, and what do we get when we go and introduce a CDN like this? The biggest bang for the buck I think is this idea of edge caching, so if Demandbase is building up its React app and storing it off in US East 1, say in Virginia, meanwhile me as a user coming from Berkeley, California, I want to be able to fetch those static assets from an edge cache that is close to my location, and CloudFront enables this pretty natively, so as a user coming in from California, I'm going to go hit the edge cache say in US West Maybe we've already got the latest build there so I don't even need to fall on back down to the origin, or in the case I need to, the origin is going to go and warm up the cache on the edge so the next user, say coming from San Francisco, gets the full application. And already here, you need to start thinking about, well how does introducing this CDN technology impact your CI, CD strategy. In this case, right, you need to make sure that you're going to invalidate the cache as you deploy new builds, such that a user coming, say from one of these West Coast edges, is able to pick up the latest build. And then also, let's talk a little bit about what are the pre-reqs for the CI-CD system that we started with, when we really wanted to shift left. So number one, dev and staging, right, full pre-production environments, and a way locally for a developer to go run the full frontend application but talking against, say, that staging or that dev backend. Number two, we have a large suite of Selenium integration tests, and importantly these only really will run in the staging environment, due to a general dependency on a stable backend. In the dev environment, backend teams might be testing out feature branches and things like that, and also due to a general dependency on a certain quality and volume of data that's only actually available in our staged environment. Then also, right, we've got a CI-CD system. As I showed, we're using GitLab here. But imagine a system that can go run your scans, run your security checks, run whatever lint things you need to do, and then also run all of your unit tests whenever a developer makes a change set into your develop branch. And then at Demandbase, we're doing a weekly release branch, where, say, every Monday we cut a release candidate from develop, deploy that up to staging, spend Tuesday and Wednesday running a suite of integration, regression, load tests, manual validation, all that, and then deploy out every Thursday to production.

So with that in mind, let's start to shift left. So our first goal was, we wanted to allow UX, say, designers on the team or PMs or even other developers on the team, to be able to really fully review a branch change before merging it in. But of course, right, our designers and our PMs often don't know how to use Git. They don't want to learn how to use it. Even for a developer, we don't want you to have to stash your changes, check out your peers branch, load that locally and all that just to see what all is going on. So let's start to shift left. And here's an example of the GitLab pipeline that we ended up building up. So imagine I've got my feature branch, I'm making it an MR to develop. Like I said, we're going to go and run all of these checks and scans and unit tests and all that. But then importantly, at the end of this process, we're going to go build up the full application and push it off to our staging origin, but in this specific path, right off into those branches and then a named path. And then in order to support this from the CDN side of things, we introduced Lambda at Edge, which allows CloudFront to run some serverless function, either in the request coming in or the response going out. And this allowed us to do some simple branch rewrite. So if I'm trying to go and access a particular branch deploy CloudFront, the Lambda at Edge will catch this, will rewrite the path, and it'll go load the proper origin. And then finally, right, this wires up really nicely with GitLab. So our designers, our PMs can just go look at an MR, click the view app button, and they're off to the races.

3. Canary Releases and Shifting Left

Short description:

Our GitLab pipeline supports canary releases of the latest develop into the staging environment. We use a cookie schema and feature flag management to enable developers to easily switch between the current build and the canary build. However, pushing the canary deploy to production is complicated due to avoiding different versions of the application for users within the same customer tenant. We have a large internal user pool, and their workflows are mission-critical. As a result, we decided to focus on targeted integration tests against our branched employees.

So our designers, our PMs can just go look at an MR, click the view app button, and they're off to the races. All right. So great progress, but we want to keep shifting left. And like I mentioned, the name of the talk is Canaries in the CloudFront, so let's talk canaries now.

So we wanted to support canary releases of our latest develop into the staging environment so that we could run this full integration regression suite, load suite, all those test suites ahead of ever even cutting a release candidate. So again, right, our GitLab pipeline can support this. So now after my changes merged to develop again, we'll run scans. We'll run tests. We'll build libraries, all of that. We'll go auto deploy the build out to the DEV environment, but then also we're going to go and build up this canary package and we're going to push it off to the staging environment again, in some canary path. And doing this then required us to be a little more complex with the way we're handling this Lambda at edge in front of our CloudFront, namely, because we wanted the ability to move automation users back and forth from canary to the current build. We introduced a cookie schema and introduced this into the edge cache key such that a user requesting a will either actually load the canary build or the current build based on the presence of these cookies. And then in our React app, we wanted to add a little flexibility around this. So we're using launch darkly pretty extensively for feature flag management at demand base. We continue to do that here. But then we also wanted to add some method where a developer could easily flip back and forth from the current build to the canary build. And so the React bootstrap ends up doing some checks against the feature flags, against cookie state, and determining, does it need to go and reload from CloudFront, which is going to enable the edge cache to check for those cookies and load the proper build.

So we've got canaries now all the way up to staging. We're using them to test the build ahead of ever even cutting a release branch. So now which way do we go? Which way is left from here if we want to keep shifting left? Well number one, right, we originally thought, let's go and push this canary deploy out to production. So similar to what we do for a lot of our backend services at Demandbase, this will enable us to go say deploy the green build out, run some subset of traffic against it, understand is performing correctly, and then cut all traffic over. However this is where I think it's important to think about. Who are your customers and what is the market that you're in? Demandbase is in B2B SaaS, right? And so in particular, this becomes a little more complicated, because we always want to avoid multiple users within the same customer tenant experiencing different versions of the application. And like I said, right, we use feature flagging heavily, but for the most part, this is always tenant-specific. Right? A particular customer might get a alpha feature turned on, but it's quite rare to turn things on for individual users and have them have a different experience than other users in that same customer tenant. And then finally, we do have this large internal user pool, Demandbase employees who are both using our platform for Demandbase's own marketing and sales and advertising, but also Demandbase employees that are supporting our customers. And as we kind of talk to the PM team, we talk to some of the implementation and customer support teams, we realize that these workflows are really mission critical. And so it's not a good candidate pool to say, hey, we're going to push you off into a new build and then we'll cut everybody over once we're happy. So then we thought about, well, where should we go from here? You know, what does shift left mean in this case? So for us, this meant targeted integration tests against our branched employees.

4. Shift Left and Integration Testing

Short description:

We realized that running integration tests against our branched employees was essential for shift left. Our existing test suites were slow, expensive, and required stable data and backend. However, running these suites nightly against the latest develop was not enough as it was already too late to identify issues. We wanted to detect problems on the merge request before it reached the develop branch.

So then we thought about, well, where should we go from here? You know, what does shift left mean in this case? So for us, this meant targeted integration tests against our branched employees. Like I mentioned, right, we have this full rich integration, load, regression, all these test suites and they're slow and they're somewhat expensive and they require stable data and a stable backend to be running against. But so now with our initial shift left, we're going to go run all those suites nightly against the latest develop. But already, right, it's too late. If there's a problem, if there's a bug, well the code has already been committed. What we would really like is to find those issues on the merge request itself before it ever even hits the develop branch.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Remix Conf Europe 2022Remix Conf Europe 2022
37 min
Full Stack Components
Top Content
Remix is a web framework that gives you the simple mental model of a Multi-Page App (MPA) but the power and capabilities of a Single-Page App (SPA). One of the big challenges of SPAs is network management resulting in a great deal of indirection and buggy code. This is especially noticeable in application state which Remix completely eliminates, but it's also an issue in individual components that communicate with a single-purpose backend endpoint (like a combobox search for example).
In this talk, Kent will demonstrate how Remix enables you to build complex UI components that are connected to a backend in the simplest and most powerful way you've ever seen. Leaving you time to chill with your family or whatever else you do for fun.
React Summit Remote Edition 2021React Summit Remote Edition 2021
43 min
RedwoodJS: The Full-Stack React App Framework of Your Dreams
Top Content
Tired of rebuilding your React-based web framework from scratch for every new project? You're in luck! RedwoodJS is a full-stack web application framework (think Rails but for JS/TS devs) based on React, Apollo GraphQL, and Prisma 2. We do the heavy integration work so you don't have to. We also beautifully integrate Jest and Storybook, and offer built-in solutions for declarative data fetching, authentication, pre-rendering, logging, a11y, and tons more. Deploy to Netlify, Vercel, or go oldschool on AWS or bare metal. In this talk you'll learn about the RedwoodJS architecture, see core features in action, and walk away with a sense of wonder and awe in your heart.
React Advanced Conference 2021React Advanced Conference 2021
19 min
Automating All the Code & Testing Things with GitHub Actions
Top Content
Code tasks like linting and testing are critical pieces of a developer’s workflow that help keep us sane like preventing syntax or style issues and hardening our core business logic. We’ll talk about how we can use GitHub Actions to automate these tasks and help keep our projects running smoothly.
DevOps.js Conf 2022DevOps.js Conf 2022
33 min
Fine-tuning DevOps for People over Perfection
Demand for DevOps has increased in recent years as more organizations adopt cloud native technologies. Complexity has also increased and a "zero to hero" mentality leaves many people chasing perfection and FOMO. This session focusses instead on why maybe we shouldn't adopt a technology practice and how sometimes teams can achieve the same results prioritizing people over ops automation & controls. Let's look at amounts of and fine-tuning everything as code, pull requests, DevSecOps, Monitoring and more to prioritize developer well-being over optimization perfection. It can be a valid decision to deploy less and sleep better. And finally we'll examine how manual practice and discipline can be the key to superb products and experiences.
DevOps.js Conf 2022DevOps.js Conf 2022
27 min
Why is CI so Damn Slow?
We've all asked ourselves this while waiting an eternity for our CI job to finish. Slow CI not only wrecks developer productivity breaking our focus, it costs money in cloud computing fees, and wastes enormous amounts of electricity. Let’s take a dive into why this is the case and how we can solve it with better, faster tools.

Workshops on related topic

JSNation 2023JSNation 2023
174 min
Developing Dynamic Blogs with SvelteKit & Storyblok: A Hands-on Workshop
Featured WorkshopFree
This SvelteKit workshop explores the integration of 3rd party services, such as Storyblok, in a SvelteKit project. Participants will learn how to create a SvelteKit project, leverage Svelte components, and connect to external APIs. The workshop covers important concepts including SSR, CSR, static site generation, and deploying the application using adapters. By the end of the workshop, attendees will have a solid understanding of building SvelteKit applications with API integrations and be prepared for deployment.
GraphQL Galaxy 2021GraphQL Galaxy 2021
161 min
Full Stack GraphQL In The Cloud With Neo4j Aura, Next.js, & Vercel
In this workshop we will build and deploy a full stack GraphQL application using Next.js, Neo4j, and Vercel. Using a knowledge graph of news articles we will first build a GraphQL API using Next.js API routes and the Neo4j GraphQL Library. Next, we focus on the front-end, exploring how to use GraphQL for data fetching with a Next.js application. Lastly, we explore how to add personalization and content recommendation in our GraphQL API to serve relevant articles to our users, then deploy our application to the cloud using Vercel and Neo4j Aura.

Table of contents:
- Next.js overview and getting started with Next.js
- API Routes with Next.js & building a GraphQL API
- Using the Neo4j GraphQL Library
- Working with Apollo Client and GraphQL data fetching in Next.js
- Deploying with Vercel and Neo4j Aura
DevOps.js Conf 2022DevOps.js Conf 2022
152 min
MERN Stack Application Deployment in Kubernetes
Deploying and managing JavaScript applications in Kubernetes can get tricky. Especially when a database also has to be part of the deployment. MongoDB Atlas has made developers' lives much easier, however, how do you take a SaaS product and integrate it with your existing Kubernetes cluster? This is where the MongoDB Atlas Operator comes into play. In this workshop, the attendees will learn about how to create a MERN (MongoDB, Express, React, Node.js) application locally, and how to deploy everything into a Kubernetes cluster with the Atlas Operator.
React Summit 2023React Summit 2023
88 min
Deploying React Native Apps in the Cloud
Deploying React Native apps manually on a local machine can be complex. The differences between Android and iOS require developers to use specific tools and processes for each platform, including hardware requirements for iOS. Manual deployments also make it difficult to manage signing credentials, environment configurations, track releases, and to collaborate as a team.
Appflow is the cloud mobile DevOps platform built by Ionic. Using a service like Appflow to build React Native apps not only provides access to powerful computing resources, it can simplify the deployment process by providing a centralized environment for managing and distributing your app to multiple platforms. This can save time and resources, enable collaboration, as well as improve the overall reliability and scalability of an app.
In this workshop, you’ll deploy a React Native application for delivery to Android and iOS test devices using Appflow. You’ll also learn the steps for publishing to Google Play and Apple App Stores. No previous experience with deploying native applications is required, and you’ll come away with a deeper understanding of the mobile deployment process and best practices for how to use a cloud mobile DevOps platform to ship quickly at scale.
DevOps.js Conf 2022DevOps.js Conf 2022
13 min
Azure Static Web Apps (SWA) with Azure DevOps
Azure Static Web Apps were launched earlier in 2021, and out of the box, they could integrate your existing repository and deploy your Static Web App from Azure DevOps. This workshop demonstrates how to publish an Azure Static Web App with Azure DevOps.