Building a sophisticated CodePipeline with CDK in a Monorepo Setup

Rate this content

Many companies are going all-in AWS and thus adopting their complete CodeSuite for their CI/CD processes. However, while CodePipeline is the platform for this process, it may not be the most user-friendly. In a Monorepo setup, it's typical to create multiple CI/CD pipelines for each package. However, there are several caveats to be aware of. For instance, you may encounter scenarios where multiple pipelines get triggered even if you just modified one file, or you may question the need to create multiple branches for each pipeline. In this talk, we provide valuable tips for building a sophisticated CodePipeline using CDK in a Monorepo environment. The techniques discussed in this talk are also transferrable to other CI/CD tools.

8 min
15 Feb, 2024

Video Summary and Transcription

Imagine starting as an AWS DevOps engineer in a small company. Your boss wants an all-in AWS approach, with CICD entirely on AWS for automated deployments. Use AWS CDK for creating a pipeline, stages, and actions. Learn how to handle multiple pipelines for different accounts and handle manual approval for staging and production environments. Discover how to trigger pipelines with EventBridge and Lambda, and create sophisticated pipelines for different scenarios.

Available in Español

1. Introduction to AWS CDK and Pipeline Creation

Short description:

Imagine starting as an AWS DevOps engineer in a small company. Your boss wants an all-in AWS approach, with CICD entirely on AWS for automated deployments. Use AWS CDK for creating a pipeline, stages, and actions. Import code from CodeCommit, create stages with CodeCommit source action and code bit action. Next, deploy on different AWS accounts.

Hi. Imagine you started as an AWS DevOps engineer in a small company and your boss wants us to go all-in AWS. Of course, the CICD should be entirely on AWS for automated deployments. All the infrastructure should be put into code. And we are using a monorepo containing a backend and frontend both in TypeScript.

Great. You have been working with GitLab CI or GitHub Actions where you just need to define the Yammer file and it creates on the platform the pipeline for you. On AWS, it's a bit different. Each resource you have to create by yourself. So you decided to use AWS CDK because you can use TypeScript. You can use the same tooling as the developers. You are able to put the code inside their monorepo. And you have also a nice abstraction, which also the developer may understand.

Let's get your hands dirty. So you start by creating a pipeline using this abstraction new pipeline. And each code pipeline contains stages. And a stage contains actions. The actions result and outputs artifact. This artifact can be stored into an S3 bucket. So you import the code from CodeCommit where the monorepo is hosted. And you can use the to create your first stage by using the add stage. And here you create the CodeCommit source action. So this will be triggered whenever something happening on the main branch. And you create a second stage by using the code bit action. And in there, you use a code bit project. The code bit project you can take, you can define each phases. In this case, I use the build phase, where I want to install linting and test unit. And the result of this action can be then stored inside this folder in this case. And now the next stages are the deployment stages. So you want to deploy on different accounts on different AWS account.

2. Handling Multiple Pipelines and Manual Approval

Short description:

Create multiple pipelines for different accounts using CDK's abstraction. Use a custom method for pipeline creation, assuming roles and deploying with CDK. Handle manual approval for staging and production environments. Create separate pipelines for front end and back end deployments. Use environment variables to trigger the appropriate pipeline.

So you create the loop over all accounts. And you can also make use of CDKs abstraction. So you can create your own method, in this case, the create pipeline project. So instead of using this code, again, you can actually make it like easier by abstracting that and putting it in your own method, and then just make use of each phase. First, you assume the role, and then you use the CDK deploy command. After the deployment, you can run integration tests. You can also use Canary operations. In the staging and production environment, we have the manual approval. This manual approval action can be merged together with the corporate action. Only if it's not on default, otherwise, we just skip the manual approval action. But each time there's a change on the front end, the back end will also be deployed. To handle that, you need to create multiple pipelines. Now you have a pipeline for the front end and another pipeline for the back end. This could be the code, how it looks like. You could find multiple pipelines, and you can handle it by using environment bio.

3. Triggering Pipelines with EventBridge and Lambda

Short description:

Learn how to trigger pipelines with EventBridge and Lambda. Each AWS service emits an event that can be caught using EventBridge. Use a single Lambda function to handle events and trigger the appropriate pipelines. Define pipelines and their triggers inside the Lambda function. Handle file changes and trigger pipelines accordingly. Use CDK and abstraction methods for efficient pipeline creation. Utilize EventBridge to trigger pipelines based on events like pull requests. Create sophisticated pipelines for different scenarios.

But the real question will be when to trigger which pipeline. And the solution for that using EventBridge and AWS Lambda. So you have to know that every AWS service emits an event. In this case, whenever something happening on the code repository, then we want to catch this event. We want to catch it by using EventBridge. And we want to tell the pipeline, hey, use EventBridge for emitting this event. And this event will be then forwarded to a Lambda function. We define the Lambda in here. And with this code, we're making sure we only have one Lambda. Otherwise, we would create multiple Lambdas.

But with this code, we're making sure we only have this Lambda one time. And the Lambda makes use of the code commit get differences. So it checks what file has been changed, and then it triggers the pipeline accordingly. So the inside the Lambda, we define each pipeline. So we create an object where we have the front end, the back end front end stack pipeline and the back end stack pipeline. And we also define when this pipeline should be triggered. And here we define the path of each pipeline.

So in this case, whenever a file changes in the back end folder, or we change something in the back end stack, then the back end stack pipeline should be triggered. We also have the ability to have common folder. So let's say you have like a package, which is shared between the back end and the front end. And that we can use the spread operator in order to trigger all the pipelines. And we make use of the JavaScript SDK for code commit and code pipeline. And the handler uses the get differences command to get all the file changes. And then we loop over each pipeline and loop over each pipeline path to check if the filename is part of the change. And if so, we start to execute the code pipeline.

So what we have seen here is we are able to use CDK to make code pipeline less suck. We can also do something like platform engineering to make use of abstraction so that anytime you create a new pipeline, you can make use of certain methods like the create pipeline project method. We can use event rich to trigger each pipeline accordingly. So we can also say whenever there's a new pull request, we trigger then the pipeline, or there are many other events, you can check it, you can find it online. And with that, you are able to create sophisticated pipeline. Thank you. You'll find all the information in here. You can also find the repository. And thanks for your time.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

DevOps.js Conf 2024DevOps.js Conf 2024
25 min
End the Pain: Rethinking CI for Large Monorepos
Scaling large codebases, especially monorepos, can be a nightmare on Continuous Integration (CI) systems. The current landscape of CI tools leans towards being machine-oriented, low-level, and demanding in terms of maintenance. What's worse, they're often disassociated from the developer's actual needs and workflow.Why is CI a stumbling block? Because current CI systems are jacks-of-all-trades, with no specific understanding of your codebase. They can't take advantage of the context they operate in to offer optimizations.In this talk, we'll explore the future of CI, designed specifically for large codebases and monorepos. Imagine a CI system that understands the structure of your workspace, dynamically parallelizes tasks across machines using historical data, and does all of this with a minimal, high-level configuration. Let's rethink CI, making it smarter, more efficient, and aligned with developer needs.
React Summit 2022React Summit 2022
21 min
Scale Your React App without Micro-frontends
As your team grows and becomes multiple teams, the size of your codebase follows. You get to 100k lines of code and your build time dangerously approaches the 10min mark 😱 But that’s not all, your static CI checks (linting, type coverage, dead code) and tests are also taking longer and longer...How do you keep your teams moving fast and shipping features to users regularly if your PRs take forever to be tested and deployed?After exploring a few options we decided to go down the Nx route. Let’s look at how to migrate a large codebase to Nx and take advantage of its incremental builds!
JSNation 2022JSNation 2022
25 min
The Age of Monorepos
The history of the web can be divided into evolutionary development leaps. The age of inline scripts, the age of jQuery, the age of SPAs, the age of JAMStack...We are now entering the next stage that has been carefully prepared in the past few years. Let me invite you to the world of modern monorepo solutions and share with you the benefits you will reap by using them in every project size and setup. It's time you automate those boilerplate tasks and reduce the bottlenecks so you can focus on what truly matters.Get ready for the next leap! Welcome to the age of monorepos!

Workshops on related topic

React Summit 2023React Summit 2023
145 min
React at Scale with Nx
Featured WorkshopFree
We're going to be using Nx and some its plugins to accelerate the development of this app.
Some of the things you'll learn:- Generating a pristine Nx workspace- Generating frontend React apps and backend APIs inside your workspace, with pre-configured proxies- Creating shared libs for re-using code- Generating new routed components with all the routes pre-configured by Nx and ready to go- How to organize code in a monorepo- Easily move libs around your folder structure- Creating Storybook stories and e2e Cypress tests for your components
Table of contents: - Lab 1 - Generate an empty workspace- Lab 2 - Generate a React app- Lab 3 - Executors- Lab 3.1 - Migrations- Lab 4 - Generate a component lib- Lab 5 - Generate a utility lib- Lab 6 - Generate a route lib- Lab 7 - Add an Express API- Lab 8 - Displaying a full game in the routed game-detail component- Lab 9 - Generate a type lib that the API and frontend can share- Lab 10 - Generate Storybook stories for the shared ui component- Lab 11 - E2E test the shared component
Node Congress 2023Node Congress 2023
160 min
Node Monorepos with Nx
Multiple apis and multiple teams all in the same repository can cause a lot of headaches, but Nx has you covered. Learn to share code, maintain configuration files and coordinate changes in a monorepo that can scale as large as your organisation does. Nx allows you to bring structure to a repository with hundreds of contributors and eliminates the CI slowdowns that typically occur as the codebase grows.
Table of contents:- Lab 1 - Generate an empty workspace- Lab 2 - Generate a node api- Lab 3 - Executors- Lab 4 - Migrations- Lab 5 - Generate an auth library- Lab 6 - Generate a database library- Lab 7 - Add a node cli- Lab 8 - Module boundaries- Lab 9 - Plugins and Generators - Intro- Lab 10 - Plugins and Generators - Modifying files- Lab 11 - Setting up CI- Lab 12 - Distributed caching
Node Congress 2021Node Congress 2021
245 min
Building Serverless Applications on AWS with TypeScript
This workshop teaches you the basics of serverless application development with TypeScript. We'll start with a simple Lambda function, set up the project and the infrastructure-as-a-code (AWS CDK), and learn how to organize, test, and debug a more complex serverless application.
Table of contents:        - How to set up a serverless project with TypeScript and CDK        - How to write a testable Lambda function with hexagonal architecture        - How to connect a function to a DynamoDB table        - How to create a serverless API        - How to debug and test a serverless function        - How to organize and grow a serverless application

Materials referred to in the workshop:,HYgVepLIpfxrK4EQNclQ9w
DynamoDB blog Alex DeBrie:
Excellent book for the DynamoDB:
DevOps.js Conf 2024DevOps.js Conf 2024
59 min
Frontend to the Cloud Made Easy - A ReactJS + AWS Workshop
This workshop enables you to learn how to develop React applications, and then deploy them to the cloud (or building them to the console) coupled with a backend, fully abstracted, with no complex backend configuration, simplifying the building and deployment of frontend & web apps to the cloud.