Infra vs Apps – Where are my Pipelines?

Rate this content
Bookmark

Automation of a single monolithic app is pretty straight-forward. Split it into a frontend and backend and it's still manageable. Throw in more components or infrastructure and suddenly you're scratching your head at why a build ran - or didn't run. How many pipelines do I need? How many git repos should I have? Let's walkthrough use cases from small teams who own their entire stack to organizations with central IT units that manage shared infrastructure. Learn which scenarios and criteria determine how to slice but not spaghettify your pipelines.

Julie Ng
Julie Ng
32 min
01 Jul, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

This Talk on CICD covers various use cases, challenges, and best practices. It emphasizes the importance of people and learning in CICD, as well as the complexity of coordinating teams and managing infrastructure. The speaker shares insights on frontend development, microservices, and security considerations. The Talk concludes with discussions on Jenkins files, bash scripts, and the challenges of DevOps in organizations.

Available in Español

1. Introduction to CICD and My Background

Short description:

Hi, my name is Julie. I'm an engineer at Microsoft and today I'm going to talk to you about CICD and how it all works, and how it all works when you have both applications and infrastructure. I am part of the Fast Track for Azure program, which means I help onboard customers to Azure. Before that, I was an Enterprise Architect at Allianz Germany, which is a multi-billion dollar insurance company, and actually many of the opinions and recommendations I'm giving you today come from that experience as well as my experience at Microsoft. I come from the Mac world, open source. I like Node.js, Ruby, I really don't like Windows. I'm a very opinionated person, so I will try to mention when, yeah, something is my personal opinion and recommendation. The photo here I put nostalgically because it was literally the last week of February before lockdown because of Corona. So, it feels very strange not just to work remotely without ever having met your colleagues, but also giving a talk right now over video. But it seems to work.

Hi, my name is Julie. I'm an engineer at Microsoft and today I'm going to talk to you about CICD and how it all works, and how it all works when you have both applications and infrastructure.

So, a little bit about me. As I said, I'm an engineer at Microsoft. I am part of the Fast Track for Azure program, which means I help onboard customers to Azure. Before that, I was an Enterprise Architect at Allianz Germany, which is a multi-billion dollar insurance company, and actually many of the opinions and recommendations I'm giving you today come from that experience as well as my experience at Microsoft. Before that, I was a full-stack engineer, still am actually, and a designer.

So, I come from the Mac world, open source. I like Node.js, Ruby, I really don't like Windows. I'm a very opinionated person, so I will try to mention when, yeah, something is my personal opinion and recommendation. The photo here I put nostalgically because it was literally the last week of February before lockdown because of Corona. So, it feels very strange not just to work remotely without ever having met your colleagues, but also giving a talk right now over video. But it seems to work.

2. CICD Use Cases and Mono Repo with Jenkins

Short description:

Today I'm going to give you various use cases for CICD. Let's start with a mono repo and Jenkins as the build server. After pushing to the main branch, Jenkins deploys to the production environment. To ensure it works, we need continuous delivery and automated promotion. Running end-to-end tests on the deployed application helps verify its functionality. If the tests fail, the job ends. If they pass, Jenkins commits the changes to the production branch, triggering another job to deploy it.

Okay, so let's start with a very simple example. Today I'm going to give you various use cases. I'm going to try to start simple, and then it gets really complicated really quickly, but the point being I want to teach you how to fish, and not give you a fish, when it comes to figuring out CICD for yourself.

So let's start with the easiest thing possible, right, a mono repo, because we come from monoliths. Very simple. I'm going to make a push and a build server will pick it up. So I have Jenkins here. Jenkins is my favorite server of all time build server. Yeah, I use Azure DevOps and GitHub Actions as well, but I still prefer Jenkins.

Anyway, so let's say I push to the main branch. It's going to deploy to my production environment. Let's say eventually I'm happy. I'm going to somehow on my local computer, merciless changes into production, and then I'm going to push the change to production and Jenkins pushes that over to my production environment. All is good, I think. How do you know it actually works? You know, like that kind of CI just goes there. It does some tasks. But does it actually work? Right. How do you get to the point of continuous delivery? Can you do automated promotion? That is a little bit more complicated than many people expect when they first do it.

So, we still have the same mono-repo, the same sort of monolithic application. We're going to make a push to our main branch, which remember, corresponds to our development environment. So Jenkins will have deployed it. It's all done. Let's run some end-to-end tests. So in this theoretical example, let's say I have even a single-page application that actually has an end-to-end test suite, that will fire up a browser, click through everything. And what my end user is trying to do in the application, we can verify that it works as expected. So maybe I can buy a t-shirt, for example. Based on the results of that test, if they don't work, then we say, oh, failed, end of job, end of story, end of the build job, that is. Let's say it actually works. What you can do is then have Jenkins make that commit for you to that production branch. Whereas before, you might have sort of, you know, by hand, went and clicked through everything to make sure it works, you could run an end-to-end test suite and say, okay, I'm confident, let's put it into production, which will kick off another job, and then we'll deploy it to production.

QnA

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Why is CI so Damn Slow?
DevOps.js Conf 2022DevOps.js Conf 2022
27 min
Why is CI so Damn Slow?
We've all asked ourselves this while waiting an eternity for our CI job to finish. Slow CI not only wrecks developer productivity breaking our focus, it costs money in cloud computing fees, and wastes enormous amounts of electricity. Let’s take a dive into why this is the case and how we can solve it with better, faster tools.
End the Pain: Rethinking CI for Large Monorepos
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
End the Pain: Rethinking CI for Large Monorepos
Scaling large codebases, especially monorepos, can be a nightmare on Continuous Integration (CI) systems. The current landscape of CI tools leans towards being machine-oriented, low-level, and demanding in terms of maintenance. What's worse, they're often disassociated from the developer's actual needs and workflow.Why is CI a stumbling block? Because current CI systems are jacks-of-all-trades, with no specific understanding of your codebase. They can't take advantage of the context they operate in to offer optimizations.In this talk, we'll explore the future of CI, designed specifically for large codebases and monorepos. Imagine a CI system that understands the structure of your workspace, dynamically parallelizes tasks across machines using historical data, and does all of this with a minimal, high-level configuration. Let's rethink CI, making it smarter, more efficient, and aligned with developer needs.
Atomic Deployment for JS Hipsters
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
Atomic Deployment for JS Hipsters
Deploying an app is all but an easy process. You will encounter a lot of glitches and pain points to solve to have it working properly. The worst is: that now that you can deploy your app in production, how can't you also deploy all branches in the project to get access to live previews? And be able to do a fast-revert on-demand?Fortunately, the classic DevOps toolkit has all you need to achieve it without compromising your mental health. By expertly mixing Git, Unix tools, and API calls, and orchestrating all of them with JavaScript, you'll master the secret of safe atomic deployments.No more need to rely on commercial services: become the perfect tool master and netlifize your app right at home!
How to Build CI/CD Pipelines for a Microservices Application
DevOps.js Conf 2021DevOps.js Conf 2021
33 min
How to Build CI/CD Pipelines for a Microservices Application
Top Content
Microservices present many advantages for running modern software, but they also bring new challenges for both Deployment and Operational tasks. This session will discuss advantages and challenges of microservices and review the best practices of developing a microservice-based architecture.We will discuss how container orchestration using Kubernetes or Red Hat OpenShift can help us and bring it all together with an example of Continuous Integration and Continuous Delivery (CI/CD) pipelines on top of OpenShift.
Automated Performance Regression Testing with Reassure
React Advanced Conference 2022React Advanced Conference 2022
16 min
Automated Performance Regression Testing with Reassure
As developers we love to dive into performance metrics, benchmarks, compare one solution to another. Whether we enjoy it or not, we’re often required to fix performance issues in our React and React Native apps. But this process is not sustainable and prone to regressions, especially as the app and team grow. What’s worse, those issues are often discovered by your users, making their experience miserable. In my talk I’ll introduce you to Reassure—a performance regression testing library for React and React Native— which happens to be a missing piece in our automated testing and performance suites. Spotting problems before they hit production.
How to Get CI/CD Right in 2021: A Guide to CI and CD
DevOps.js Conf 2021DevOps.js Conf 2021
9 min
How to Get CI/CD Right in 2021: A Guide to CI and CD
Software delivery is a top priority for organizations that own software, yet it remains one of the most challenging problems enterprises face today. Continuous integration (CI) and continuous delivery (CD) are software practices that allow organizations and teams to deliver code to customers quickly, safely, and repeatedly. Whether it's to improve development, operations, or security, CI/CD pipelines give engineers and teams more time to work on things that matter and less time struggling with the risk, standards, and velocity of deployments. Join this session to learn about the components of CI/CD and how to build and scale pipelines for the future.

Workshops on related topic

Bring Code Quality and Security to your CI/CD pipeline
DevOps.js Conf 2022DevOps.js Conf 2022
76 min
Bring Code Quality and Security to your CI/CD pipeline
WorkshopFree
Elena Vilchik
Elena Vilchik
In this workshop we will go through all the aspects and stages when integrating your project into Code Quality and Security Ecosystem. We will take a simple web-application as a starting point and create a CI pipeline triggering code quality monitoring for it. We will do a full development cycle starting from coding in the IDE and opening a Pull Request and I will show you how you can control the quality at those stages. At the end of the workshop you will be ready to enable such integration for your own projects.
Powering your CI/CD with GitHub Actions
DevOps.js Conf 2022DevOps.js Conf 2022
155 min
Powering your CI/CD with GitHub Actions
Workshop
David Rubio Vidal
David Rubio Vidal
You will get knowledge about GitHub Actions concepts, like:- The concept of repository secrets.- How to group steps in jobs with a given purpose.- Jobs dependencies and order of execution: running jobs in sequence and in parallel, and the concept of matrix.- How to split logic of Git events into different workflow files (on branch push, on master/main push, on tag, on deploy).- To respect the concept of DRY (Don't Repeat Yourself), we will also explore the use of common actions, both within the same repo and from an external repo.
Debugging JavaScript Apps in CI/CD
DevOps.js Conf 2022DevOps.js Conf 2022
124 min
Debugging JavaScript Apps in CI/CD
Workshop
Cecelia Martinez
Cecelia Martinez
- Causes of failed builds in CI/CD pipelines- Approaches to debugging (reviewing logs, accessing environments, reproducing issues)- Debugging application-related causes (failing tests, failed application builds)- Debugging pipeline-related causes (pipeline setup, environment issues, container issues)
CI/CD 101 with CircleCI
DevOps.js Conf 2021DevOps.js Conf 2021
149 min
CI/CD 101 with CircleCI
Workshop
Angel Rivera
Zan Markan
2 authors
Continuous Integration and Continuous Delivery/Deployment (CI/CD) concepts are increasingly adopted many technology organizations and teams. CI/CD enables teams to establish processes that increase velocity, collaboration and quality of their codebase. CI/CD enables developer & operations teams to break down unnecessary silos and gain a deeper knowledge of their respective arenas.
In this workshop the participants will be introduced to the basic fundamentals of Continuous Integration and Continuous Delivery/Deployment. Participants will learn the core principles of CI/CD and have the opportunity to reinforce what they’ve learned in a hands on workshop featuring the CircleCI platform. The workshop will demonstrate CI/CD build configuration, code commits, commit builds, code testing and packaging. The participants will leave with a hands-on experience and understanding of what it takes to CI/CD.
Table of contents- Introduction to the topic of CI/CD, and motivation for it- How different kinds of JavaScript projects are built and deployed (from static sites to APIs)- Overview of common manual steps & how we might automate them- Implementing a CI/CD pipeline from 'scratch'- Overview of CircleCI orbs- Testing across multiple versions of Node- Debugging builds with SSH- Caching dependencies- Security / vuln scanning- Deploying to various outlets
Prerequisites- Code and git installed- GitHub account
github.com/CircleCI-Public/cicd-workshop-js