Atomic Deployment for JS Hipsters

Rate this content

Deploying an app is all but an easy process. You will encounter a lot of glitches and pain points to solve to have it working properly. The worst is: that now that you can deploy your app in production, how can't you also deploy all branches in the project to get access to live previews? And be able to do a fast-revert on-demand?

Fortunately, the classic DevOps toolkit has all you need to achieve it without compromising your mental health. By expertly mixing Git, Unix tools, and API calls, and orchestrating all of them with JavaScript, you'll master the secret of safe atomic deployments.

No more need to rely on commercial services: become the perfect tool master and netlifize your app right at home!

25 min
15 Feb, 2024


Sign in or register to post your comment.

Video Summary and Transcription

This Talk discusses atomic deployment for JavaScript and TypeScript, focusing on automated deployment processes, Git hooks, and using hard links to copy changes. The speaker demonstrates setting up a bare repository, configuring deployment variables, and using the post-receive hook to push changes to production. They also cover environment setup, branch configuration, and the build process. The Talk concludes with tips on real use cases, webhooks, and wrapping the deployment process.

Available in Español

1. Introduction to Atomic Deployment

Short description:

We're going to talk about atomic deployment for JavaScript and TypeScript Hipsters. In an atomic deployment, you take two versions of an existing application and deploy the changes between them. It's self-contained and eliminates the need to worry about dependencies and building. The goal is to have a functional, workable, and compatible solution with a simple developer experience.

Hi, folks. Welcome to this talk for us, DevOps JS Conference. We're going to talk about atomic deployment for JavaScript and TypeScript Hipsters.

Right before starting, a little bit of information about what atomic deployment and immutable deployment are. In an atomic deployment, you take two versions of an existing application and you're just deploying the changes between these two versions, but everything is self-contained. So you can take a whole archive of version at any point of time and everything is self-contained and you don't have to worry about anything about that. This is what makes Netlify so successful in the past because suddenly you just had to push a branch on your version server like GitHub or whatever, and it was automatically deployed on your production server in a staging URL like mybranch slash

So it's really powerful because you just have to push some changes in any kind of branch, you automatically have a lot of previews directly accessible. And each time you update your branch like updating your PR, the preview updates itself as the same way. So it's definitely useful, especially when deployment is a painful process when you have to worry about dependencies and building and so on. So what is the hipster ways to do these kind of atomic or immutable deployments?

My idea is that I want to keep it working. I want to stick to this principles of atomic and immutable deployments. I want it to be functional, workable, and compatible with any hosting provider, whichever it was. And I want to stick to the simple developer experience. Like, I just want to push my branch and my preview is automatically deployed or updated, and that's all. And I don't care about how to use it and how to work with it and so on. So what are the expected features on these ones specifically?

So I want it to be configurable because I don't want to revamp the whole process for each project. I want to have some things that is easily deployable to any kind of repository and where I just have to tweak a few variables to configure it for this very specific project at a time. I want it to be band-lasting compatible, so working on any kind of hosting provider in the wild. I want some kind of micro-CI integrated like deployment dependencies and building and so on. Everything handled by the solution by itself. So I don't have to worry about anything about that. But without the tests you eat and so on, it doesn't care. I want no too minimal downtime on deployment to keep it really working. So I don't want to break something on the production just because I pushed a brand that suddenly didn't build or something else. So I want some kind of safeguards at some points to be sure that nothing can be down at some point. And I want it to be auto-publishable. So I just want to push it in a branch and boom, my branches, my preview associated to my branch is automatically updated. So I'm Matt, folks. I'm a developer experience engineer at Always Data, which is a hosting provider, an independent one.

2. Setting Up Automated Deployment

Short description:

I will show you the foundation of our automated deployment process with atomic and immutable deployment features. We will work on the server, have the web hosts and repositories in the same place, and rely on SSH remote access. I'll initialize a bare repository and link it to a new application.

So my responsibility at Always Data is working on the developer experience, most specifically on the CLI and the tools associated with the platform itself. What I will show you is the foundation of our automated deployment process with atomic and immutable deployment features. It's currently in development in CLI, but it will be available soon. I'm excited to share it with you because it's the result of my search. Let's dive into the code to see it in action.

For this talk, we will work on the server, whether it's a container, VPS, or bare metal. The idea is to have the web hosts and repositories responsible for atomic deployment in the same place. We will rely heavily on SSH remote access. Webhooks are not part of this talk, but I'll briefly mention them in the conclusion.

To get started, I'll SSH directly onto the server in my container with my hosting account. I'll ensure that I'm in the /var folder, where the www folder is located. This is the document root for our web server engine. I'll initialize a bare repository in /var/repo. A bare repository contains only the Git elements and not the source code itself. Back on my local computer, I'll create a new application and link it to the repository in the container using SSH.

3. Configuring Deployment

Short description:

The idea is that you rely on Git basics. I will SSH directly onto the server in my container and initialize a bare repository in the same folder. Then I will create a new application and link it to the repository. I will configure the project using git config for specific variables, such as the dist folder for deployment.

The idea is that you rely on Git basics. So I will SSH directly onto the server in my container with my hosting account, and I will make sure that I'm in the slash var folder, because in slash var there's a www folder. This is the document route for our web server engine, like Apigee or Nginx or Traphic or whatever you want, I don't care.

I will initialize a repository, a bare repository in the very same folder at slash var slash repo. A bare repository is give or take the same thing that you have in the .git folder in your working tree, but you just have the different ashes and elements dedicated to git and not the overall content of your source trees, because I don't care on the server to give access of the codebase at any time.

So if I check, I still have in my slash var folder, two subfolders, the repository in the bare format, and the www1 ready to handle our web apps. So get back locally on my computer. I will cd in my project folder wherever you want to store your project. It's totally up to you. I will create a new application, for example, using Vite with templates, so I will create an application in my app. So it's k-folded in my app, and now I can directly enter into my application.

Then I will link the previously created repo inside my container to this app project locally. So I will git init the application, git add everything that is available in it, and making the first initial commit. So everything is in the root commit in the first commit at the initial commit, first one, initializing it.

So I will add a remote deploy, which is pointing through SSH to the container and the var repo folder we created before. Why deploy and not origin? Because we will probably want to keep origin for our versioning server, like github or gitlab or whatever you want. And you can have as much remote as you wish in git, so why not name this one deploy, because it's ready to handle the deployment process and keep origin for the origin one, the upstream regular one. So I can git push deploy main. So I push on deploy the main branch, and I've got a feedback from git saying, okay, on containers, a var repo, I created a new branch main pointing to the main branch. Success!

Okay, so now let's configure it. We want the project to be configurable, so let's go back through SSH to the var repo. So we are still in our bare repository, and if I check the git config locals, I can see that I have a very few configuration keys, which is interesting, because I don't want to rely on the new YAML configuration file in my source tree. I want to have a few configurable variables that I want to tweak just for this very specific repository dedicated to deployment.

So why not rely on git config tool to do so? So the format is always the same, rearm.keyed.value. So we can create a git config local, because we want to keep it local., because in this project, it's a vid project, so vid will build a dist folder, and this is this dist folder that I want to deploy on my web apps for my web host and so on. I don't want to deploy the whole source tree, just the dist folder. So I specified that for this very specific project, and I used git config to do so, there is a dist build, and I can use so. So I recommend to use a dist or whatever. Don't use anything that git already relies on, like core and so on, because there is a risk of overwriting existing keys, but you can definitely rely to store your very own configuration.

4. Git Hooks and Post Receive Hook

Short description:

Git hooks allow you to react to events in your repository's lifecycle. The post receive hook is particularly interesting for pushing to production. It reacts to git push events, executes on the remote repository, and handles branches one by one. We can write it in Bash, automating terminal commands.

It's open, it's okay. So if we check, it's okay. It's already stored. So let's hook it. So everything right now will occur in the ssh.var.repo folder.

A few words on git hooks. Git hooks are a mechanism that allow you to react to some events occurring in the lifecycle of your git repository. So in the whole list, which is very long, of different events available, there is a few ones that are interesting, and one specifically is really interesting. It's a post receive hook, which is often dedicated to push the cart to production. Interesting. This is exactly what we are trying to do. So let's rely on the post receive hook. Let's have a quick look at it.

It's a hook that is reacting to git push events. When any updates, branches are coming, it executes on the remote repository. Excellent. And once after all the refs have been updated, which is interesting, I could push whole branches, my repository will handle them, and will react to all branches one by one. So I could definitely update and deploy each branch one by one by one every time I receive a git push event if it contains different branches. But if there is only one, it's okay. So it reads lines from stand-in, and it's an automated process. So I will write it in Bash. You could rely on Python or Deno or whatever you want to do so. But it's just automating a bunch of different terminal commands. So Bash is a great tool to do so. So we check we are still in the repo. We create a look at the post receive file and make it executable. So let's initialize it. It's a Bash content. We have a while loop to read the stand-in for each line. And in each line, we've got three kinds of information, one line, one branch.

5. Environment Setup and Branch Configuration

Short description:

We retrieve the path to the git pair folder on the server. If the deploy configuration key is not defined, we fall back to a folder in the VVV directory. For each branch, we set up the branch by removing the ref's heads prefix, using a high-cost commit, and creating a temp dir. We also create a link dir to the deploy dir ref name for better readability.

And we've got three information, the old revision commit, the new revision commit, and the reference, which is obviously the name of the branch. So we can echo in, working on the branch, blah blah blah.

Okay. So we have to set up the environment. So we have to retrieve the path to the git pair folder on the server itself. I could hard-code it, but I want it to be agnostic, so I have to detect it. It's a bit tricky. I don't have deep diving into it, but you've got it. We've got access to the git folder, and we pushed inside the git folder like a CD. So we enter the git folder to execute the rest of the commands.

We check if we have this deploy configuration key and we didn't define this one. So we can fall back to a folder at the same level of the git folder in the VVV directory, which is exactly what we have, slash virus slash VVV for the deployment. So we've got a deployment here. We check that the folder is already ready to handle the rest, and we go where we're ready to continue. So we can for each branch know, for each one, we can set up the branch. We are making some tricks to improve the readability. So we have a ref name, which is removing the ref's heads prefix. We use just a high-cost commit, and we make a temp dir to host our source tree for this very specific branch. So we're also making a link dir to the deploy dir ref name, which is a reman readable pass, because I don't want to configure all my web host to slash virus slash VVV slash a commit ID, but I want them to be pointing to var VVV, my branch name.

6. Working Tree and Build Process

Short description:

We move a link from the ref name to the current commit for the branch. Then, we create a working tree in the temp folder and check it out. Next, we handle the build process by checking the configuration, package design, package lock, and running the build script. Finally, we explain the use of Unix hard links to copy only changed files for better performance and maintenance.

So I can just move a link, which is just a pointer from the ref name to the current commit dedicated to this branch. So it will be better. So we use a link to do so, and we prepare the link name.

Okay, so now we have a working tree in the temp folder. So check out the work tree. We use the work tree command, which is like a checkout, but in a more powerful way. So we add a new work tree to this temp folder based on this current branch. And we push into it to move into this working tree.

We have to handle the build process, so we check if we have a build dir configured. This is the case. We defined it before, so we have it. Otherwise, it's just a current folder. We test if we have a package design available. If so, we test if we have a package lock. If so, we make a clean install, because we are clean people, otherwise. Let's do an install, but remember to make your log files, and then we can run the build script if the script is present. If not, it's okay. It's perfectly fine. If the build fails, the script will exit, and we will go to the next branch. And everything is protected like that. So go next.

A quick word on Unix hard links. The idea in the atomic process, if you remember correctly, is that I just want to copy the files that have changed from my previous version to this one. So I don't want to copy everything and duplicate the content. So my idea is to say, okay, let's check if we have a previous version of this branch. So we want to check where is the forking point from the main branch or from the originating branch, and we want to check if there is a commit dedicated to that. If so, if the file hasn't changed between this originate branch and this new one, just copy this one. Just hard link to this version. Otherwise, copy a new version. It will be more performance, faster, and definitely easier to maintain.

7. Deployment and Hard Links

Short description:

We use hard links to copy changes and make each version self-contained. To find the forking point of the current branch, we need to get the revision list and check if each commit is a valid ancestor. If it is, we can use it as a base for deployment. If there is a common ancestor, we add options to air sync to use the base folder and make a hard link if the file hasn't changed. We deploy using air sync, remove the working folder, and create links between reference and revision names.

And making just you to copy the changes, but each version will be self-contained. This is why we don't rely on symbolic links, but we prefer hard links to do so. So it means that we have to find a git command ancestor. So let's say that we're in the fixd branch. If we are there, we can guess that the forking point is a C1 commit. But if we ask git, hey, what is the forking point for my current branch, git don't write anything because git don't really store a tree of the different branches, but just a list of commits. So you have to rely on yourself to do so.

So you have to get the revision list from the tip of your branch to the main branch and the initial commit. Then you have to check for each one of this commit if it's a valid ancestor. So if it's a forking point from a previous branch, and if so, then you could consider that this commit, if it's still in a branch, could be used as a base for the deployment. So how to do so in code. We've got the rev list, thanks to the revlist command. And for each revision, so each commit in this list, we go back in history. And for each of them, we check if it's a possible ancestor for my branch for this specific revision.

If it's the case, so if it's the forking point, then let's check that it's still in a branch, and that this branch were deployed before. So is this revision still available? If so, then we could consider that we can, like we forked the code, we can fork the deployment and use this folder as a base folder for the revision. Otherwise, just insert the rev, and we will just deploy from scratch, and it's okay. It's perfectly legit to do so. So if we've got a common ancestor, then we just add a few options to air sync, which will be our deploy tool to say, okay, for this specific copy task, use this folder as a base.

And if the file didn't change between this folder and the new one, just make a hard link between the two. So let's go to deploy. We will rely on air sync, as you expected. So we will deploy from the working build directory inside it. So my disk directory inside my work tree, and we will deploy it to my deploy directory. The revision ID commit. We rely on checksum to do so with the air sync option activated. So hard links activated, if it's reliable in this context, and so on. After that, we can just pop from the working folder, remove it from the git repository because we don't need it anymore, and we are clean people, so we are cleaning behind us. And then we can publish it, which means create the links between the reference name and the revision name. So moving the pointer of the symbol in the branch name to the correct commit.

8. Publishing and Application Update

Short description:

To publish the application, ensure everything is okay, create the link, restart the server, and check the new commit. In less than 60 lines of bash code, achieve the same success as Netlify.

Ensure everything is okay, creating the link. Then after that, we restart the server. So in this demo, I fixed the restarting. So I just killed the existing server previously launched, and I randomized the port and launched an HTTP server pointing to this new production link that I created before, and that is pointing to my new commit. And ta-da, everything is okay. So this is for the publishing.

So let me show you how it works. Go back to my local folder. I update my application. I check that I'm still in my application. I check out a new branch, name fit slash update name. Switching to this branch, editing the application, committing the content with a fancy commit message update name, and then I can ship it. So I git push deploy fit update name. So I push to deploy my fit update name branch. And it says that, okay, I deploy branch to VAR vvv. We're working on the branch fit update name.

I'm preparing the work tree. So we are moving to the work tree in a detached state. Here, we are installing the npm packages. So we are building things to npm run build things to VIT. So we are producing the dist folder. We found the deploy base commit because the branch were forked from a demand branch, and we are deploying this dist folder to VAR vvv, my reference commit name. Then we can make the link to from the fit update name to the final commit id. And I restart the application. And if I check, I can assure you that it works. And in less than 60 lines of bash code. For the same thing, as successful as Netlify. This is just insane.

9. Real Use Cases, Webhooks, and Wrapping Tips

Short description:

Real use cases include creating/updating websites, performing DB actions, cache invalidation, and restarting the website. Webhooks can automate the pushing and retrieving process. Tips for wrapping: keep it simple, use Git config, abstract web hosting changes in API calls, and leverage Deno or Node.js as a build system. Thanks for having me today!

So what are the real use cases? Because I have said I faked the restarting process. So you will probably have some kind of actions like creating or updating the website with the correct URL like my branch name dot my app dot my domain dot com. Probably performing some DB actions like migration and so on, invalidation of cache, restarting the website, whatever you want. So you have to do so, wrap them in API calls, replacing the fake content of restarting the HTTP server. But if it's just static files like we have here, it's just creating a new VOS in your web engine and it's okay.

What about webhooks? As I said before, you can use them mainly just to automate the pushing from the origin remote to the deploy remote. So each time you push to GitHub or GitLab or whatever, your upstream server, thanks to webhooks, you can automate the retrieving process in the deploy server branch. I wrote a blog post about it if you want. So you can use them for that and it's perfectly okay, it's perfectly fine. It's just reducing the friction in terms of developer experience.

Some few tips for wrapping. Keep it simple and stupid. I make it really agnostic. I don't want to make it really clever or so on. Once again, it's just less than 60 lines of bash code. So make it really case and it will be really easier to maintain and to adapt to a lot of different applications. Use Git config a lot to store your variable and configuration process tweaking and so on. Abstract any changes to your web hosting configuration in API calls using CLI, using API, using whatever you want. And you can for sure rely on Deno or Node.js as a build system thanks to the package json file. You can create a lot of scripts to run to make sure the dependencies are installed, run pre-build process, build process, post-build process, even on complex applications written in Python, Ruby, Rust, whatever you want. But you can definitely use Node.js as a CI engine for the build process.

Thanks again. Thanks for having me today and I hope you enjoy the rest of the conference. Bye-bye.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

React Advanced Conference 2021React Advanced Conference 2021
19 min
Automating All the Code & Testing Things with GitHub Actions
Top Content
Code tasks like linting and testing are critical pieces of a developer’s workflow that help keep us sane like preventing syntax or style issues and hardening our core business logic. We’ll talk about how we can use GitHub Actions to automate these tasks and help keep our projects running smoothly.
DevOps.js Conf 2022DevOps.js Conf 2022
33 min
Fine-tuning DevOps for People over Perfection
Demand for DevOps has increased in recent years as more organizations adopt cloud native technologies. Complexity has also increased and a "zero to hero" mentality leaves many people chasing perfection and FOMO. This session focusses instead on why maybe we shouldn't adopt a technology practice and how sometimes teams can achieve the same results prioritizing people over ops automation & controls. Let's look at amounts of and fine-tuning everything as code, pull requests, DevSecOps, Monitoring and more to prioritize developer well-being over optimization perfection. It can be a valid decision to deploy less and sleep better. And finally we'll examine how manual practice and discipline can be the key to superb products and experiences.
DevOps.js Conf 2022DevOps.js Conf 2022
27 min
Why is CI so Damn Slow?
We've all asked ourselves this while waiting an eternity for our CI job to finish. Slow CI not only wrecks developer productivity breaking our focus, it costs money in cloud computing fees, and wastes enormous amounts of electricity. Let’s take a dive into why this is the case and how we can solve it with better, faster tools.
DevOps.js Conf 2022DevOps.js Conf 2022
31 min
The Zen of Yarn
In the past years Yarn took a spot as one of the most common tools used to develop JavaScript projects, in no small part thanks to an opinionated set of guiding principles. But what are they? How do they apply to Yarn in practice? And just as important: how do they benefit you and your projects?
In this talk we won't dive into benchmarks or feature sets: instead, you'll learn how we approach Yarn’s development, how we explore new paths, how we keep our codebase healthy, and generally why we think Yarn will remain firmly set in our ecosystem for the years to come.

Workshops on related topic

DevOps.js Conf 2022DevOps.js Conf 2022
152 min
MERN Stack Application Deployment in Kubernetes
Deploying and managing JavaScript applications in Kubernetes can get tricky. Especially when a database also has to be part of the deployment. MongoDB Atlas has made developers' lives much easier, however, how do you take a SaaS product and integrate it with your existing Kubernetes cluster? This is where the MongoDB Atlas Operator comes into play. In this workshop, the attendees will learn about how to create a MERN (MongoDB, Express, React, Node.js) application locally, and how to deploy everything into a Kubernetes cluster with the Atlas Operator.
React Summit 2023React Summit 2023
88 min
Deploying React Native Apps in the Cloud
Deploying React Native apps manually on a local machine can be complex. The differences between Android and iOS require developers to use specific tools and processes for each platform, including hardware requirements for iOS. Manual deployments also make it difficult to manage signing credentials, environment configurations, track releases, and to collaborate as a team.
Appflow is the cloud mobile DevOps platform built by Ionic. Using a service like Appflow to build React Native apps not only provides access to powerful computing resources, it can simplify the deployment process by providing a centralized environment for managing and distributing your app to multiple platforms. This can save time and resources, enable collaboration, as well as improve the overall reliability and scalability of an app.
In this workshop, you’ll deploy a React Native application for delivery to Android and iOS test devices using Appflow. You’ll also learn the steps for publishing to Google Play and Apple App Stores. No previous experience with deploying native applications is required, and you’ll come away with a deeper understanding of the mobile deployment process and best practices for how to use a cloud mobile DevOps platform to ship quickly at scale.
TestJS Summit 2021TestJS Summit 2021
85 min
Automated accessibility testing with jest-axe and Lighthouse CI
Do your automated tests include a11y checks? This workshop will cover how to get started with jest-axe to detect code-based accessibility violations, and Lighthouse CI to validate the accessibility of fully rendered pages. No amount of automated tests can replace manual accessibility testing, but these checks will make sure that your manual testers aren't doing more work than they need to.
DevOps.js Conf 2022DevOps.js Conf 2022
13 min
Azure Static Web Apps (SWA) with Azure DevOps
Azure Static Web Apps were launched earlier in 2021, and out of the box, they could integrate your existing repository and deploy your Static Web App from Azure DevOps. This workshop demonstrates how to publish an Azure Static Web App with Azure DevOps.
DevOps.js Conf 2022DevOps.js Conf 2022
163 min
How to develop, build, and deploy Node.js microservices with Pulumi and Azure DevOps
The workshop gives a practical perspective of key principles needed to develop, build, and maintain a set of microservices in the Node.js stack. It covers specifics of creating isolated TypeScript services using the monorepo approach with lerna and yarn workspaces. The workshop includes an overview and a live exercise to create cloud environment with Pulumi framework and Azure services. The sessions fits the best developers who want to learn and practice build and deploy techniques using Azure stack and Pulumi for Node.js.