Why is CI so Damn Slow?

Rate this content
Bookmark

We've all asked ourselves this while waiting an eternity for our CI job to finish. Slow CI not only wrecks developer productivity breaking our focus, it costs money in cloud computing fees, and wastes enormous amounts of electricity. Let’s take a dive into why this is the case and how we can solve it with better, faster tools.

Nicholas Yang
Nicholas Yang
27 min
24 Mar, 2022

Comments

Sign in or register to post your comment.

Video Summary and Transcription

Slow CI has a negative impact on productivity and finances. Debugging CI workflows and tool slowness is even worse. Dependencies impact CI and waiting for NPM or YARN is frustrating. The ideal CI job involves native programs for static jobs and lightweight environments for dynamic jobs. Improving formatter performance and linting is a priority. Performance optimization and fast tools are essential for CI and developers using slower hardware.

Available in Español

1. The Impact of Slow CI on Developer Productivity

Short description:

Hi everybody. My name is Nicholas. I'd like to talk to you about why your CI is so damn swell. Waiting for CI has a direct dire effect on productivity and finances. When encountering a time sync like a slow CI, slacking off or starting another task are not effective solutions. Task switching in software development projects leads to less task completion and disrupts flow. Slow CI interrupts the workflow and decreases efficacy and efficiency. Forgetting about a CI job can cause significant delays in merging PRs.

Hi everybody. My name is Nicholas. I'm a software developer at Rome Tools, and I'd like to talk to you all about why your CI is so damn swell.

We've all been there. You've pushed your latest code to GitHub, your CI service is spitting up, and it's taking forever. You wait and wait and wait, only to get your result back 5, 10, even 20 minutes later. It's annoying, it's disruptive, it's a waste of your damn time. Normally, we just accept this as an eternal truth. CI is swell. I don't think that's okay.

Having to wait for CI has a direct dire effect, both on your productivity and on your finances. Let's start with a resource that matters to all of us. Developer time. I don't know about you all, but when I encounter a time sync such as a slow CI, I do one of two things. I slack off, or I start another task. Slacking off is clearly a net negative. Sure, we all deserve breaks, but not every single time we push up some code. Our CI shouldn't determine when we work or not. But Nick, you could just start another task. And yeah, starting another task seems tempting. We're all capable multitaskers. Turns out no, we are not.

In the paper, Task Interruption in Software Development Projects, the authors measured the effect of task switching on productivity. It wasn't good. In particular, they noted that self-interruptions, basically when you purposely switch between tasks, they tend to be more disruptive than external interruptions and lead to less task completion. With slow CI, this flow gets constantly interrupted. You end up switching between the CI jobs and your new task, losing your working context and therefore your efficacy and efficiency. Or worse, you get focused on your new task, forget about your CI job, and only remember a few hours later that your job is done. You ping your reviewer just to realize that they've gone home. Suddenly, a quick PR takes two days or more to merge.

2. Debugging CI Workflows and Tool Slowness

Short description:

Even worse is when you have to debug a CI workflow. A 15-minute development cycle is not acceptable. Slow CI means more compute time, which means more money. Let's figure out why CI is slow. There are two types of CI jobs, static and dynamic jobs. If your CI job is slow, it's likely because your tools are slow. These tools are also slow in their installation time. Dependencies are a necessary part of modern software development.

Even worse is when you have to debug a CI workflow. Whenever I have to debug one, it's so awful. I end up tweaking a setting, waiting for the job to complete, getting distracted, and only seeing the results 15 minutes later. A 15-minute development cycle is not acceptable in this day and age. But it's not just developer time. Slow CI means more compute time, which, as anyone who stared in shock at their AWS bill knows, means more money.

One such example is the free desktop GitLab instance, which hosts a bunch of free software projects, such as Mesa, the Linux kernel drivers, and many others. They experienced a massive period of growth in the late 2019 to early 2020 era. However, their expenses ballooned accordingly, first at $75,000 in 2019. And then they were projected to hit $90,000 in 2020. They managed to cut costs before they ran out of money, but still, for an open source project, that's a massive amount to be spending on CI. Let's do better. Let's figure out why CI is slow.

To do so, we should look at what a CI job does. At its core, there are two types of CI jobs, static and dynamic jobs. Static jobs apply developer tools, such as a Linter, Bundler, Formatter, etc. to your code without executing the code. Dynamic jobs may also apply tools, but they have to run your code. We're going to focus on static jobs, but many of these lessons apply to dynamic ones too. Regardless, if your CI job is slow, it's likely because your tools are slow. But what do I mean exactly by slow? Of course, I mean the tools are slow in their runtime. However, for CI, there's a second type of slowness. These tools are also slow in their installation time. Let's take ESLint. It's a great tool. However, like many tools in the JavaScript ecosystem, it has a lot of dependencies. This isn't a dependency shaming talk. I use dependencies. You use dependencies. They're a necessary part of modern software development.

3. Impact of Dependencies on CI

Short description:

Dependencies impact CI. Waiting for NPM or YARN is frustrating. Node is a massive runtime. Most tools are written in JavaScript, which is not ideal for developer tooling. We use Rust, Go, and Zig for better performance. Rust compiles to a static binary, eliminating the need for dependencies. JavaScript tooling involves redundant work due to the fragmented ecosystem and the need to parse syntax trees.

But dependencies do have an impact on your CI. Namely, you have to install them. And no one likes waiting for NPM or YARN to do its thing. And with JavaScript, there's always the biggest dependency of all. Node. You have to download this massive runtime just to run your other dependencies. How many times does a CI service have to spin up a Docker container with Node just to run a tool like ESLint or webpack?

And back to running time. Why are these tools slow to run? At Rome, we put a lot of thought into this question. First and most obviously, it's because most tools are written in JavaScript, and JavaScript is not the ideal language for developer tooling. It uses a lot of memory, it's hard to write safe concurrent code, and it requires a runtime. The solution is, well, don't write the tools in JavaScript. We use Rust. ESbuild uses Go. Bun uses Zig.

In our case, we found that Rust has given us significant benefits. We first wrote Rome in TypeScript. We had to squeeze out performance with carefully optimized JavaScript and our own low level primitives. In Rust, we got very good performance right out of the gate. Second, because Rust compiles to a static binary, we no longer have to install dependencies or a CI process. We can simply download a binary. And finally, there's an aspect of JavaScript tooling that we believe is underestimated. And that's the problem of redundant work. The JavaScript tooling ecosystem is extremely fragmented. We use different tools to lint, format, bundle, and so on. Each tool therefore needs to parse and represent the JavaScript as a syntax tree. If you remember your compilers, you'll recall that syntax trees take up a significant amount of memory. Ever wonder why IntelliJ is slow? It's storing all these syntax trees. Not to mention, parsing involves I.O. which always comes with its own overhead. Therefore, each tool needs to do its work.

4. The Ideal CI Job and Making Formatting Better

Short description:

Use three tools, three times the parsing time and three times the memory. Our ideal CI job looks like native programs for static jobs and more lightweight environments for dynamic jobs. Roam aims to be the one-stop shop for all of JavaScript. Linting, particularly the prettier rule, can be slow and hinder development flow. We are working to make formatting better.

Use three tools, three times the parsing time and three times the memory. With Roam, it handles linting, formatting, bundling, and so on. That means one parsing stage and one syntax tree. There's no wasted compute and no wasted memory.

With all this in mind, what does our ideal CI job look like? In the case of static jobs, these jobs should operate as native programs with no JavaScript runtime involved. There's really no reason to run Node or Docker for a linter or bundler. They can be used as a command line or as an API. In which case, the performance expectation should be on par with any other REST API. You call it, get a result in under a second. Of course, for dynamic jobs, where we need to execute the code, we can still spin up a runtime. But perhaps we can stick to more lightweight environments. Something like an AWS Lambda or CloudFlare worker could easily suffice.

So where does Roam come into this? For now, we're focused on delivering fast, native JavaScript tools. Our long-term plan is to be the one-stop shop for all of JavaScript. Hopefully in the future, these tools would be the replacement for static CI jobs. But where do we start? Of course, we couldn't tackle every tool simultaneously. We decided to do some research, look at Open Source repositories, and see what was slow in their tooling setup. We found that linting was generally pretty slow. But in particular, one rule stood out – prettier. If you're not familiar, there's an ES lint rule that checks if your code is formatted correctly by running prettier on your code and seeing if anything changes. In the case of Babel, a JavaScript compiler originally written by Roam's founder, Sebastian, the prettier ES lint rule takes over six seconds to complete. With Webpack, this rule takes five seconds. You may be saying at this point, six seconds? That's not that much. And I'd agree in abstract. But in reality, six seconds is the difference between running your linter as a local watcher and running it on CI. If it takes six seconds to get your lint results back, you're probably not going to have the linter integrated into your development flow. You're probably not going to want to call it through a REST API. Instead, you're going to rely on running it as a CI task. You're going to wait longer before seeing potential issues in your code. With this evidence in mind, we made it our mission to make formatting better.

5. Improving Formatter Performance and Winting

Short description:

I love prettier, but it can get slow in large files. We wanted a formatter with the same performance as Rust formatter but for JavaScript. Our formatter is error tolerant and will be released by March 28th. It performs significantly faster than prettier on various frameworks and large files. Our next goal is to improve the speed of winting and leverage CST-based parsing for better code preservation.

Don't get me wrong. I love prettier, but it can get remarkably slow in large files. In particular, we were inspired by Rust formatter, which is super-fast. We decided we wanted a formatter of the same level of performance, but for JavaScript. We also wanted a formatter that was integrated into your development environment.

One particular feature of developer tools that we love is called error tolerance. A tool is error tolerant if it works even when your code has errors, whether they're syntactic or semantic. We put a lot of work into making the formatter error tolerant. That way, it can still make your code look good no matter what.

We're super-excited to announce that our formatter will be released by March 28th. We'd love to give a sneak peek of the performance of our formatter. First up, we have Dojo, a front-end framework. Running the formatter on a bundled library, prettier takes 75 milliseconds, which is not bad, but roam is all the way down at 10 milliseconds. For Svelte, prettier is up to 750 milliseconds, almost a second, while roam is all the way down at 200 milliseconds. And finally, there's the behemoth that is TypeScript's checker.ts, clocking in at over 44,000 lines, or 2.64 megabytes. I don't know if you've ever opened this file in an editor, but I'll tell you now, it slows mine down to a crawl. Prettier takes over two seconds to format this file. We do it in 335 milliseconds.

But that's just the beginning. Our next goal is winting. As I mentioned before, winting is quite slow in JavaScript. ESLint can easily take 15, 30 seconds to run on a top computer. We want to be a tenth of that. Winting will also lash off the power of our Concrete Syntax Tree or CST-based parsing. Once again, if you're familiar with compilers, you may know that parsers generally produce what's called an Abstract Syntax Tree or AST. ASTs are great because they abstract away unnecessary details, such as whitespace or comments. However, it turns out that these supposedly unnecessary details are rather important for other developer tools. Specifically, it's really hard to do something like parse the code into an AST, modify the AST, then convert it back into code without completely rearranging the code, thereby creating a massive git diff. With a CST, everything is preserved. Comments, whitespace, everything.

QnA

CST-based Linting and Q&A

Short description:

That means converting a CST back into code with only a minimal diff is extremely easy, which in turn opens up a whole host of powerful code modification features. To recap, CI is slow because our tools are slow, both in their installation time and their running time. We need fast, native tools for JavaScript. Yeah, a really great talk and Nicholas, thank you for sharing that with us. Let's check out our Slido results. What is their slowest and most annoying tool? Let's start with some of the Q&A. All right, cool. So let's start with what about testing? Isn't there inherent performance issues with testing? Yes. So I would agree with that. Testing is arguably the thing that makes CI the slowest, and there are potential solutions in this area.

That means converting a CST back into code with only a minimal diff is extremely easy, which in turn opens up a whole host of powerful code modification features. We're really excited to demonstrate the possibilities of CST-based linting. Our linter will come out by the end of Q2, and we're so excited to demonstrate the possibilities of cross-language CST-based linting.

To recap, CI is slow because our tools are slow, both in their installation time and their running time. They are slow because they use slower languages that rely on runtimes and containers. To make CI faster, we need fast, native tools for JavaScript.

If these goals sound interesting, we're going to be hiring soon. Feel free to reach out at Nicholas at RoamTools.com or via Twitter. Thank you.

Yeah, a really great talk and Nicholas, thank you for sharing that with us. It was actually a pretty quick talk for me about CI being so slow, right? Yeah, I didn't want to waste your time. But it was a good one. But you've unpacked a lot there, so great talk. Let's check out our Slido results. I'm really intrigued to see what people are thinking. What is their slowest and most annoying tool? Let's check that out together. So Bundler, 50% say Bundler. OK, that's interesting. But in second place, we have IDE. So if you want to share in the chat which IDEs you're using, we'd love to hear which ones you're using and you find pretty slow. And unless that's like etcd there, I'd love to unpack also the etal or et cetera to know what's in third place. What's in third place, so tell us what other tools you think didn't make the list. We'd love to hear about those. And yeah, let's hear what people had to ask, Nicholas. Let's start with some of the Q&A.

All right, cool. So let's start with what about testing? Isn't there inherent performance issues with testing? Yes. So I would agree with that. I would say that testing is arguably the thing that makes CI the slowest, and there's inherent boundary to tests being just code that you have to run. But I think, you know, there are potential solutions in this area.".

Optimizing Runtime and Testing in CI

Short description:

You can optimize the runtime overhead, explore potential JavaScript optimizations, and precompile the JavaScript. Testing and benchmarking are heavy lifts in the process, but there are techniques to steal from large companies. For example, running relevant benchmarks and tests instead of the entire suite. Cleaner code in both tools and normal code can help bypass the slowness of CI. Performance has been neglected in the tooling world.

For instance, you can make the runtime overhead a lot smaller. You can use something like a work curve as optimized to have a fast speed up and fast start. You can also look into potential optimizations within JavaScript. Like maybe you can have a JavaScript run faster. Maybe you can figure out ways to precompile the JavaScript. I think, you know, we've been kind of treating it as this black box. But I think there are possibilities there that we can really use to make CI a little faster and a little more responsive.

Yeah, on the testing front, I agree. I think there's plenty that can be done there to optimize. So, in testing, I mean, what are the other heavy lifts in that process? Just wondering what else kind of, you know?

Yeah, so I would say testing. Benchmarking is another classic one because, again, with benchmarking, you know, you got to run the code and you got to keep it. You have to oftentimes run the code multiple times. You know, a lot of modern benchmarking frameworks do statistical analysis. And so they have to run it a statistically significant amount of times. And so, you know, if there isn't as great of a solution because, you know, it's, again, benchmarking. But I think, you know, there are some techniques that we can steal from large companies. For instance, I know, at Facebook, they only run relevant benchmarks and relevant tests for the commit. So they don't run the entire testing suite because obviously Facebook is running at Facebook's scale and you can't run every test there is. So I think there's really... I think we're really excited to see what other people tackle, how other people tackle this problem. And also we're really excited to show people our solutions to this problem.

Awesome. So Sissy Miller asks, coming from the PHP world, Pest PHP is rocking the testing world over there and it's making tests massively faster. But he liked to ask, or not, he or she, I don't know. Would you agree that some of the slowness of CI can be bypassed by cleaner code? I would. I mean, certainly, of course. I think, you know, there's both cleaner code in the tools and also cleaner code in your normal code. I mean, obviously if you're running the code, it's... You want your code to be fast and not slow. And I would say, from my perspective, as someone who writes tools, I think that that is something that has been neglected in the tooling world is performance.

Performance Optimization and Shout-outs

Short description:

I would like to give a shout-out to the authors of ESBuild and SWC for thinking about performance. Our benchmarks are significantly better now, around 20% to 30%. There's always more performance to squeeze out.

And, you know, I would like to give a shout-out to like a lot of people who are writing tools who think about performance really carefully, such as the authors of ESBuild, and the authors of SWC. Ourselves at Roam, we put a lot of effort into that. And in fact, I actually... The benchmarks I showed you guys in the talk are actually significantly out of date. We've been doing a lot of performance optimization since then, and I would say that our numbers are perhaps like maybe 20% or 30% better than that, which is really fantastic work. And I'm so pumped to show that to people. I think, you know- That's really cool. Yeah, there's always more that you can do. There's more performance you can squeeze out. So yeah.

Performance Improvements and Fast Tools

Short description:

At Roam, we have implemented performance improvements by optimizing the parsing process and minimizing redundant work in formatting. We aim to make parsing faster by reducing backtracking when the parser encounters a different syntactic construct. Additionally, we focus on ensuring the formatter performs only the necessary actions without unnecessary text rearrangement. Fast tools are essential not only for CI but also for developers using slower hardware. It is unfair to write code that only works on high-end machines.

Just, I guess, share a little bit. What kind of performance improvements have you implemented at Roam? Just off the top of your head. So for instance, a big thing that you always have to do is parse the code. And parsing is notoriously slow for a bunch of reasons. First, you have to read in the code from I.O. And second, you have to create this big data structure. So we've been doing a lot of work to basically make sure that when you parse, you're doing the minimal amount of work because oftentimes you might parse something and then realize that it's a different syntactic construct. So for instance, you may be part, your parser might be parsing some code and realize, oh, wait a minute, it's not a JSX expression, it's a TypeScript type assertion because it turns out that these two things look very, very similar. And we've been putting in a lot of work into that process where the parser realizes that it's something different than expected. It has to backtrack. We're trying to do that less and less so that the parsing is way, way faster.

We're also putting in a lot of work into formatting, into making sure that the formatter is doing just the bare minimum it needs to. And it really isn't doing redundant work in moving text around and rearranging the code.

Okay, I see that, folks, I'm just going to encourage folks to first, reminding you about the Slido, if you want to tell us what other tools you find annoying and slow, we'd love to hear from the 11% of you. And also which IDEs you're using. We'd love to hear that. And also if you answer in time, maybe Nicholas can give us thoughts on why he thinks those are specifically slow tools. But, I mean, I guess from my personal perspective, I'd like to say that I don't find my CI very slow. So why is this something that should matter? Why should I care about this? That's a really good point. And, you know, I'm happy that your CI isn't slow. I'm happy that, you know, you're not waiting like some other people are. I would say that I think Fast Tools are universally a good thing, even if it's not for your CI. I think, you know, it's really, again we have to remember that not everybody is running, you know, a super modern computer. Like, as I said in the talk, you know, we, like often as developers at, you know, companies we get, you know, modern hardware, like, you know, pretty often. But like, you know, there's plenty of people out there who are just starting out, or maybe at companies who don't have a budget where they're running on, you know, slower hardware and we have to accommodate them. I think it's really not fair to write code that only works on, you know, a brand new MacBook Pro. It makes sense. That's, you know, yeah. Yeah, I can appreciate that, obviously. So that was spoken from very deeply within Turbulent.

Slow Internet Connections and Linter Details

Short description:

Running code on slower internet connections is an important lesson for developers. Our Linter focuses on the slowest rules in ESLint to provide fast feedback. While there is currently no collaboration to share a single AST among popular tools, the TreeSitter Project is an inspiring example of a parser that can parse multiple languages. Sharing ASTs would require significant changes. Cece Miller asks about AI interpretation of code, such as GitHub Copilot.

That was one of those things, and I'm like, okay. Yeah, I mean, I've been guilty of that too. You know, I run the code and I think, oh, it's funny fast, but, you know, I think, I forget which company it is, but they had a habit of basically running their internet connection slower, like basically throttling it for the express purpose of accommodating people who are on, like, a 3G or even a 2G connection, because I think that's a really, you know, important lesson to give to developers.

Yeah, that's very true. Dainy asks, can you share some details about the Linter? How will it compare to ESLint in terms of rules, plugins, other factors? Yeah, so I'm really excited about this one because Linting is something that, you know, is and has been slow for a while. And obviously, ESLint is a fantastic project, and they have just a lot of rules, and I'm not going to come out and say that we're going to capture all these rules at once, because, you know, as everyone always says to me all the time, Rome was not built in a day. So I would say is that what we're focusing on is what are the rules that are the slowest in ESLint, what are the things that really make it painful, and then we're going to build a winter for those specific rules. And then, you know, the idea maybe is then you can run our winter, get fast feedback for certain things. And if you want a full, complete Linting setup, you can still run ESLint, and you can still get all of the feedback from ESLint, but you'll get the fast feedback first.

That's cool. So Nicholas, somebody's, J. Reid is saying, Nicholas mentioned that some of the various tools in JS ecosystem, JavaScript ecosystem, each have to build their own ASTs. Apologies, I don't know what ASTs stands for, to do their thing. Anyone know if there is any collaboration happening between some of the most popular tools with the goal of sharing a single AST? Just tell me what an AST is quickly. Abstract Syntax Tree. So it's the thing that parser outputs. Yeah, and as far as I know, I don't think there is. I mean I could be wrong, and I'm happy to be proven wrong in this, because I think that would be a great project. One thing that I think is really inspiring is the TreeSitter Project, which is a parser that basically can parse multiple different languages, pretty much any language you can think there's a TreeSitter parser for, including JavaScript. And that's a great instance of basically someone writing like one really great parser, which then a lot of people can use. I think I'm not really sure if there's any way to share the AST currently. I mean obviously there's some issues with if one tool changes the AST, then the other tools can't really use the same AST. So it's a little complicated. And I'm not certain that that would necessarily work without some serious changes. Awesome. Thank you. That's a... that was a very good answer. Cece Miller asks another question. Have you looked at AI interpretation of code such as GitHub go pilot, GitHub copilot? Perhaps this could step it into another level.

ML and Parser Error Recovery

Short description:

I've played around with Tab 9 and Copilot, and I find them to be really cool projects. While Roam is not currently heading in that direction, we are open to exploring ML plus code in the future. One area we're considering is using machine learning for parser error recovery, leveraging data sets of incorrect code to estimate the intended code and offer fixes or better semantic analysis. Regarding supporting custom linter rules, we're still figuring that out. We aim to provide a wide variety of rules, but we acknowledge the flourishing eswint community.

Personally, I've played around with them. I'm a huge fan of Tab 9 and Copilot as well. I think those really cool projects. Yeah, Tab 9 is a fantastic project. I wouldn't say that that's necessarily the direction we're taking at Roam right now. I think it's definitely something that we're open to exploring in the future. I mean, I think, you know, ML plus code is a really fascinating area that will definitely develop more in the coming years.

I do think one thing that we have looked into and I have been considering is perhaps using machine learning for parser error recovery. So as I noted, in the talk, one big thing about our parser is that it has very good error recovery. Basically, it can notice when there's an error and basically try to guess what you meant by it. And right now we do that with some heuristics, but I think in the future, it could be a really interesting opportunity for machine learning where you can basically use data sets of incorrect code to estimate what the intended code is and then offer a fix or just better semantic analysis.

That's a really good question. I'm gonna be honest, we're still trying to figure that one out ourselves. I think, you know, again, we have this idea of like, sort of like R12 being like the scout that kind of goes out ahead. And like, you know, if our rules are incorrect you should fix those first. And then if our rules are all, you know, valid and the code is correct for these roles then you can run eswint and get the full completion. I don't, you know, again, I think we're going to do our best to offer a, you know, wide variety of rules and we're gonna keep adding more but the reality is eswint, you know, has a wonderful job and is also a very flourishing community. And, you know, we have to acknowledge that.

Awesome. You're getting really great feedback here. Just gonna tell you that what the community thinks really good insights from your partner in the discussion room, in the build room, Ante Tomić. Really good insights. He's curious about Rome, wants to check it out and a lot of really great feedback about the talk. I don't see, now is the time folks, this is it. We're coming down to the wire. Just get your last questions in for Nicholas.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Levelling up Monorepos with npm Workspaces
DevOps.js Conf 2022DevOps.js Conf 2022
33 min
Levelling up Monorepos with npm Workspaces
Top Content
Learn more about how to leverage the default features of npm workspaces to help you manage your monorepo project while also checking out some of the new npm cli features.
Automating All the Code & Testing Things with GitHub Actions
React Advanced Conference 2021React Advanced Conference 2021
19 min
Automating All the Code & Testing Things with GitHub Actions
Top Content
Code tasks like linting and testing are critical pieces of a developer’s workflow that help keep us sane like preventing syntax or style issues and hardening our core business logic. We’ll talk about how we can use GitHub Actions to automate these tasks and help keep our projects running smoothly.
Fine-tuning DevOps for People over Perfection
DevOps.js Conf 2022DevOps.js Conf 2022
33 min
Fine-tuning DevOps for People over Perfection
Top Content
Demand for DevOps has increased in recent years as more organizations adopt cloud native technologies. Complexity has also increased and a "zero to hero" mentality leaves many people chasing perfection and FOMO. This session focusses instead on why maybe we shouldn't adopt a technology practice and how sometimes teams can achieve the same results prioritizing people over ops automation & controls. Let's look at amounts of and fine-tuning everything as code, pull requests, DevSecOps, Monitoring and more to prioritize developer well-being over optimization perfection. It can be a valid decision to deploy less and sleep better. And finally we'll examine how manual practice and discipline can be the key to superb products and experiences.
The Zen of Yarn
DevOps.js Conf 2022DevOps.js Conf 2022
31 min
The Zen of Yarn
In the past years Yarn took a spot as one of the most common tools used to develop JavaScript projects, in no small part thanks to an opinionated set of guiding principles. But what are they? How do they apply to Yarn in practice? And just as important: how do they benefit you and your projects?
In this talk we won't dive into benchmarks or feature sets: instead, you'll learn how we approach Yarn’s development, how we explore new paths, how we keep our codebase healthy, and generally why we think Yarn will remain firmly set in our ecosystem for the years to come.
End the Pain: Rethinking CI for Large Monorepos
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
End the Pain: Rethinking CI for Large Monorepos
Scaling large codebases, especially monorepos, can be a nightmare on Continuous Integration (CI) systems. The current landscape of CI tools leans towards being machine-oriented, low-level, and demanding in terms of maintenance. What's worse, they're often disassociated from the developer's actual needs and workflow.Why is CI a stumbling block? Because current CI systems are jacks-of-all-trades, with no specific understanding of your codebase. They can't take advantage of the context they operate in to offer optimizations.In this talk, we'll explore the future of CI, designed specifically for large codebases and monorepos. Imagine a CI system that understands the structure of your workspace, dynamically parallelizes tasks across machines using historical data, and does all of this with a minimal, high-level configuration. Let's rethink CI, making it smarter, more efficient, and aligned with developer needs.
Atomic Deployment for JS Hipsters
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
Atomic Deployment for JS Hipsters
Deploying an app is all but an easy process. You will encounter a lot of glitches and pain points to solve to have it working properly. The worst is: that now that you can deploy your app in production, how can't you also deploy all branches in the project to get access to live previews? And be able to do a fast-revert on-demand?Fortunately, the classic DevOps toolkit has all you need to achieve it without compromising your mental health. By expertly mixing Git, Unix tools, and API calls, and orchestrating all of them with JavaScript, you'll master the secret of safe atomic deployments.No more need to rely on commercial services: become the perfect tool master and netlifize your app right at home!

Workshops on related topic

Deploying React Native Apps in the Cloud
React Summit 2023React Summit 2023
88 min
Deploying React Native Apps in the Cloud
WorkshopFree
Cecelia Martinez
Cecelia Martinez
Deploying React Native apps manually on a local machine can be complex. The differences between Android and iOS require developers to use specific tools and processes for each platform, including hardware requirements for iOS. Manual deployments also make it difficult to manage signing credentials, environment configurations, track releases, and to collaborate as a team.
Appflow is the cloud mobile DevOps platform built by Ionic. Using a service like Appflow to build React Native apps not only provides access to powerful computing resources, it can simplify the deployment process by providing a centralized environment for managing and distributing your app to multiple platforms. This can save time and resources, enable collaboration, as well as improve the overall reliability and scalability of an app.
In this workshop, you’ll deploy a React Native application for delivery to Android and iOS test devices using Appflow. You’ll also learn the steps for publishing to Google Play and Apple App Stores. No previous experience with deploying native applications is required, and you’ll come away with a deeper understanding of the mobile deployment process and best practices for how to use a cloud mobile DevOps platform to ship quickly at scale.
MERN Stack Application Deployment in Kubernetes
DevOps.js Conf 2022DevOps.js Conf 2022
152 min
MERN Stack Application Deployment in Kubernetes
Workshop
Joel Lord
Joel Lord
Deploying and managing JavaScript applications in Kubernetes can get tricky. Especially when a database also has to be part of the deployment. MongoDB Atlas has made developers' lives much easier, however, how do you take a SaaS product and integrate it with your existing Kubernetes cluster? This is where the MongoDB Atlas Operator comes into play. In this workshop, the attendees will learn about how to create a MERN (MongoDB, Express, React, Node.js) application locally, and how to deploy everything into a Kubernetes cluster with the Atlas Operator.
Azure Static Web Apps (SWA) with Azure DevOps
DevOps.js Conf 2022DevOps.js Conf 2022
13 min
Azure Static Web Apps (SWA) with Azure DevOps
WorkshopFree
Juarez Barbosa Junior
Juarez Barbosa Junior
Azure Static Web Apps were launched earlier in 2021, and out of the box, they could integrate your existing repository and deploy your Static Web App from Azure DevOps. This workshop demonstrates how to publish an Azure Static Web App with Azure DevOps.
How to develop, build, and deploy Node.js microservices with Pulumi and Azure DevOps
DevOps.js Conf 2022DevOps.js Conf 2022
163 min
How to develop, build, and deploy Node.js microservices with Pulumi and Azure DevOps
Workshop
Alex Korzhikov
Andrew Reddikh
2 authors
The workshop gives a practical perspective of key principles needed to develop, build, and maintain a set of microservices in the Node.js stack. It covers specifics of creating isolated TypeScript services using the monorepo approach with lerna and yarn workspaces. The workshop includes an overview and a live exercise to create cloud environment with Pulumi framework and Azure services. The sessions fits the best developers who want to learn and practice build and deploy techniques using Azure stack and Pulumi for Node.js.