We've all asked ourselves this while waiting an eternity for our CI job to finish. Slow CI not only wrecks developer productivity breaking our focus, it costs money in cloud computing fees, and wastes enormous amounts of electricity. Let’s take a dive into why this is the case and how we can solve it with better, faster tools.
Why is CI so Damn Slow?
AI Generated Video Summary
Slow CI has a negative impact on productivity and finances. Debugging CI workflows and tool slowness is even worse. Dependencies impact CI and waiting for NPM or YARN is frustrating. The ideal CI job involves native programs for static jobs and lightweight environments for dynamic jobs. Improving formatter performance and linting is a priority. Performance optimization and fast tools are essential for CI and developers using slower hardware.
1. The Impact of Slow CI on Developer Productivity
Hi everybody. My name is Nicholas. I'd like to talk to you about why your CI is so damn swell. Waiting for CI has a direct dire effect on productivity and finances. When encountering a time sync like a slow CI, slacking off or starting another task are not effective solutions. Task switching in software development projects leads to less task completion and disrupts flow. Slow CI interrupts the workflow and decreases efficacy and efficiency. Forgetting about a CI job can cause significant delays in merging PRs.
Hi everybody. My name is Nicholas. I'm a software developer at Rome Tools, and I'd like to talk to you all about why your CI is so damn swell.
We've all been there. You've pushed your latest code to GitHub, your CI service is spitting up, and it's taking forever. You wait and wait and wait, only to get your result back 5, 10, even 20 minutes later. It's annoying, it's disruptive, it's a waste of your damn time. Normally, we just accept this as an eternal truth. CI is swell. I don't think that's okay.
Having to wait for CI has a direct dire effect, both on your productivity and on your finances. Let's start with a resource that matters to all of us. Developer time. I don't know about you all, but when I encounter a time sync such as a slow CI, I do one of two things. I slack off, or I start another task. Slacking off is clearly a net negative. Sure, we all deserve breaks, but not every single time we push up some code. Our CI shouldn't determine when we work or not. But Nick, you could just start another task. And yeah, starting another task seems tempting. We're all capable multitaskers. Turns out no, we are not.
In the paper, Task Interruption in Software Development Projects, the authors measured the effect of task switching on productivity. It wasn't good. In particular, they noted that self-interruptions, basically when you purposely switch between tasks, they tend to be more disruptive than external interruptions and lead to less task completion. With slow CI, this flow gets constantly interrupted. You end up switching between the CI jobs and your new task, losing your working context and therefore your efficacy and efficiency. Or worse, you get focused on your new task, forget about your CI job, and only remember a few hours later that your job is done. You ping your reviewer just to realize that they've gone home. Suddenly, a quick PR takes two days or more to merge.
2. Debugging CI Workflows and Tool Slowness
Even worse is when you have to debug a CI workflow. A 15-minute development cycle is not acceptable. Slow CI means more compute time, which means more money. Let's figure out why CI is slow. There are two types of CI jobs, static and dynamic jobs. If your CI job is slow, it's likely because your tools are slow. These tools are also slow in their installation time. Dependencies are a necessary part of modern software development.
Even worse is when you have to debug a CI workflow. Whenever I have to debug one, it's so awful. I end up tweaking a setting, waiting for the job to complete, getting distracted, and only seeing the results 15 minutes later. A 15-minute development cycle is not acceptable in this day and age. But it's not just developer time. Slow CI means more compute time, which, as anyone who stared in shock at their AWS bill knows, means more money.
One such example is the free desktop GitLab instance, which hosts a bunch of free software projects, such as Mesa, the Linux kernel drivers, and many others. They experienced a massive period of growth in the late 2019 to early 2020 era. However, their expenses ballooned accordingly, first at $75,000 in 2019. And then they were projected to hit $90,000 in 2020. They managed to cut costs before they ran out of money, but still, for an open source project, that's a massive amount to be spending on CI. Let's do better. Let's figure out why CI is slow.
3. Impact of Dependencies on CI
4. The Ideal CI Job and Making Formatting Better
Use three tools, three times the parsing time and three times the memory. With Roam, it handles linting, formatting, bundling, and so on. That means one parsing stage and one syntax tree. There's no wasted compute and no wasted memory.
5. Improving Formatter Performance and Winting
One particular feature of developer tools that we love is called error tolerance. A tool is error tolerant if it works even when your code has errors, whether they're syntactic or semantic. We put a lot of work into making the formatter error tolerant. That way, it can still make your code look good no matter what.
We're super-excited to announce that our formatter will be released by March 28th. We'd love to give a sneak peek of the performance of our formatter. First up, we have Dojo, a front-end framework. Running the formatter on a bundled library, prettier takes 75 milliseconds, which is not bad, but roam is all the way down at 10 milliseconds. For Svelte, prettier is up to 750 milliseconds, almost a second, while roam is all the way down at 200 milliseconds. And finally, there's the behemoth that is TypeScript's checker.ts, clocking in at over 44,000 lines, or 2.64 megabytes. I don't know if you've ever opened this file in an editor, but I'll tell you now, it slows mine down to a crawl. Prettier takes over two seconds to format this file. We do it in 335 milliseconds.
CST-based Linting and Q&A
That means converting a CST back into code with only a minimal diff is extremely easy, which in turn opens up a whole host of powerful code modification features. We're really excited to demonstrate the possibilities of CST-based linting. Our linter will come out by the end of Q2, and we're so excited to demonstrate the possibilities of cross-language CST-based linting.
If these goals sound interesting, we're going to be hiring soon. Feel free to reach out at Nicholas at RoamTools.com or via Twitter. Thank you.
Yeah, a really great talk and Nicholas, thank you for sharing that with us. It was actually a pretty quick talk for me about CI being so slow, right? Yeah, I didn't want to waste your time. But it was a good one. But you've unpacked a lot there, so great talk. Let's check out our Slido results. I'm really intrigued to see what people are thinking. What is their slowest and most annoying tool? Let's check that out together. So Bundler, 50% say Bundler. OK, that's interesting. But in second place, we have IDE. So if you want to share in the chat which IDEs you're using, we'd love to hear which ones you're using and you find pretty slow. And unless that's like etcd there, I'd love to unpack also the etal or et cetera to know what's in third place. What's in third place, so tell us what other tools you think didn't make the list. We'd love to hear about those. And yeah, let's hear what people had to ask, Nicholas. Let's start with some of the Q&A.
All right, cool. So let's start with what about testing? Isn't there inherent performance issues with testing? Yes. So I would agree with that. I would say that testing is arguably the thing that makes CI the slowest, and there's inherent boundary to tests being just code that you have to run. But I think, you know, there are potential solutions in this area.".
Optimizing Runtime and Testing in CI
Yeah, on the testing front, I agree. I think there's plenty that can be done there to optimize. So, in testing, I mean, what are the other heavy lifts in that process? Just wondering what else kind of, you know?
Yeah, so I would say testing. Benchmarking is another classic one because, again, with benchmarking, you know, you got to run the code and you got to keep it. You have to oftentimes run the code multiple times. You know, a lot of modern benchmarking frameworks do statistical analysis. And so they have to run it a statistically significant amount of times. And so, you know, if there isn't as great of a solution because, you know, it's, again, benchmarking. But I think, you know, there are some techniques that we can steal from large companies. For instance, I know, at Facebook, they only run relevant benchmarks and relevant tests for the commit. So they don't run the entire testing suite because obviously Facebook is running at Facebook's scale and you can't run every test there is. So I think there's really... I think we're really excited to see what other people tackle, how other people tackle this problem. And also we're really excited to show people our solutions to this problem.
Awesome. So Sissy Miller asks, coming from the PHP world, Pest PHP is rocking the testing world over there and it's making tests massively faster. But he liked to ask, or not, he or she, I don't know. Would you agree that some of the slowness of CI can be bypassed by cleaner code? I would. I mean, certainly, of course. I think, you know, there's both cleaner code in the tools and also cleaner code in your normal code. I mean, obviously if you're running the code, it's... You want your code to be fast and not slow. And I would say, from my perspective, as someone who writes tools, I think that that is something that has been neglected in the tooling world is performance.
Performance Optimization and Shout-outs
I would like to give a shout-out to the authors of ESBuild and SWC for thinking about performance. Our benchmarks are significantly better now, around 20% to 30%. There's always more performance to squeeze out.
And, you know, I would like to give a shout-out to like a lot of people who are writing tools who think about performance really carefully, such as the authors of ESBuild, and the authors of SWC. Ourselves at Roam, we put a lot of effort into that. And in fact, I actually... The benchmarks I showed you guys in the talk are actually significantly out of date. We've been doing a lot of performance optimization since then, and I would say that our numbers are perhaps like maybe 20% or 30% better than that, which is really fantastic work. And I'm so pumped to show that to people. I think, you know- That's really cool. Yeah, there's always more that you can do. There's more performance you can squeeze out. So yeah.
Performance Improvements and Fast Tools
At Roam, we have implemented performance improvements by optimizing the parsing process and minimizing redundant work in formatting. We aim to make parsing faster by reducing backtracking when the parser encounters a different syntactic construct. Additionally, we focus on ensuring the formatter performs only the necessary actions without unnecessary text rearrangement. Fast tools are essential not only for CI but also for developers using slower hardware. It is unfair to write code that only works on high-end machines.
Just, I guess, share a little bit. What kind of performance improvements have you implemented at Roam? Just off the top of your head. So for instance, a big thing that you always have to do is parse the code. And parsing is notoriously slow for a bunch of reasons. First, you have to read in the code from I.O. And second, you have to create this big data structure. So we've been doing a lot of work to basically make sure that when you parse, you're doing the minimal amount of work because oftentimes you might parse something and then realize that it's a different syntactic construct. So for instance, you may be part, your parser might be parsing some code and realize, oh, wait a minute, it's not a JSX expression, it's a TypeScript type assertion because it turns out that these two things look very, very similar. And we've been putting in a lot of work into that process where the parser realizes that it's something different than expected. It has to backtrack. We're trying to do that less and less so that the parsing is way, way faster.
We're also putting in a lot of work into formatting, into making sure that the formatter is doing just the bare minimum it needs to. And it really isn't doing redundant work in moving text around and rearranging the code.
Okay, I see that, folks, I'm just going to encourage folks to first, reminding you about the Slido, if you want to tell us what other tools you find annoying and slow, we'd love to hear from the 11% of you. And also which IDEs you're using. We'd love to hear that. And also if you answer in time, maybe Nicholas can give us thoughts on why he thinks those are specifically slow tools. But, I mean, I guess from my personal perspective, I'd like to say that I don't find my CI very slow. So why is this something that should matter? Why should I care about this? That's a really good point. And, you know, I'm happy that your CI isn't slow. I'm happy that, you know, you're not waiting like some other people are. I would say that I think Fast Tools are universally a good thing, even if it's not for your CI. I think, you know, it's really, again we have to remember that not everybody is running, you know, a super modern computer. Like, as I said in the talk, you know, we, like often as developers at, you know, companies we get, you know, modern hardware, like, you know, pretty often. But like, you know, there's plenty of people out there who are just starting out, or maybe at companies who don't have a budget where they're running on, you know, slower hardware and we have to accommodate them. I think it's really not fair to write code that only works on, you know, a brand new MacBook Pro. It makes sense. That's, you know, yeah. Yeah, I can appreciate that, obviously. So that was spoken from very deeply within Turbulent.
Slow Internet Connections and Linter Details
Running code on slower internet connections is an important lesson for developers. Our Linter focuses on the slowest rules in ESLint to provide fast feedback. While there is currently no collaboration to share a single AST among popular tools, the TreeSitter Project is an inspiring example of a parser that can parse multiple languages. Sharing ASTs would require significant changes. Cece Miller asks about AI interpretation of code, such as GitHub Copilot.
That was one of those things, and I'm like, okay. Yeah, I mean, I've been guilty of that too. You know, I run the code and I think, oh, it's funny fast, but, you know, I think, I forget which company it is, but they had a habit of basically running their internet connection slower, like basically throttling it for the express purpose of accommodating people who are on, like, a 3G or even a 2G connection, because I think that's a really, you know, important lesson to give to developers.
Yeah, that's very true. Dainy asks, can you share some details about the Linter? How will it compare to ESLint in terms of rules, plugins, other factors? Yeah, so I'm really excited about this one because Linting is something that, you know, is and has been slow for a while. And obviously, ESLint is a fantastic project, and they have just a lot of rules, and I'm not going to come out and say that we're going to capture all these rules at once, because, you know, as everyone always says to me all the time, Rome was not built in a day. So I would say is that what we're focusing on is what are the rules that are the slowest in ESLint, what are the things that really make it painful, and then we're going to build a winter for those specific rules. And then, you know, the idea maybe is then you can run our winter, get fast feedback for certain things. And if you want a full, complete Linting setup, you can still run ESLint, and you can still get all of the feedback from ESLint, but you'll get the fast feedback first.
ML and Parser Error Recovery
I've played around with Tab 9 and Copilot, and I find them to be really cool projects. While Roam is not currently heading in that direction, we are open to exploring ML plus code in the future. One area we're considering is using machine learning for parser error recovery, leveraging data sets of incorrect code to estimate the intended code and offer fixes or better semantic analysis. Regarding supporting custom linter rules, we're still figuring that out. We aim to provide a wide variety of rules, but we acknowledge the flourishing eswint community.
Personally, I've played around with them. I'm a huge fan of Tab 9 and Copilot as well. I think those really cool projects. Yeah, Tab 9 is a fantastic project. I wouldn't say that that's necessarily the direction we're taking at Roam right now. I think it's definitely something that we're open to exploring in the future. I mean, I think, you know, ML plus code is a really fascinating area that will definitely develop more in the coming years.
I do think one thing that we have looked into and I have been considering is perhaps using machine learning for parser error recovery. So as I noted, in the talk, one big thing about our parser is that it has very good error recovery. Basically, it can notice when there's an error and basically try to guess what you meant by it. And right now we do that with some heuristics, but I think in the future, it could be a really interesting opportunity for machine learning where you can basically use data sets of incorrect code to estimate what the intended code is and then offer a fix or just better semantic analysis.
That's a really good question. I'm gonna be honest, we're still trying to figure that one out ourselves. I think, you know, again, we have this idea of like, sort of like R12 being like the scout that kind of goes out ahead. And like, you know, if our rules are incorrect you should fix those first. And then if our rules are all, you know, valid and the code is correct for these roles then you can run eswint and get the full completion. I don't, you know, again, I think we're going to do our best to offer a, you know, wide variety of rules and we're gonna keep adding more but the reality is eswint, you know, has a wonderful job and is also a very flourishing community. And, you know, we have to acknowledge that.
Awesome. You're getting really great feedback here. Just gonna tell you that what the community thinks really good insights from your partner in the discussion room, in the build room, Ante Tomić. Really good insights. He's curious about Rome, wants to check it out and a lot of really great feedback about the talk. I don't see, now is the time folks, this is it. We're coming down to the wire. Just get your last questions in for Nicholas.