Testing CLI Utilities

Rate this content
Bookmark

Ever wondered what is the best way to end-to-end test your custom command line utilities? In this talk Florian Rappl will give you some insights what you can do to automatically verify your CLI tools and avoid regression.

- Introduction: Why test CLI tools

- Challenges: File system pollution, network and database issues, environment variables

- Demo: Showcase issues with a demo CLI tool - Solutions: Test plan implementation, choosing the right level of containerization

- Demo: Show solution using the previous CLI tool

- Conclusion

Florian Rappl
Florian Rappl
34 min
03 Nov, 2022

Comments

Sign in or register to post your comment.

Video Summary and Transcription

CLI utilities are important to test because they act as an intersection point between different parts of an application. The main challenge in testing CLI utilities is performance, which can be improved by using temporary directories. Managing ports and resources is crucial to avoid conflicts when running multiple test suites. The test context ensures that processes run in the correct context, including the use of the right directories. Running tests on different configurations helps identify compatibility issues and provides comprehensive test coverage.

Available in Español

1. Introduction to Testing CLI Utilities

Short description:

Welcome to the session, Testing CLI Utilities. CLI utilities are important to test because they act as an intersection point between different parts of an application. They can access network resources and interact with the system locally, which requires proper coordination and sandboxing. These challenges make CLI utilities particularly interesting for testing. For example, PyCLI is a utility that helps with web development tasks like scaffolding and running debug processes.

Welcome, everyone to the session, Testing CLI Utilities. I hope you are as motivated as I am to get started.

Before we begin, let's have a look at my person. Hi, I'm Florian. I'm a solution architect at a smaller company based in Munich, Germany, called SMAPIOK. We are mostly doing IoT and embedded computing and we are specialized in building digital transformation projects, especially using distributed web applications. Also, I'm an open-source enthusiast. What does that mean? Well, I've been awarded a Microsoft MVP in the area of development tools for the last decade. I spent most of my time doing projects in .NET, JavaScript, TypeScript, web development space. I'm writing a lot of articles and I've also written one book already about micro frontends and I'm currently writing another book about development of frontend applications with NoJazz, so be sure to get a copy. But enough about me, let's just jump right into a topic before we run out of time.

So CLI utilities, what makes them difficult to test? What makes them also appealing to test? Well, first of all, I mean, CLI implies they're running the command line, so this is on the one hand good because spawning a terminal process is always quite easy and in some sense also easy to work with, much easier to work with, for instance, a graphical user interface. On the other hand of course, you need to deal with some things like for instance, receiving the standard output or also placing some inputs on the standard in stream. And you of course need to coordinate that and have your asynchronous processes right. CLI utilities often provide, let's say kind of an intersection point between two parts of an application so they are quite important to test to get right. And we are relying a lot of CLI utilities, so I mean having them working reliably is of course always what we strive after. Anyway, these CLI utilities like any other application, they might also access some resources. For instance, of course, network resources. And yeah, I mean, you may want to mock these, you may need to run local services and you may also need to coordinate these resources. So that's something you need to keep in the back of your head. Also of course with each step that you do in a mocking direction, of course you remove a potential source of error for a later run and you will need to accommodate for that. The most important area though is of course what happens locally on the system. So for instance on the file system, read and write operations need to be sandboxed properly. If you run especially multiple tests in parallel, you can't just go in blindly and say, whatever does these utilities that I'm testing are doing, I just trust them that they always of course work in, I don't know, dedicated directories and that there are no race conditions or whatever, really running them in parallel. And that of course applies to any kind of system resource that they access. So we've got already a set of challenges but also a set of things that make CLI utilities particularly interesting for testing.

Right, so going a little bit further and thinking about some challenges that arise in that, we've already heard about input output serving content was already briefly touched just to remind you. So for instance, let's say what we will be testing is so-called PyCLI. It's a little utility that helps with a couple of web development related tasks. One of these is scaffolding, but another one that you would do quite after the scaffolding is running, for instance, a debug process with it.

2. Verifying Port and Content, Handling Fragmentation

Short description:

The CLI utility opens a port and serves output on that port. To ensure correctness, we can ping the port and use utilities like BlayWrite to access and verify the content. Fragmentation is a common challenge where the utility places content in multiple directories. This can be avoided by cleaning up after each test and ensuring test isolation.

And this debug process is actually opening a web server on your local machine. Now we want, of course, to verify that this has been opened successfully. And so of course, this part of the CLI that it opens some port is something that we need to consider. Now, how do we ensure that the correct port was opened, but also then, of course, that the output served on the port is right? You need to, of course, have all those questions answered.

In our case, what we did, of course, is we ensure we have just some ping on the port, that the port is alive, but then afterwards what we can do is using utilities such as BlayWrite for actually accessing the content, reading out it, and verifying it against, let's say, an expectation source that the actual reserved resources is the same.

Now, fragmentation is something that will appear quite often, which means that a CLI utility might, let's say, place content that it, for instance, creates in a couple of directories. And you don't want that to happen so often, because first of all, you need to clean up after each test. And second, you need to ensure that the test, of course, stays isolated, right? That you can run, for instance, multiple test cases in parallel or multiple test suites. And if you don't have control over where the utility that you want to test is placing files, you might run into these race conditions that I mentioned earlier.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Atomic Deployment for JS Hipsters
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
Atomic Deployment for JS Hipsters
Deploying an app is all but an easy process. You will encounter a lot of glitches and pain points to solve to have it working properly. The worst is: that now that you can deploy your app in production, how can't you also deploy all branches in the project to get access to live previews? And be able to do a fast-revert on-demand?Fortunately, the classic DevOps toolkit has all you need to achieve it without compromising your mental health. By expertly mixing Git, Unix tools, and API calls, and orchestrating all of them with JavaScript, you'll master the secret of safe atomic deployments.No more need to rely on commercial services: become the perfect tool master and netlifize your app right at home!
Effective Performance Testing to your Server with Autocannon
TestJS Summit 2021TestJS Summit 2021
36 min
Effective Performance Testing to your Server with Autocannon
Top Content
Performance testing expertise that is developed for a long time. In order to measure your server performance you need a tool that can efficiently simulate a lot of abilities and give you good measurements according your analysing criteria.Autocannon NPM library gave me exactly that - that library is super easy to install and has a very simple API to work with. Within a really short amount of time you can start do performance testing to your application and get good measurements in development environment and in your performance labs, and generate complicated testing scenarios.In this talk I will introduce Autocannon, explain how to efficiently analyse your server performance with it, and show how it helped me to understand complicated performance issues in my Node.js servers. At the end of this lecture, developers will be able to have the ability to integrate a fast and easy tool in order to measure your server performance.
Delightful Integration Tests With Testcontainers
TestJS Summit 2022TestJS Summit 2022
21 min
Delightful Integration Tests With Testcontainers
Top Content
Dockerized services are an excellent tool for creating repeatable, isolated environments ideal for integration tests. In this session, we'll look at the Testcontainers libraries which provide flexible and intuitive API for programmatically controlling lifecycle of your service dependencies in Docker containers. Running databases, Kafka, Elasticsearch, and even cloud technologies, straight from your test code ensures environment config is always up-to-date and consistent during local development and in CI pipelines.You’ll learn everything necessary to start adding powerful integration tests to your codebase without the headache of managing external service dependencies manually!
Visual Regression with Puppeteer, Playwright and Cypress
TestJS Summit 2021TestJS Summit 2021
9 min
Visual Regression with Puppeteer, Playwright and Cypress
Top Content
Visual Regression tests components via screenshot matching. I'll show how you do that in three different libraries/frameworks. Additionally, I will use Storybook to extract the components from your SPA choice.
Playwright Can Do This?
TestJS Summit 2022TestJS Summit 2022
23 min
Playwright Can Do This?
Guaranteeing that your application doesn't break while constantly shipping new features is tough. Obviously, with a continually growing app or site, you can't test everything manually all the time!Test automation and monitoring are crucial to avoiding shipping broken apps and sites. But what functionality should you test? When should you run your tests? And aren't complex test suites super slow?In this session, we'll get our hands on Playwright, the end-to-end testing framework, and learn how to automate headless browsers to ensure that you confidently ship new features.
The Lazy Developer Guide: How to Automate Code Updates?
DevOps.js Conf 2022DevOps.js Conf 2022
22 min
The Lazy Developer Guide: How to Automate Code Updates?
How to update hundreds of projects all at once? With organizations rapidly growing, demand for the scalability of the teams grows which directly impacts projects structure and ownership. The usual dilemma is mono- vs. multi-repos, but ... What if I tell you that it does not matter much? Both approaches can punch you in the face at some point, so perhaps it is better to think bottom-up.
Today I will walk you through some of the biggest challenges that exist in both approaches and those are managing dependencies across a few hundred projects, global code updates and many other things. I will also show you examples of how we solved this inside Infobip through building our own internal libraries.

Workshops on related topic

Automated accessibility testing with jest-axe and Lighthouse CI
TestJS Summit 2021TestJS Summit 2021
85 min
Automated accessibility testing with jest-axe and Lighthouse CI
Workshop
Bonnie Schulkin
Bonnie Schulkin
Do your automated tests include a11y checks? This workshop will cover how to get started with jest-axe to detect code-based accessibility violations, and Lighthouse CI to validate the accessibility of fully rendered pages. No amount of automated tests can replace manual accessibility testing, but these checks will make sure that your manual testers aren't doing more work than they need to.
Automated Testing Using WebdriverIO
TestJS Summit 2022TestJS Summit 2022
163 min
Automated Testing Using WebdriverIO
Workshop
Kevin Lamping
Kevin Lamping
In this workshop, I cover not only what WebdriverIO can do, but also how you'll be using it day-to-day. I've built the exercises around real-world scenarios that demonstrate how you would actually set things up. It's not just "what to do," but specifically "how to get there." We'll cover the fundamentals of Automated UI testing so you can write maintainable, useful tests for your website and/or web app.
JS Security Testing Automation for Developers on Every Build
TestJS Summit 2021TestJS Summit 2021
111 min
JS Security Testing Automation for Developers on Every Build
WorkshopFree
Oliver Moradov
Bar Hofesh
2 authors
As a developer, you need to deliver fast, and you simply don't have the time to constantly think about security. Still, if something goes wrong it's your job to fix it, but security testing blocks your automation, creates bottlenecks and just delays releases...but it doesn't have to...

NeuraLegion's developer-first Dynamic Application Security Testing (DAST) scanner enables developers to detect, prioritise and remediate security issues EARLY, on every commit, with NO false positives/alerts, without slowing you down.

Join this workshop to learn different ways developers can access Nexploit & start scanning without leaving the terminal!

We will be going through the set up end-to-end, whilst setting up a pipeline, running security tests and looking at the results.

Table of contents:
- What developer-first DAST (Dynamic Application Security Testing) actually is and how it works
- See where and how a modern, accurate dev-first DAST fits in the CI/CD
- Integrate NeuraLegion's Nexploit scanner with GitHub Actions
- Understand how modern applications, APIs and authentication mechanisms can be tested
- Fork a repo, set up a pipeline, run security tests and look at the results
Security Testing Automation for Developers on Every Build
GraphQL Galaxy 2021GraphQL Galaxy 2021
82 min
Security Testing Automation for Developers on Every Build
WorkshopFree
Oliver Moradov
Bar Hofesh
2 authors
As a developer, you need to deliver fast, and you simply don't have the time to constantly think about security. Still, if something goes wrong it's your job to fix it, but security testing blocks your automation, creates bottlenecks and just delays releases, especially with graphQL...but it doesn't have to...

NeuraLegion's developer-first Dynamic Application Security Testing (DAST) scanner enables developers to detect, prioritise and remediate security issues EARLY, on every commit, with NO false positives / alerts, without slowing you down.

Join this workshop to learn different ways developers can access NeuraLegion's DAST scanner & start scanning without leaving the terminal!

We will be going through the set up end-to-end, whilst setting up a pipeline for a vulnerable GraphQL target, running security tests and looking at the results.

Table of contents:
- What developer-first DAST (Dynamic Application Security Testing) actually is and how it works
- See where and how a modern, accurate dev-first DAST fits in the CI/CD
- Integrate NeuraLegion's scanner with GitHub Actions
- Understand how modern applications, GraphQL and other APIs and authentication mechanisms can be tested
- Fork a repo, set up a pipeline, run security tests and look at the results