The Lazy Developer Guide: How to Automate Code Updates?

Hi all and I'm sending you a big, big, big hello from Croatia. My name is Ante and I am going to talk to you today about very interesting topics which are code automations and how we did them in Neo4j. I hope that you will enjoy my session so let me begin.

Hi, I'm Ante and I'm 28 years old. I am senior software engineer working in Neo4j for nearly 6 years and I am focused on bad infrastructure. I am also a frequent speaker on meetups and and I have a lot of interests like React, Webpack, Application Build, Application Optimization, Performance, Web Architecture, Automation and Experience. And you know everyone who knows me would tell you that I am a huge photography buff and I love photography and I'm a happy photographer. So if you are interested, feel free to check out my photography blog on Instagram but let me start from a question, how do you handle repeating tasks at work? This is something I would like you to answer in the poll so let's see, maybe we are doing copy, paste and then repeat, repeat and repeat for the same thing or for example, we count 20 projects and then we change the same thing in all of them. Well, we do have a lot of repetitive tasks and this can take us a lot of time, especially with you know 50 or more projects, and we can see like in this blue line that we can do this manually you know, and we can also see this red which is actually what I want to explore in this session as it is also related to the concept of automation, which is extremely important, and I would like to start from a quote and that is I will always choose a lazy person to do a difficult job because a lazy person will find an easy way to do it and let's see actually how this applies.

First things first it is not related to that we should be lazy about learning. No, programming is a skill that takes hours and hours to perfect and we definitely should work hard to become better developers and to expand our knowledge. And then really then what the hell am I talking about? Well, we need to be able to choose our battles wisely because working hard is just as important as working smart. And in development or the missions are the key to achieving all of this and we are going to check this on a few examples. But you also need to mention thinking. Obviously it is the hardest part of development right? Especially when you have to name them. But thinking is actually a really powerful tool for the missions because it helps us produce ideas and solutions and also helps us actively seek challenges and improvements. But wait! We have to be careful not to overthink it because our time is valuable and we shouldn't spend too much time on complications as it will make it harder to support and update code in the future. Maybe we can have good intentions but our code can simply be too complex to understand. And this is something we will check in the coding examples. You can see QR code and repository link here so you can follow with me or you can check the repository later.

Okay then let's jump onto it. What I want to show you is this package.json file. This is my project for the dummy project or example one for the medical equations and I want to point your attention to these dependencies. What I have on the dependencies is CLI tool that is called oclf. Now it is your preference for what you would choose. I simply chose oclf because it is easy to integrate and it sets up a lot of stuff out of the box. So it helps me and it offers a lot of possibilities and integrations. But you know you have more options like commander or yars and of course you have oclf as well if you want to use it. But what you can also see is some other dependencies. You have some minor ones maybe less known ones like edit json file or replace-in file which you know you will maybe search for and use based on your necessity.

And you have more popular tools like stemware which is used for semantic versioning. It can help you you know check what version you have and if it is matching with the version that you need and yada yada yada.

Then we have shelljs which is unix-like library which offers you the option to execute unix commands in any environment like windows or mac os or linux or any of the other ones. And you know some of you definitely are familiar with uh you just go ships for doing updates and of course it can be used as well. But I would like to talk about two examples I'm going to show you.

The first one is eslint-update commands where we are simply going to check if we have eslint link fix which you can see also in packages on the right and if you check scripts here you will see that we don't have it. So we will add link fix to auto updating. The second command is update dependencies where we will simply check if we have react 16 and if we do we will simply bump the version to react 17.

Okay let's check so I have eslint upgrade commands with four parts the first one is filter where usually in gi I we use big packet api to download the package json file so that we can do some sort of quick checks. As you can see here in the filter so we use package.json file for each downloaded package.json for the project to check if we have eslint of course we have eslint so if I execute the command that you have here in the sample which can also help you you will see that I will get through azure result which means that my migration will execute. So that's the first step done. The second step is cloning the project. Every project for which we run filter every turn true will be cloned and then before executing the migration inside indexts I first check these migrations and this is a expanded filter where thanks to package package, you know, thanks to the whole project cloned, I can check absolutely anything I want to. I don't know for simplicity reasons I'm checking package.json here to simply see if I have linpicks command and since I don't have linpicks command, here you can see that it will return true. So, if I execute my migration command, it will return true and I will then be able to execute the main migration or at least it will be executed in the TI job. So, I go to index.js and I can see my main migration command. But it's written here like you can see here is additional dependencies that you usually install. This is where you will mostly use them. So, inside your migration job. And here I'm using the JSON file. As you can see here in the migrate command where I first get my path to the project, then I have added a JSON file and inside my lintfix step where I simply also check if I already have lintfix command. And if I don't have lintfix command, then I add it to a JSON file and it will also be automatically saved. And again, let's check the code sample here. So I will copy paste it and I will execute the run eslintupgrade command. And you will see here that now lintfix has appeared and I have changes inside my github project. So, voila, I know that it is working. The second step that I want to show you is update dependencies. But before I jump on to update dependencies, I want to mention briefly get migration options, which I won't get it but you can use it to get whichever options and information you need for automatic job execution or even for pull requests and that. So here you can pass, you can retrieve, you can calculate any information you need for your TI environment. But what I would like to do is I would like to jump on to update dependencies.

And what I have inside update dependencies, as I said, I am verifying if I have react-16, which if you look at the packages and I'm showing you from the sample test project here, you can see that I have react-dom version 16, which means that I am eligible for the update. Now, I won't show you all the four steps, but you can see here that I have updating react-dom. I have some exec command and npm installs. And I'm going to start the project. So this means that I'm doing some sort of Unix-like commands. And I am, as you can see at the top, I have shelljs where I'm executing some commands. So, as I have shelljs here, we have import xtd from shelljs, I have the commands. And if I run them, something will happen. And as you will see, you will be able to see how it actually reacts and react-dom updating step by step. Because see it reacts and then react-dom, and then everything else. And it will finish, and it will wrap up, and it will be successfully executed. But, this is not something I will wait for, because I want to show you one last example. And that is this file.

You know that when we write a project, we have ESLint and we have VDR. And we have those two. Those are for code formatting for any potential bug fixes basically to ensure the code quality. But, this is also something that can count as automation. We can also count as laziness. Why? Because we won't format this automatically. I can set up my IDE, for example, to save this and then format it, which is what I'm gonna do. And as I save it, okay, it doesn't work. But, I can simply execute the code commands here, and I can see that as I fix something, it is automatically formatted and it's done exactly the way how I want it to look like. So, it's also a very good example of laziness. And let's call it also automation. You can explore this project by yourself, if you want to, and it is extremely useful. And I'll also add more examples. So, I highly recommend you to check this out.

Okay. I would like to jump back on to the presentation. And let's look at the next step.

So, now, imagine that we have Git repositories. And we want to do something similar to what we did in the code examples. So, we want to update each project. And imagine that it takes us 15 minutes to do all kind of reupdates or 50 minutes per project. Now, if we do the little math, we will see that for 200 projects and 50 minutes for project, we need about 50 hours or a little bit more than six working days to do the manual code update. And we don't really have that kind of time, you know, especially if we do this periodically, like we intend to do. Automations can help us here, as we saw, as mentioned earlier.

So we need to be careful and we need to be aware about what we want to achieve and with which impact. So let's explain a little bit about our GI setup. Of course, in InfoWIP, we generally have a lot of teams and a lot of projects. And since we have that, it can be really hard and time consuming to update everything. And it takes a lot of time. And of course, it can help the team do the changes but it is prone to errors because we don't know why we need these changes or they don't know why we need them and they can make mistakes while updating them.

So to avoid that issue, we have our internal migration library with job and automatic executions to continuous integration and Jenkins. So what step do we do? We solve this. We do filter where we check which projects are eligible for updates according to Bitbucket API. We eliminate unnecessary projects and then we can continue. According to that or the scope or the results that we have left, for all of those projects, we simply clone them. After we have cloned them, before doing the actions, we can do the additional check to eliminate more projects. But of course, after we check, I don't know, this migrate, we move to index or migrate command where we perform whichever actions are defined. Then we have our code changes which we need to commit. Then push them and also then create a pull request so that the teams can check it and review it. And we have more examples here.

One example is very familiar for all of you and that is continuous integration and continuous deployment where we can also use a lot of tools like Jenkins or Docker or Kubernetes with even some built tools like webpack, rollup, I don't know, parcel, and much more. And we have one good use case of InfoWeb site and that is continuous localization. What we want to have is we don't want the teams to worry about localizing the product, and we want to achieve automatic integration or localization inside a project and we want to offer multiple languages to our customers. So we have about 50 web projects that require updates and localization. So instead of manually updating everything, we use automation.

And the use case that we have here is UXWriter will change the text. They will save the changes inside cloud-based letters management system. And then we will periodically run automation job over all web projects that will replace our text with the new one that is added by the UXWriter. And then it will generate a pull request for the teams to review. And we have one more example, which is maybe something we are not familiar with, but it's extremely useful and it is related to development text, which are very important, but they are hard to support because complex tasks, such as integration and end-to-end are especially very infeasible to support when you are under the deadline. But we have automated and autogenerated testing to motivate testing, where you can tell the program, hey, here is how the application is supposed to behave and work like. So please generate the test and also first record the edge cases. And then thanks to State Machine and State Charts, we will have the tests generated for all use cases. And here we can see a simple example with walking a dog, and we can see how we get from one state to the other one.

And we can see how you get from a wading to a walk. You simply need to leave the house. And this is really good and really useful, and it's a very good and wide topic. And I recommend you to check the links above which are documentation and visualization. And we have an awesome person whose name is David Kershad, who is the author of S-State. I'm really doing a lot with it, and I definitely recommend you to check his sessions. Especially in the QR code below.

And take a look at the graph. I have to tell you that it is not a contest about getting the least amount of work done, because we want to contribute as much as possible, but also use our time with as much value and impact as possible. And we are here to help each other. So this is not a competition. All of us should win. So if we can automate something that can help others, then just do it. Especially if it can help you be more productive and allow you to work in parallel on something else.

And some of the factors that are important for automation are time, especially how much it takes to do some repetitive tasks, energy, because we can spend a lot on repetitive tasks, and it can be exhausting mentally. Then, doing the same repetitive tasks can be really annoying and frustrating and affect our productivity. Then, if we switch to a different task, we can lose focus and take more time to get back to it. And also, situations like this are good when you have already met repetitive tasks, so a good learning curve with which you can grow is learning automation, learning something new and taking outside the box. So, this is something good for the wrap up because we should always look for the patterns and we can check if we need to automate something. And time is valuable, and we should focus on productivity and changes that can have impact with the client. And we need to keep our code simple and easier to update later. If you want to learn more about this, you can check my small page which has summary, materials, and some additional information. Feel free to check it out. I hope that you can find this session useful and that you can learn something new. You can contact me on email and thank you. And I look forward to hearing your questions in the Q&A in a few moments. Thank you and bye. Are you surprised? No, I'm not. This is how people, I want to remind this how people handle repeating tasks at work, right? Go ahead, Ante. Well, you know, it is usually something frequently because automations are very much a part of everyday work.

We have the tools, which I have mentioned, like ESLint or PDR or even some similar examples. So all of that, you know, added together really means that all of us have automation basically every day. Yeah, absolutely. And why do you think that there is like a delay in the adoption of automation to this day 2022? Do you think that we automate enough? So that we also made it up, it could be yes, because you know, everything is evolving. And as everything is evolving and growing, you know, web development or anything else, then of course, my automation will grow as well. We have a lot of possibilities for exploring and achieving everything today.

Absolutely. And what libraries or tools do you recommend for automation? What are your favorites? Well, we have a lot of possibilities today. You know, definitely number one tool that I recommend if you do some of the metascripts is something that everyone knows that it's done well. Because we have semantic versioning in Node.js project. What we get with it is you can actually check the version. We can see if we actually meet some condition. And it is pretty powerful tool. Especially when you need to check, let's say, packages, like I did in example.

Then we can also have Unix command. Because sometimes you need to execute and PM install or uninstall something. Or even come to some file. That is something when if you have, let's say, Windows computer and you are running your CI job on, I don't know, Ubuntu or Linux machine, then you can have issues with inconsistencies. If you can use something like unifying Unix command, which is precisely shelljs which I mentioned, it is quite good. And I also have to especially mention something I didn't mention much during the session, and that is Cisco ship. And running code mode and even writing your own one. Which is something that can definitely be extremely useful.

Right. So, what you're saying is, it's good to be a lazy developer, and automate as much as possible. Right? Because we have many opportunities and we have many tools that could help us or can help us do these repetitive jobs. Yeah. And not manually. Right? Well, yeah. You know, we don't format code manually or something like that. We already have some sort of laziness inside and that is in form of such tools. And if we can automate some action, which you know, also, because machines really define what we want and as we define what we want to achieve, we can get that automatically. We can focus on doing some action that can actually really contribute further because if we eliminate the need to manually print that code, we can have more time to deliver some other features to achieve more. Yeah, that is more important, right? We can focus on the code, we can focus on a new feature, and instead of focusing on non-differentiative tasks. Yeah, that's right.

And how do developers or teams or tech leads decide when is the right moment to automate or build an automation pipeline for a task. Well, that's actually something that usually depends.

For me, deciding when you should automate is all about knowing, having the feeling. Because as you do some action, it will frequently not be only about time. It can also be something that can give you some insights about whether you have your action. If you do it again and again and again, then what you can do is if you can do this automatically, like the example with replacing the phases, then you will know it. Because you know you'll get impact. And I think it's all about that. So it's about impact. And you can also consider the time a little bit, but it is not the highest factor. So if you consider how you work with something and that, then you can know when it is the right time for automations.

Right. When you see you're doing something over and over and over again, manually, maybe you need to consider automating, the task. But you need to also consider what kind of code it is, because if you are doing not exactly the same thing, so it is something that's hard to write by script, then that would be the case when you will not be able to automate. But if you have something where you can actually do at least one part, then it is worth considering it, especially if you take into consideration the impact. Absolutely.

What made you interested in this topic of automation? Do you have responsibilities at work that involve automation? Well yeah, we had a lot of projects and we had an idea. We had the job inside the team when we were working out because we had to do a lot of changes and we saw that it was something that would take us a lot of time because imagine for example that you have to update, back then it was 30 projects and our estimation was that we needed to manually apply the changes gone and that we needed at least 30 minutes per project. So we saw that would take us too much time because basically you could spend a whole week only updating the projects and not doing anything else. So we started to think what if we could actually create some sort of CI environment or Jenkins environment where in the project we could define what kind of changes are necessary for all those projects and then when we take that, let's say define migration job, we execute it on all projects or the defined set of projects, then we get the result and the teams get the pull request. Of course we have to be involved with some way, but we can actually do is we can build the code. And that is actually easier because yes, we can have the teams also do the changes manually but the problem in that case is they will actually have no knowledge about why we need that, why we need this part, why we need that part, why we need that. And we know that. And if we apply the changes by the script, we will be sure that it is exactly what we need and we will also narrow down the margin of errors. So we have the teams by providing the changes efficiently and we also have avoiding the margin for errors. So again we were thinking about the impact and what we can get with all of that. I think you mentioned something very important and it's that automation usually is self-documenting. So when you see an automation pipeline, you can by default understand the rationality behind it. If it's perhaps documented or there is a reference to documentation of course, but in general that's a very good point. Well yeah, I would like to add one more thing, so when you are a developer, let's say learning React and you know you learn the basic React, what is usually natural with developers is you actually start exploring more, exploring the details, actually see how everything is working, so you try one thing, you try the other thing, and then you see what happens. And the same thing with automation, when you do them you actually also learn how some code is working and what actually the necessary changes are, because when we have let's say gsgo ship or codemodes, we then explore abstract secret tree, which is how we can read the code, as these are how computer can read the code, because we made it understandable for us, and we also made it understandable for the computer, because we actually defined the necessary actions you know. Absolutely, absolutely, and I think the gain from automation is measurable as well, so it definitely helps you see what is the actual performance of a team, right, instead of having the team again being busy with with tasks that really bring no value at all.

Yep, that's right, yep. Yeah. So can you tell us very quickly what was the last thing you automated within your project? Well, the last thing I automated was actually applying some translation updates over, I think, about 50 projects. So we did some sort of updates like preparation for a future feature and we had to apply the same set of changes. So we did it with a TI job to actually call the project and do whatever is necessary.

Wow. Great. So remember for everyone that is watching that you can continue to ask questions on automation on Discord, devops-talk-Q&A for Ante to reply and to give you his ideas on this amazing topic. Thank you so much, Ante. Great to have you here. Thank you also and wonderful to be here.