Bring Code Quality and Security to your CI/CD pipeline

I will first present myself, myself and Tibor. So we are both developers from the Sonar Source company. Now we call it Sonar. And we are both developers of the javascript Analyzers. And today we will be speaking about how you can use the products developed by Sonar for bringing code quality and security into your project CI. We will see this with Sonar cloud and Sonar Lint. I'm not sure if you've heard of these products, but definitely you will learn about it soon. I will show you the plan. So let's say it's one hour and a half, but that's, let's say, a rough estimation. Maybe it will be faster. It depends on how things are going. So here's the plan. I will not just go over it, but you have an understanding that there will be eight, let's say, models. Each model is a baby step on setting up the pipe of your quality, your project quality analysis. And we will go one by one. I will stop sharing these slides. Okay, so now let's start slowly with the products which we're going to discuss today. So Sonar is a company which provides with code quality tools. This is, let's say, an ecosystem. Sonar Lint is an extension for the IDE, which gives you instant feedback while you're coding. Sonar Cube is a self-managed tool, so you will need to host it yourself. And Sonar cloud is a cloud-based tool which is hosted by Sonar. So you just buy a subscription there, or it is free for open source projects, and you just use it. So today, for the simplicity, we will use Sonar cloud, of course, so that we don't need to take care of setting up the Sonar Cube. So let's start. And I will ask you in the next slide, on slide number six, there is a link. So this is, I have it open already. That's the fork of the simple small project. This is a project from MDN. I forked it, and I changed it a tiny bit to have it more appropriate for our purposes. So could you please now fork this repository to your organization? So you have this fork button over here. I will not do it because I have already my fork, and you please do this. So you fork it under your organization where you have admin rights. And so you press here, and you should see your organization here, and you press submit. So as soon as you have forked it, please open it, so you should see your, I don't know, john slash to do react. And now you should, you can clone this project locally. You can copy the SSH link to it. And you can clone the repository locally with git clone. And insert this link. So I just have my, some empty directory, which I created for our workshop, and I just clone it in there. So we have cloned the wrapper. And now let's go to VS code. So VS code is one of the IDs which is supported by SonarLint. I picked it up because it's the most common ID for the javascript development. Also JetBrains IDs are supported, Eclipse, Visual Studio. I think that's it. So let's open our project into this. Okay, we'll just use the menu here. Open folder, and I will open this project here. So let's open the file source. I don't know where I got the file again. So let's open the file source slash app. So this will be the file we will be focusing today. So you might ignore the rest. So that's a simple react project which does the addition of to-do items. So your daily tasks or whatever. So it's a small demo project based on YouTube's yarn. So you can install yarn dependencies while it's not in fact so much required now. And now let's install the SonarLint extension. So in this extensions tab in VS code, where were these four squares, you can just type SonarLint and it's on the top and you will click on the install button. And it is installing and here it is. And so instantly we are getting some underlined code. So this is SonarLint telling us that there are some problems in the code or things to improve for the maintainability purposes. For example, there is a commented code here which might be confusing for maintainers. Here you have an import from react two times so you can merge these imports to be more concise, etc. So we have different issues here. In the VS code at the bottom you have this warning tab. At the bottom in this tab you see the SonarLint reporting you the problems in the code. So there are warnings. Okay, I'm not sure what is this icon meaning. In SonarLint you can configure the rules. So if you open this explorer tab, the first tab where you see the files, at the bottom there is a SonarLint rules tab. It's not super handy, but that's what VS code gives us. So you have the list of rules here. And those rules which are activated have on and those which are deactivated have off. So we have this default set of rules which is running by default and which we expect will satisfy everybody. But for sure you might want to disable something. For example, here we have like, okay, you don't need to create this variable. Just directly return this rule or this expression. So you might want to disable the rule and you just can click it from here. Also for each rule you have a description. You can also access it from this bulb. And you have here a description of the rule and examples of good and bad code. And the last thing we're going to do now is let's find in this list of the rules the rule about shorthand syntax. So here it is, object literal shorthand syntax should be used. So this rule is not activated. You have off here. So let's say that I want to activate it. And we instantly have a new issue. So in here we can instead of writing name, column name, we can just write name. And by the way, we have quick fixes just currently in the sprint. So we should have quick fixes available here super soon. So here it is. So we are done with our first model. So we cloned the wrapper. We installed SonarLint extension in the VS code. And we looked how we can enable, disable the rules, how we can browse the issues, et cetera. Okay, so I hope that everybody managed that to work. And now let's go to the next step that will be, I believe, a bit more interesting, I hope. So in your slides, you have a link to the SonarCloud. So this is a link to the login page. So you can use here this GitHub login. So you can simply click on here and you're logged in. So you will not see anything super interesting here because you have an empty account. But you can already, there is an explore project here on the right with this. I'm not sure how it's in English, binocular icon. And in here, you can see the projects which are already analyzed on SonarCloud. There are some featured projects. And so you can get some kind of understanding what is it like. So you can filter by size, language, and there is a project name filter. So this is SonarCloud. So this is a platform where you track the quality of your projects. So it contains many things, so you get bugs, vulnerabilities, maintainability issues, code smells, code coverage, duplications, plenty of metrics, the evolution with time, and many other great things. Some of them we will consider today. But what we want to do now is to add a new organization. So we press on this plus on the left top corner and we press and create new organization. So here we will press on this, let's say, default option, which is suggested, import organization from GitHub, because that's how we logged in. And you will be asked to which organization you want to install. I have several, so I just picked my personal one. And here you will be asked which repositories you want to select. You can select all or only the one which we forked. I will select the one which we forked. So it is to do here. And I will install it. I need a password. So here there is a key for the organization which is created. And the organization on SonarCloud is the same organization as you would have on GitHub, so where you can store the projects, configure them, share the configurations between the projects, etc. So I continue. So as I said, there is a free plan for the open source, sorry, for public projects, so these projects will be available to browse to everybody under this binocular sign. Or you can have paid, then it will be invisible. And the price depends on the number of lines of code. So here we are. We have our organization. So this organization has only one member who is me and all you. But of course you can have the whole team under one organization. And you can have different access rights, etc. So as soon as we have the organization, we can create the project under it. So I pressed on this, or you can press on the plus, doesn't matter. So SonarCloud suggests me to add this to do project. So I agree with it. And pressing on the setup button will automatically add the project to SonarCloud and trigger the first analysis. So now analysis is running. So that's the first initial analysis of the project. It will take some time, I don't know, maybe a couple of minutes. Okay, so I see somebody said they created it. Matias created it, finished import. I have one bug. Oh, I'm sorry. I'm sorry for you. Yeah, I did my best to put this bug. Indeed, this project had so little code and it was done so nicely that it didn't have any bugs. So you can see here that it's me who committed it. Committed it yesterday. So, yeah. So, indeed, so now you see your project. If you open the main branch, you see the overview of the things, of the issues. So you can see the same issues which we overviewed in the SonarLint. So this react import, commented code, etc. But we don't see the issue with the name, shorthand, the object property. Because this rule was activated only in SonarLint, so it was local configuration. This configuration is not available on SonarCloud. SonarCloud doesn't know about it. But on SonarCloud, there is also a way to configure your rule set. And that's where we stop at our second model. So you guys should open now your organization, which by default probably has your name. And in the organization on the top, you have quality profile. So quality profile, you see you have quality profile for each language. I think I didn't say that yet, but SonarCloud supports many languages. So you see here, so like probably almost every modern language, not even modern, we have COBOL and I don't know what else is, ABAP. Yeah, ABAP is on the top. Those languages are supported by SonarCloud. And those profiles, all these profiles, they're called SonarWay. So this is default profiles. This is the same profile you have on SonarLint. And of course, if you want some adjusted profile, as we did on SonarLint, you should create your own one. So we click on the Create button on the right top. And we will have, I don't know, for example, DevOpsJS. I always spell it incorrectly, DevOpsJS profile. We don't need ABAP in javascript. So I will find javascript in the list. And we need to pick SonarWay as a parent because that will make it inherit all the rules there, which is great for the start because you don't need to click all the rules you want, but you already have something by default. And this is also great because if in one month we add 10 more rules to the platform, you will get those rules activated automatically. So I press Create. And here is my profile. So that's profile in the organization. That means that you can make it available for every project in your organization. And that makes it super good in sharing the coding practices across the company or team. So currently we see here that no projects are explicitly associated to this profile because by default, default profile is used. And we can change this. And in all projects, we can pick our to-do project. And now we see that this to-do project is using this profile. Or you can even on this settings button here, you can set it as default. That means that every new project which you will add to the SonarCloud under this organization will use this profile. Okay, so I hope that was all good. Now those two profiles, so default profile and our devops.js profile are identical. And you see here that here you see the inheritance. It's all the same. So let's activate more rules. And in the slide, you see the rule key which I wanted us to activate. So it is javascript 1066. So you can search by rule key or you can search by the title. It's in parentheses. So this is a rule about two nested if statements which you can easily refactor. So you can activate it from here or from the list here. So you activate it in devops.js with some severity. And here we are. So now if we go into the profiles, okay, here we are. So now we see that devops.js profile has one more rule. But if we go into our project, we still see the same number of issues, so we don't have new issues for this rule. Because SonarCloud triggers the analysis only for each push. So let's go to the VS Code. And for example, let's fix this react import or you can do whatever else you want. It doesn't matter. We just need to push something. So we'll commit this import duplicate and I will push it. You can push with VS Code or with Terminal, whatever you want. So you see now that as soon as I push, there is a tab here that analysis progress. It tells you that it's up to 30 minutes, but that's really like the top because there are sometimes big projects, but it should take, you already know how much it will take. You already have one. Okay, so that's it. We have our results. So I will put main branch, that's handy for us now, and we will put code smell. Here is our new issue, merge test if statement on line 55 on this object. So this rule was not in the full profile. You can go to the rule here and see the description of the rule, some recommendation how to fix the issue. And you can just close it. Here it is. So here we are. So we finished with this profile configuring analysis rules. So now here is the situation that you have your SonarLink, you have SonarCloud. We activated one rule here, one rule there. So they're not in sync. That's not great for the user experience. And clearly you would like here to have the synchronization between the rule sets. That's where we can have the connection between them. So to connect them, in fact, okay, first I wanted you to show that in this line 55 that we don't have this issue for the collapsible if statements because this rule is not in the default profile and it's activated only on SonarCloud. So now let's open the settings. So you can go to the code preferences settings and you have a hotkey here which you can use for the future. So it's a comma for Mac users. And either you can search for SonarLink and you will see here the properties available or you can just find it under extensions. So here you can set up connection to SonarCloud. First, you need this property so you can press edit in settings. And here you'll see the, I'm not sure if everybody is aware of how settings for VS code are working, but this is basically a JSON file, which is just nicely displayed. So we already have here SonarLink configuration so you can also do it manually here if you want. But what we need now is that we need a SonarLink connected mode to SonarCloud. So you can start typing this property and just, and so VS code should suggest you the full property and you just click it and then you should create this, you should pick the last one with the organization key and token. Suggest that you should wrap it into array so that it's not complaining. So now we need to fill these two properties here, organization key and token. For the organization key, you can click on your organization, SonarCloud and you have your key on the right or maybe you just remember it from the moment you set it up. And for the token, it's a bit trickier. So you can click on the link in the slides. Or you can just go directly from SonarCloud, you open your account, you click on your account and you open security tab. So as you wish. So here you generate the token, so you put whatever name. So I will put DevOpsJS workshop SonarLink. So now this token appears at the bottom and I copied and put it in my VS code. So now we have this connection to the SonarCloud, but we still need connection to the project. So that's on the next slide, number 10. So we search for SonarLink connected mode project. And we pick this single one, single property project key. So to get the project key, we go to our project and you can find it under the information tab. And in fact, the organization key is here. Or you can also find it in the URL. So it's id equal and this is the project key. So here we are, we're almost there. So now you need to reload the extension. So I'm not sure if reload. So I think the best way would be just to close the VS code and to reopen it again. Okay, that took time. To have the proper startups for the SonarLink. And ta-da, we have our issue with the merge of if statements available because for the analysis, we used the profile from SonarCloud. Now I will show you one more trick. I hope it will work. So when syncing the projects between SonarCloud and SonarLint, the rule set is what we already have looked at. But also what you might want to sync is the issues resolution. So let's open our issues. And okay, let's say that, okay, here is the commented out code issue. Here it is in SonarLint. And let's say I don't want to fix it. I think that's great that this code is commented here and that makes it only better. So you can change the status of the issue. Resolve is won't fix. That means that you just don't want to fix it for whatever reason. So I like this comment and I want to keep it. So now this issue is not anymore open. You will not see it on your overview page. So you see here it is 14 instead of 15. And also SonarLint should be able to load this resolution and not display this issue anymore. But for that you need to update the connection. And of course you can restart VS Code. That will definitely work. But also you should be able to update the binding to the project. So for that in the view, command palette, maybe you know the hotkey for command palette already. And you can update the binding. And everything disappears. Sometimes I need to edit the code to make the SonarLint trigger the issue. And nothing is working. That happens when you want to show somebody something. Ok guys, you can just close and open the VS Code. Or try to reload Windows so that kind of soft restart of the VS Code. And it doesn't work as well. Sorry for that. Ok, let's then restart for the VS Code. Ok, here we are. I'm sorry for that. So, and you see that we have a lot of issues. But this command is not underlined anymore. Because the status of the resolution of the issue was pulled from the SonarCloud. So this way you don't need to leave any comments here like, yes, Lint, I don't know what to ignore, whatever. All those things are on the server and available on the SonarCloud. So here it is, the connected mode of SonarLint to SonarCloud. You can the same way connect it to SonarCube. And you will have the same setup for your ID as in your continuous integration. Ok, so we are done with the connected mode and now will be the next step, which is, I believe, the most interesting and the most difficult one. So I want us to be ready for it. It would be great if people... Maybe I will just mention one thing because there was one interesting question. So basically, the question was how you can configure SonarLint differently for different projects. And using connected mode is actually a way to do that, because in the connected mode, you have a quality profile that you can define per project. So if you are in connected mode, you can have a different quality profile for different projects. So this is the way how you can have different configuration for different projects. And also, the advantage is that you can share the configuration in the team. So SonarCloud can be a central place where you configure your clean thing, basically. And also, if you don't use connected mode to have different configurations. So these are user settings, but it's also possible in VS Code to have workspace settings. So if you open this, there is a workspace. It's currently empty, but this is the same way you can configure it. So you see it's created in my project and you can commit it and this will be specific to the projects. So now let's go to the SonarCloud. So we are back in here and you might see that we here don't have any coverage. So as we in the automatic analysis, which is done by SonarCloud to have these issues, we don't execute tests there as for javascript that's not always trivial and we don't have coverage. That's not possible to have this coverage with the automatic analysis. But we're going to have a GitHub action, which will run this for us, and we're going to have a coverage pretty soon. So for that, let's go under the administration tab. So administration at the bottom left and here you have analysis methods. So we have this SonarCloud automatic analysis on because that was the default when we were editing the project. And let's turn it off because we will not be using it anymore. And let's follow the tutorial on github actions. So again, cloud automatic analysis would trigger analysis on each push to the branch or on pull request. So now we're going to create a GitHub action, which will basically do the same thing, but there we will be able to add more steps and we will be able to add test execution. So first, you need to go to your GitHub repository. There is a link here. And you need to click here on the new repository secret. And you copy the value and name. So it will be Sonar token. So just simple copy paste both values from this tutorial to these two fields. And let's edit. Here it is. So now it's available. Let's continue. I hope that works for everybody. Now we need to create a built YAML file. So here it says create update, but as we don't have any, we will create it. So you can copy the path. I think it will be even handier here. So we have javascript. So you click on the right. Then you can copy the path. And in the VS code from the root of the project, you can create a new file and just insert it. VS code will automatically create all the directories which are required. So this is the default location for github actions. So GitHub will automatically know that, okay, this is the file I should treat as a GitHub action. And you can copy this file here. So here you have everything to have the same setup, the same analysis as you used to have. So we now continue. And you need to create one more file which will contain the configuration of the analysis. So this file calls for project properties. You also create it in the root. And you copy the content of it. You can drop all this content. So it says you are done. And indeed, if we push this, we will have analysis, but we will not have coverage. So in order to have the coverage, I will ask you to open the slides. Okay, I'm a bit far. So here it is. So you please add these two lines to the server project properties. So this will set the project version and this will set the location of the coverage report. So the coverage report should be in the ELCO format. In my project, we have here Jest test executor and we use here the results processor, Jest SONAR reporter, which will convert to the ELCO. So it will already generate it in ELCO format. So we are almost there. Now we need to add more steps to the build YAML because we need to run the tests. So we do it between these two steps, between the checkout of the project and between the SONAR cloud scan. Be careful here because this indentation is important to the format. So these dashes should be aligned and the width and users, et cetera, they should be aligned. So we use here another action available, which has yarn. Here we install dependencies and here we run yarn tests, which will generate coverage for us. Okay, Tibor, somebody is asking for the link to the slides. I don't hear you, Tibor, you are muted. Sorry, I will share the link again. Okay, thank you. Okay, I will try to be slower. Sorry, guys. I just noticed that we already spent an hour and that's why I was trying to be a bit faster. Okay, so we have here, so we added so far two files, build YAML, which is the configuration of GitHub Action and SONAR project properties. That's the properties for the analysis. There is nothing fancy here. Project key, organization key, that's what we already saw with you on SONAR Insight. Project version, it's not required, but that will be useful for us a bit in the next step. And the link to the, not link, the relative path to the reports. So, of course, this relative path should be there during the CI step, not on YOLO code. So, as soon as you have, as you are ready with these two files, you can commit it to the master. Set up SONAR cloud analysis. So, now I'm ready to push. So, as soon as I will push, GitHub will create GitHub Action. If you open the project now, okay, there are some github actions which are old, which I did like 20 hours ago when I was preparing. But you probably have nothing there. And as soon as I will push, you should have a new GitHub Action running. So, but maybe I can just show you how it would look like. As soon as you push, you're going to have here this, an item appearing, this Action tab. And you're going to have this SONAR cloud here with these steps, which is, first it builds the github actions. Then it checkouts, and so dependencies, generates coverage report and scans, and executes the scan SONAR cloud. And so, with this github actions analysis, you're able to see all the logs of the analysis. And if something goes wrong, you're able to see it. You see what is taking how much time, et cetera. So, maybe try it again, maybe it will pass. And as soon as we have this set up, we will also, in fact, it's already here that we see the SONAR cloud analysis in this GitHub. Yeah, so you will not see, okay, you will see something like this after you will have this GitHub Action executed. So, you will see the coverage of your project. And if you click on it, you will see the dropdown on which files have which coverage. And you can open it and see the green lines, which are covered, red lines, which are not covered, et cetera. So, this way, you might add not only coverage, but also, for example, import of external issues. If you're using some other tool to link your projects, and you can add external issues. So, you can add in your GitHub Action one more step on running your YesLink, for example, in here. And in the SONAR project properties, you can set this path to this YesLink issues report. Okay, so I see that, guys, people are writing that it's not working. So, while it's not working, I will go to another module about security hotspots. So, in the title of this workshop, you see that it's not only about quality, but also about security. So, SONAR cloud provides you with many security rules. For this project, we don't have any, but we have two security hotspots. So, I'm going to show you what is this. So, for security, it's not always easy to say that you have a problem here, that you are vulnerable here. You can have some places which are dangerous and which require a manual review. So, in our project, we have two security hotspots. So, those security hotspots are not just simple issues, because you can just review them manually and say that, okay, I just had a look, I understand what is it about, and it's safe. So, for each security hotspot, you have a description of it. And there is a tab where you can read how you can understand if you are at risk. So, if this is really dangerous for you. And also, you have a tab about how to fix this. So, here I have a HTTP, not HTTPS, protocol used for my URL. And I, let's say that, okay, I reviewed it here, there is no danger for me to use unsafe protocol. And I say that it is safe. So, as soon as it is safe, it will not be any more displayed. So, it will not be any more displayed under the security hotspots. And there is another one, security hotspot, where you create, where you load the script from the internet, and you are not checking the integrity of it. Here, let's say I reviewed it, I read the description, I say that, okay, in fact, I'm at risk. It's an external resource. I cannot be sure that this is what I expect there, and I have to put some integrity here. So, I have here some examples. I should put a SHA number for this.