Rise of the Robots
AI Generated Video Summary
This Talk discusses the possibility of robots taking over based on Asimov's three laws of robotics. It explores the use of automation robots for testing, including building and controlling them. The Talk also covers implementing interfaces, conducting math game challenges, and the capabilities of automation testing. It addresses questions about responsive design, camera attachment, and the future roadmap. The affordability of the setup and the potential for future automation are also discussed, along with a rapid fire Q&A session.
1. Introduction to the Question of Robots Taking Over
Hi, everyone. I'm Theodor, a software engineer and founder of Proxima Analytics. Let's discuss if robots will take over based on Isaac Asimov's three laws of robotics.
Hi, everyone. I hope you're enjoying the conference so far. Okay. So I'm Theodor. I'm a software engineer based in Athens, Greece. I'm also the founder of Proxima Analytics. It's an open source ethical first analytics platform that you should definitely check this out. And you can also find me online via the alliance wordless. So if there's a question that we can make for 2023, it's going to be are there robots going to actually take over? Like, is going your freeds going to take you as a hostage, or are you going to lose our jobs as software engineers? And truth to be told, this question is, like, rather too old, if we can say so. So this is Isaac Asimov, one of the most famous science fiction authors. And in one of his books, which is called iRobot, in 1953, I think, he came up with the three laws of robotics. So basically, this is a manual when robots are going to actually take over. And today, we're going to challenge this question, and we're going to try to find out if this is true or not.
2. Exploring Automation Robots for Testing
In 2018, while working for a company, I experimented with different aspects of software engineering. Mobile end-to-end testing is challenging due to sandboxed applications and external interruptions. I had the idea to build a robot for automating tests on real devices. There are three categories of automation robots based on movement: Cartesian robots, robot guns, and Delta robots.
So fast forward, in 2018, I was working for a company. And besides that, I was also trying to experiment with different aspects of software engineering, trying to mix up things like 3D printing, electronics, and stuff.
And I was working for a company where we had three major products. Like, the first one was a web application and two mobile applications, one for Android and another one for iOS. And I can really tell you this. So mobile end-to-end testing is pretty damn hard. It really is.
So if you have ever tried to run end-to-end testing in mobile applications, we're basically more or less stuck to the emulator. On the other hand, we also need to test mobile applications on real devices, right? But it's quite tricky, actually. So for security reasons, most of the applications are sandboxed. That means that we cannot actually test how our applications are interacting with the operating system. We cannot test sharing links between applications. We cannot test a workflow where we want to authenticate users using the email client in the mobile phone.
Moreover, we have external interruptions like phone calls. Mobile phones are like living organisms. So we have phone calls, notifications, push notifications, and so on. And then I had that weird idea, like, what if we could actually try and build a robot for automating tests in real devices? And you know what? I know that most people would actually think something scary or big like the Terminator or so. But truth be told, we can categorize automating automation robots in three big categories based on their movement.
So the first category is they're called Cartesian robots. We have three axes. And basically, the actuator moves into the three-dimensional space using bolts and wheels. 3D printers and CNC machines are working in that way, in that format, actually. But in our case, it does not work because you have a limited area to work with. And also, the movement does not feel that natural.
Next up, we have robot guns, which are basically made with separate motors and three or four attached parts. They are a needless standard for the car industry and medical operations as well. But they're pretty hard to operate. And moreover, they're pretty expensive and mostly used for repetitive tasks. So our pickup for today are called Delta robots. We have a base on the top, three or four motors attached.
3. Building the Robot with Tapster and Joni 5
And as you can see, we have pairs of arms that are connected to the actuator. They're pretty easy to operate. Basically, we can use basic trigonometry in order to calculate the position and move the robot along. And the movement is fast, efficient, and really natural.
But how we can actually build that kind of a device, right? Luckily for us, there is a company called Tapster. And it has been founded by Jason Huggins, who is also one of the creators of Appium and Selenium. And two of their devices are actually open source. So this is called Tapster Bot. This is the version one. And based on his design, I actually forked the entire setup and the code. And today, I'm going to present you a real device. So this is Margaret, as you can see here. Thanks so much. By the way, this whole setup runs inside my browser, so this is a live feed here. As you can see, we have arms attached here. Most of this part is using bolts, nuts, and screws, and 3D printed parts as well.
The brain and hearts of the robot is actually an Arduino nano microcontroller. So Joni 5 works as a proxy. The device is connected to my workstation, and then we can directly run a Node.js server in the backend, and we can send commands that the robot executes them. So this is a basic class that allows us to run a really basic robot. We first have to instantiate the Joni 5 board. The microcontroller has labeled pins. So we can say that we're running three server motors on pins two, three, and four. And then when the board gets instantiated, we can run the actual class.
4. Interface Implementation and Robot Control
And here is the basic interface implementation for tapping, moving, swiping, and resetting the device. With a Node.js server in the backend, we have the flexibility to expose REST APIs or web sockets. The robot can be controlled through the x, y, and z-axis, and even perform dance moves. Calibration is necessary to determine the mobile phone's position, achieved through a simple web interface. Swiping involves touching the screen, dragging to a point, and releasing. The implementation includes a Tinder-like UI for selecting options. With the Node.js server, we have various options for controlling the robot.
And here is the very basic interface implementation of doing so. You have the initialization function, and then some helper functions like tapping, moving, swiping, and resetting the device. And I told you that this is live, right? So since we have a Node.js server in the backend, we can do whatever we want to.
So here I have... We can expose REST APIs or even web sockets. So here, as you can see, I can directly move the robot through the x, y, and z-axis as well. Or I can actually make it dance for a while. The basic movement is more or less sending commands and coordinates, like go to point 0, 0, 0, and so on. Let me just unfocus this one.
So if you think about it, we can move the robot just sending the coordinates within an interval. If we're talking about taps, like tapping the actual device, that's just about lowering the z-axis to 0, like touching the actual device on the bottom of the robot. And here's the basic implementation of doing so. But there is one thing here. Because that device does not exactly know where the mobile phone is, we need to actually calibrate and let the robot be aware of where the mobile phone is. In order to do so, I have created a really simple web interface. So we can start the calibration process. The robot lowers down to the device, and when the touchscreen gets touched, it sends a command back so we can take pinpoints and find out where everything is right now.
Now let's talk about swipes. Now that the robot is actually calibrated, swiping is just touching the screen to point A, dragging to point B, and releasing. In order to do so, I have recreated the Tinder UI, you know, the one with the cards. So we can start picking our mate for today. Cool. Let me unfocus this one. Okay. So now that's everything we actually need to do. So we have taps, and we have swipes. Since we have the Node.js server in the back end, we can do whatever we want to. We can direct with the REST API. We can use a web socket. We can actually also use Opium.
5. Interactive Math Game Challenge
In this example, we're just instantiating the Opium JS SDK and sending commands back to the web server. We're going to challenge the real robot and have a simple math game where you can compete against the robot for a chance to win prizes. The game involves solving equations as fast as possible. Get ready to play!
In this example, we're just instantiating the Opium JS SDK, and then send directly commands back to the web server. So everything is in place, and we can start testing out our application.
But I know that you want the full-fledged demo, right? But as I told you, today we're gonna challenge the real robot. So I want everyone in this room or remotely to pick up your phones. That's a new one for a conference, right? So you're gonna compete against the robot.
We have a really simple game here. We also have two big prizes, right? If the robot wins, we're doomed. Try to find out the bunker. Try to run away. But if someone of you wins, then there's a full-year sponsorship by Egghead, like one full year, as well as a copy of Node.js design patterns by Luciano Mamino. I provided this one for one lucky winner. And here's the game.
You're gonna commit with math against the robot live in this room. So you can go to dab.sh slash play. You have to go inside, just pick a username to do so. Please, please, people, be kind. And once everyone gets logged in, we can start playing around. The game is really simple. You have an equation, like a mathematical addition. You have to pick up the right answer as fast as you can. Time matters here. Okay. I'm just gonna give you, like, ten seconds or so. The whole game will last about one minute. Everyone set? Everyone set? Are we good? Okay. So, are you ready? Just one minute. And you can actually start playing. Okay. It's hard, right? Okay. Almost there.
6. Automation Testing Capabilities and Conclusion
Five. Four. And we're done. Congratulations to Rahul, Mikhail, and the robot. We can test deep linking, sharing, authentication flows, run the device 24-7 on different devices, test post notifications and interactions with the operating system. We can attach a camera to detect what the robot touches or swipes, get meaningful metrics, replicate bug scenarios, stress test the application, and train our own AI models. Thank you for your time and patience.
Five. Four. And we're done. Hands on the phones. Everyone. Okay. Let's see how that went. Are you ready for the results? Yeah. Let's do that. Chatterbox Coder, is someone else? No? Okay.
In the application, we can find your ID that you can send me over through a DM. Congratulations to Rahul, Mikhail, and the robot, actually. But really good job. Actually, Chatterbox nailed it.
Okay. So. This is a wrap. We're going to try and do what else we can do with a setup like this one. So, since we have a full-fledged automation system, we can test deep linking, sharing, like authentication flows, as I told you before. We can run the device 24-7 on different devices, iPads based on the actual base that we can So, we can scale up and down the robot as we want to. We can test post notifications and interactions with the operating system.
But there is also more. So, for example, we can attach a camera and have a live feed in order to detect what the robot touches or swipes. We can get really meaningful metrics, like how much time does our application take to load. This is a really interesting one, because we can take metrics about user behavior or analytics. And because the coordinates and the movement is based on matrices of numbers through time, we can effectively replicate bug scenarios or try to go through workflows. We can also stress test the application, like fiercely tapping the screen and so on. And finally, since the movements can get described really well, we can train our own AI models and we can auto-generate user paths and workflows in order to provide automation testing as well.
So, that's all from me now. I would like to thank you so much for your time and patience.
Questions and Testing
And I hope you enjoyed the conference so far. Let's talk about some of these questions. The next question asks about Axis and responsive design. If it's a larger device, updating the test is not necessary because we can interact with the web interface and easily accommodate different designs. Tabletop Robotics has created a base that can rotate and tilt the device for testing the gyroscope. Testing push notifications can be done by interacting with a web interface and using locators.
And I hope you enjoyed the conference so far. Let's give him a round of applause. Let's see the questions.
Before we do the questions, I have a confession to make. I feel really bad about this as well. I am Chatterbox Kodar, and I realized halfway through that I was like, oh, wait, if you just quickly go through questions and you get them wrong, it still speeds up the next question. So, I was just doing maths on the first two digits and carrying one and seeing if it worked. So, I think Rahul should get the prize. So, wherever Rahul is, wave your hands. Give him a round of applause wherever they are, and if they're at home, give Theodore a DM.
All right. That's pretty fair for you, right? Yeah. Okay, so let's talk about some of these questions. So, we'll save the first question until last, but the next question which asks about Axis. If it's a larger device with responsive... I will start that again. If it's a larger device with responsive design, will you need to update the test? Fairly not, because since we're... So for my example, I'm interacting with a web interface, so you can actually place locators and you can identify the coordinates, where's everything. Also, when the device boots up, we get information about the viewport, about the actual device screen, so it's pretty easy to accommodate different designs as well. Tabletop Robotics have also created a base that can actually rotate the device also and tilt it in order to identify, to test the gyroscope and things like that. So no.
Ah, well, thank you. I think that's so interesting how you start with a really simple thing, you're like, oh, can it do this? And you add on the features. It's just like a real product you're developing. That's really cool. All right. I'm going to scroll down to the next question. We will go through some of the cheeky ones later, don't worry, I'll save them for later. But how do you test push notifications? Can we make the robot know that a push notification has been received, etc? Sure. So, in this example, I'm just interacting with a web interface, like placing locators and so on.
Camera Attachment and Robot Speed
TabSir has actually provided a way that you can attach a camera in order to identify what's on the screen and also use a toggle to screencast the results. This allows for image comparison and the ability to detect notifications or receive callbacks when push notifications appear. It's a unique and impressive use of the camera.
Another question is about the speed of the robot and how it affects the duration of CI pipelines compared to a software-only solution. The speed of the robot depends on the movement and precise calibration. It can move quite fast for concurrent taps in different places, as demonstrated by TabSir. They even have a demo of playing Tappy Bird, where the robot taps the screen quickly. It's impressive that the robot can capture images and move faster than most humans.
Multi-touch, Future Roadmap, and Camera Sensors
We've received questions about multi-touch, the future product roadmap, and implementing camera sensors. The current method of touch input involves using a stylus that can be attached to the microcontroller. Additional features, such as swiping, can be achieved by attaching multiple styluses. When implementing camera sensors, the approach depends on the desired outcome. For mobile applications, events alone cannot be used for assertions. Image comparison and locators are alternative methods. The game played during the demonstration utilized a web interface and WebSocket communication. The flexibility of using familiar technologies and APIs makes this setup accessible. The approximate cost of the setup is around $40 to $50, depending on the availability of a 3D printer.
We've got a bunch of different questions that all really are the same thing, which is about multi-touch. So, zoom, pinching, and different things. What's the future product roadmap, I guess? That's the actual question.
Right now, in order to touch the screen, we have a stylus. So basically, it's a phone pen, the cheap one, like $1 or so, that's grounded to the microcontroller. Basically, you can attach whatever you want to. If you attach a pen, that's a plotter. If you attach a hotend, that's a 3D printer. So, basically, you can attach a small... Like two fingers, like two styluses in order to create swipes and so on. It's based on your imagination and what you are capable of doing so.
Affordability, Future Automation, and Rapid Fire
All the components are widely available and affordable. The price is around 40 to 50 bucks. Tapster offers a setup using a customized toggle to control the phone using accessibility settings. In the future, automation of test case updates may depend on the setup and the ability to identify dynamic interactions. It's an 'it depends' situation, so feel free to have a chat with the speaker. Now, let's move on to a rapid fire of yes or no answers.
But all the components are widely available. Like the servo motors are pretty plain, old-school motors for remote-controlled cars. So yeah, it's pretty affordable if you want to do so, if you want to build something like that.
Oh, cool. Because you kind of preempted me on the next question. The next question was what's the price. Yeah, it's more or less 40 or 50 bucks. The cool thing is that right now we have more ways of doing so. For example, Tapster, they were working on a setup where they have actually used a customized toggle that can control the phone using the accessibility settings. So they're tricking the phone, thinking that this is a keyboard and a mouse, so you can even run a similar setup with an Arduino or a Raspberry Pi for way less, I think more or less five bucks or so. So yeah, definitely check them out.
Now that is really, really awesome. And then last one of the serious questions before we get to the fun ones that I know everybody is waiting for. So this one's about if the application UI changes every time you're going to need to update your test cases and do it all quite manually, I'm guessing at this point in time. I mean, is there a future you envision where it could be easier to kind of automate changes to the tests? I think that relies on the setup you actually have. So as I told before about the responsive design, if you... So for example, if you're running tests on the web interface, like using Cypress or Playwriter, so you can actually say, hey, go and click to this position of the screen. That's pretty static. But if you somehow have a dynamic way of identifying the interactions that you want to make, that makes pretty much... Makes the whole setup more flexible. Awesome. Awesome. And one thing I think actually is, and kind of like this answer, everything's an it depends. And honestly, what I would say is afterwards, come find him and have a chat. I'm guessing you'd love to talk about it, considering you've come on stage and chatted about it with all of us. So thank you. Thank you so much. All right. Now what we're going to do is we are going to jump into a rapid fire of yes or no answers for the next question. Is that okay? Yeah, that's fair.
Q&A - Skynet, Robot Capture, Tinder Source Code
Will Skynet kill us in 10 years? Yes. Can the robot solve the I am not a robot capture? Yes, it's pretty amazing. Have you used the robot for Tinder? No. Can you share the source code for Tinder? Yes, it's open source on the Tapster GitHub repo.
That's fair. Okay. That's pretty exciting. All right. So we'll start with the first one. Yes or no. Will Skynet kill us? In 10 years or so? Yes or no? Yes. In 10 years. Cool. Nice.
Next one. Can it solve the I am not a robot capture? Actually, yes. We have tried this one and it's pretty amazing because they do have the bug there. So for mobile devices it's pretty easy to do so. Fair enough. It's ironic, right?
All right. Next one. Yes or no. No explanations. Have you used the robot for Tinder as well? No. No. Okay. Well, I guess the answer is obvious on the next one. Can you share the source code to use it in Tinder? Asking for a friend. Wink. Emphasis on the wink. The source code is actually open source. It's on the Tapster GitHub repo for Tapster one and two. It includes all everything that you need to build like a bill of materials, a code and so on. And also the 3D designs that you can go to a makerspace and 3D print the entire setup.
Robot Programming and Connectivity
Don't worry, you've got it covered. If the robot had won, would you program it to read a Node.js book? The machine is connected over Wi-Fi, allowing the local setup to be exposed to the robot. This presentation was amazing. Thank you so much!
So whoever you are, don't worry, you've got it covered. You just need to do a little bit of work yourself.
All right. And next one. If the robot would have won, would you have programmed it to read a Node.js book? Once again? If the robot would have won, would you program it to read the book? I cannot answer this one. Like yeah. Hopefully it just never wins and you don't have to face the problem. Right.
Okay. So this is about the actual, the board, the machine. Is it connected over Wi-Fi? How is it connected? Yeah. So basically I'm connected through, I'm on the Wi-Fi network and both of them are connected to the same Wi-Fi so I can expose like my local setup to the robot, to the mobile phone that's on. That's awesome.
And the last question. It's not a question, but it's one thing that I think we all agree with. This presentation was amazing. You should keep it up. We want to see you again. Give him a massive round of applause. Thank you so much. Thank you.