Teaching ML and AI to Coders

Bookmark

Often it's thought that to be able to succeed with Machine Learning and Deep Learning, as an onramp to Artificial Intelligence, that you need a deep background in mathematics and calculus, as well as some form of PhD. But you don't. With modern APIs like TensorFlow, much of the complexity is abstracted away in pre-built libraries, so you can focus on learning. In this session, Laurence Moroney, from Google, will explain how he has used this to create courses with hundreds of thousands of students, and from there, how a certificate program was created.


Transcript


About Laurence


[00:24] Hi everybody. I'm Lawrence Moroni and I lead AI advocacy here at Google, and it's really my job to educate the world around AI and to try and make the world a better place through AI.

A little bit about me is I'm also the author of this book, AI and machine learning for coders. It was just released. So it's quite a new release and it was actually the number one bestseller and a number of AI categories on it's week of release on Amazon. And if you're interested in the book on Amazon, this is the URL and the book itself is actually based very much on the syllabus that I'm going to be talking about during this talk.


A little bit about AI


[00:59] So, first of all, I want to talk a little bit about AI and where AI is at and this curve I like to use, and this is the Gartner's life cycle curve.

And so the lifecycle curve of any technology usually begins with the technology being introduced, and then it reaches this peak of inflated expectations. And the peak of inflated expectations is the kind of thing that when you see massive hype around the technology, but that hype isn't really based around anything real on the technology. It's based on speculation on the technology itself. And then often the life cycle curve drops us into the trough of disillusionment. And despite the negative sounding name, it's actually a very positive thing because that's when we blow through the inflated expectations, we blow through the hype and we understand what the product and what the technology is really all about. And then from once we reached that point from there on, we can reach productivity.

[01:49] Unfortunately, the AI right now is probably somewhere about here on the car of this still many inflated expectations. And just to call this out sometimes. So it's inflated expectations can be positive where everybody's thinking about like the amazing things that can be done with the technology. And sometimes it can be negative where people are terrified and they're afraid of the technology, but their expectations about its capabilities are what are inflated because of this hype cycle curve.

But my job is generally, I'm trying to get people down here into the trough of disillusionment. And I sometimes joke that I'm a, I'm a professional disillusion or but really what the idea of having the world understand the, what AI really is all about, what you can do with AI, how you can build with AI and from there, then you can grow up into the productivity. So, like I said, we're here right now. And my question then becomes, why do you think we're here right now?

[02:40] Well, there's a number of reasons behind this. The first one I will show is this number and this number is 300,000 and 300,000 is the number of AI practitioners in the world, according to a survey done a company in China. And so they wanted to just take a look at why is this such a global skill shortage around AI? This was about two and a half years ago. Why was this such a global skill shortage around them? And they wanted to say, well, how many qualified people are out there? And it was 300,000 practitioners.

Now, I like to compare this with this number, which is 30 million. And there were a number of estimates out there as to the number of software developers globally. And they vary wildly. I've seen some around 22 million, I've seen some around 35 million. I'm going to like just picking a number roughly in the middle of that. And it was 30 million. And I could argue actually that the number is far greater than this. So for example, at WWDC this year Tim Cook mentioned that there were 28 million developers in the Apple ecosystem alone. So if I did a rule of thumb that half of the developers in the world are in the Apple ecosystem, we could be actually closer to 60 million developers globally.


Google's new mission: Make AI Easy


Well, let's, let's work with this number of 30 million. And I remember there were 300,000 day practitioners, according to the survey, there were 30 million software developers, according to my estimate. So I made it our vision at Google as a, what if we could train 10% of the world's developers to be effective in machine learning and in artificial intelligence. And if we did that, we'd have 3 million AI and ML developers, which is 10 times this number. So we made that our goal, can we increase the number of AI practitioners globally 10, a factor of 10, not a number of 10.

[04:27] So, you know, we said, we'd set them, we'll make this our goal. We started this journey about 18 months ago, a little over 18 months ago. And today I just want to share like the strategies that we use and the results that we've gotten.

But first of all, when working with software developers and when I talk with them and when I would look at how they were being trained, I got a lot of feedback around why they thought AI was difficult and why AI was something that was while it was something that was of interest to them. It was something that it was going to be too difficult for them to kind of give up a lot of their time and a lot of their study time to be able to pick it up. And I started seeing like a lot of terminology, like I've put in this chart here, people saying it was difficult.

[05:11] There was a lot of math. There were a lot of terms that they weren't familiar with, like unsupervised learning or supervised learning. They really like me. They hadn't done things like calculus and probability in 25 years. And as a result, the number of concepts that were being thrown at them just to get started made it like there was a massive road bump that you'd have to get across to be able to get started, to be able to transform your career and transform your skillset to be a machine learning or an AI developer.

So I sell that as a challenge. And one of the things that we do with Google is that we give ourselves a mission and our mission is defined three words. And so the three words that I used were Make AI Easy. And so I've made that my mission at Google is really, it's all about making AI easy.

[05:59] And I skipped one slide here for a moment. And then that led me to this overall mission that I wanted to be able to train millions of developers, to reach billions of people on going back to the life cycle curve and the hype, the hype cycle curve that I spoke about earlier on, if we can train millions of developers. And I think millions of developers are the key to be able to reach in billions of people with AI solutions and with AI applications that will burst us through the bubble that will burst us through the hype bubble so that we can see what's real in AI. So we'll make AI easy.


The strategy


So some of the strategy around this was that the first goal was to come up with a syllabus of material and a set of content that was aimed at developers. So it was a developer Me, and it was aimed at developers who are coders.

[06:46] You don't need a PhD, didn't need math, any of these kinds of things. I don't have a PhD. And like I mentioned before, I haven't done calculus and probability and statistics in 25 years. So I wanted to come up with a syllabus that was a code or, and it was for coders. And then to build a strategy around the syllabus and the strategy is twofold.


1. Professional's strategy


First of all, is the professional's strategy. So this is the strategy aimed at educating people who are already developers, people who are already in careers as coders. And the folks in that domain tend to be reached in three different ways. Those massively online open courses or MOOCs, there's being able to do it direct and at scale, and then there's managed to training. So let me talk a bit, a little bit about each of these.

So MOOCs we partnered with deep learning.ai and Andrew Ang, and the great folks at deeplearning.ai to produce three specializations.

[07:36] We've released two of them: TensorFlow: in practice, TensorFlow: data and deployment. And I'm currently hard at work on the third one, which is going to be released in November and December. And we're calling that TensorFlow: from basics to mastery.

And the idea behind this one was to produce 12 courses in three specializations that would take somebody from no knowledge at all, except maybe a little bit of Python programming to be able to intense flow and practice, to be able to meet the common scenarios that are needed in machine learning careers and that's natural language processing, computer vision, and sequence modeling.

The second specialization data and deployment is then taking the work that you've done and building machine learning models and making them real putting them into mobile apps, putting them into websites, using them in JavaScript and things like that.

[08:20] And then the final one TensorFlow from basics to mastery is kind of going back a little bit to the model creation from TensorFlow in practice, but going deeper learning how to extend your models, learning how to create custom. We have things in models called layers, learning how to create custom ones, and then looking at some of the advanced algorithms and being able to break them down so that you can understand them a little bit better. And that one I'm afraid. I broke my promise of not doing math cause we had to do math a little bit in that. But we tried to make it as approachable as possible.

Direct, at scale is then we launched the TensorFlow YouTube channel and we wanted to make that the go-to place where people could have short, quick lessons either about how to do something in TensorFlow or what is happening in TensorFlow, new releases, all of those kinds of things.

[09:10] We also put together a number of hands-on code labs. Again, aimed at people, you don't need a PhD on math. You sit down and you start coding and you start building out concepts. And it's in line with the three pillars that I spoke about earlier on that we saw as what employers were looking for and that's computer vision, natural language processing and sequence modeling. And then we've also been working through a number of different Google training initiatives. And I created a series for Google training. And I created another series as free training and when people were at home and lockdown and those kinds of things called ML foundations and all of that was being done direct at scale via YouTube and other Google channels. And then then finally, sorry, I just before I go into the next one, and just as an example of how the model of being developer focused has been successful Carmel, Alison and I did a talk called machine learning zero to hero at Google IO 2019 to test it out with the developer audience.

[10:10] So I I'm trying to break the machine learning audience into two. There are other people who are expert at machine learning and AI, and then there are other software developers who want to use machine learning and AI, and my focus has been on the second group. So Google IO is the audience for them. And we did a talk at Google IO that we called machine learning zero to hero. So take somebody in about 40 minutes to go from basic Python knowledge, to be able to do some pretty complex computer vision. In this case, in the screenshots, I was teaching them how to do a computer vision exercise that recognizes rock paper scissors. But again, when it was done, very developer focused, one of the nice things we've learned about this one is the viewership of this video, where we're approaching 1 million views on it. And when it comes to talks at Google IO, this was far and away the most successful from a YouTube perspective. So we realized that we have resonance with developers with this type of message. 

[11:07] And then the third one I mentioned is managed. And so this is where we have high touch managed efforts to train developers. And two of them that I show on this slide here one of them is an academy in Indonesia called Bang Kits. And Bang Kit was an exclusive machine learning learning academy led Google working with the government and working with various startups that we brought in 300 people. It was application based. We train them in machine learning and the Indonesian government really wants to be a part of this so that they could start seeding their startup ecosystem with people who are qualified and people who'd know the technology.

[11:42] And then the one at the bottom as the Google developers machine learning bootcamp is running in Korea. It actually started right at the end of October. And the idea behind this one is that Google and a number of companies and many of whom are startups got together to sponsor this bootcamp, but to make it a very different boot camp. And that is the cohort of people coming into this bootcamp. If they pass the content in the bootcamp, they have a guaranteed job. I'm particularly excited about this one, given the economic situation at the moment because of the pandemic, that to be able to work together with industry, to create a pipeline of people who are qualified to do a job, but have the motivation for those folks to go through this bootcamp will have a guaranteed job or internship at the end of it. I find it wonderful. 

[12:28] And all of this leads to what we call the employment oriented developer certificate. So we've created a rigorous certificate exam. And this certificate exam is that once you pass that, you're proving that you can be a developer with computer vision, natural language processing and sequence modeling. And this is also part of what we had with the Bang Kit academy and with the academy that I mentioned in Korea, the bootcamp I mentioned in Korea.


2. Academia strategy 


The second half of this is to work with academia so to work with universities. And so the number one thing that I, when I speak with professors at universities that I hear is that they would love to teach new courses. But every time you need to add a new undergraduate course, you have to get rid of an old undergraduate course because there's only so many that they can teach. And there's a great risk involved in teaching new courses. There's there's a lot of time involved in developing the curriculum and training teaching assistants and all of those kinds of things. And as a result of tends to be a lot of friction about adding new courses. Those of us who have graduated with a computer science, I have half of a computer science degree and I did half physics, half computer science, but I recall when I graduated the skills that I learned to graduate, where the skills of maybe four or five years prior to my graduation date and the languages and frameworks that I needed to get a job on the date, I graduated just weren't taught in academia. So we wanted to help try and solve that problem with machine learning.

[13:56] So we created a program where we help universities in developing courses. We provide them with the same curriculum that's used the MOOCs, the same curriculum code, first oriented curriculum that like we use in our own trainings and all of that kind of stuff, what supports, and in some cases, some financial assistance to, for example, hire TAs and that kind of thing. And so we've launched that with a number of different universities. And again, the idea was to drive this towards an employment oriented developer certificate and the same one as the professional folks had.


Results. Global Impact of Education Syllabus


So some of the results I'd like to share about this and the global impact of this.

So first of all, the MOOCs Coursera and deep learning AI and the two speciality, the two specializations that we launched as of now, when I recording this, we're over 600,000 learners and it's, it's growing exponentially and this new specializations launching next month we think we're going to be getting, you know hopefully above a million learners sometime next year.

[14:51] One thing that the World Economic Forum did a jobs report about the jobs of the future and the jobs of the fourth industrial revolution that AI has been called. And they actually cited these MOOCs as a necessary skill. They listed a number of them and the top two where Andrew Ang is deep learning specialization, which I learned from with terrific specialization and my specialization that tends to flow in practice. One, the ones that we created with each other. So, you know, that it shows that it's landing well from an employability perspective, even recognized the World Economic Forum jobs report. Udacity, also a MOOC, they launched an introduction to TensorFlow and to TensorFlow light courses. And they like coming close to 200,000 earners. In China the syllabus was used to create a course with company called NetEase and this launched just last week or middle of October, the time I'm filming this they have around 3000 learners already. So it's we see that it's really beginning to pick up and Udemy will be launching some courses quite soon.

[15:53] So again, the syllabus it's being used multiple folks multiple providers, and we're seeing really nice numbers. With universities we've had dozens of universities globally that we've assisted in teaching the TensorFlow syllabus as I described earlier on, but I want to call out a couple of highlights. Imperial college in London use this to create their teaching syllabus, but they also use this to create an online massively online open course of their own that they teach on Coursera, which I found really exciting. One of the things we're really committed to with this as widening access as much as possible. So I've been talking about you don't just need a PhD and things like that, but we also want to widen access to people who are traditionally not represented in AI.

[16:37] Keio university in Japan have launched a course for women. And we know that, you know, there was a representation of problem in tech, a diversity problem in tech with having women in tech and in Japan, it's particularly sharp because of some historic issues there. So Keio university really wants to see if they could bring in more women into AI and they are just launching a right around now, they're launching this course for women being taught women. And again, they became part of our university funding, the program that we set up, and this is one of the results that came out of it, and then given the issues globally and what we've been learning globally about racial issues. And no, the fact that black lives really do matter. One of the things that we've been trying to work very hard on is to work with historically black colleges and universities in the USA to help them, you know, gain the benefits of the the gifts that we can have with AI and to be able to take advantage of what the World Economic Forum was showing about the growth of jobs in AI, and be able to train their students for that.

[17:42] Earlier I mentioned the certificate program. We launched that in the middle of 2020, and sorry, already 2020, spring of 2020. And it's an employment oriented certificate, a rigorous exam covering NLP, computer vision, and sequence modeling. And as of now, we have about 1200 holders globally, and it is increasing nicely. And the vision here is to really help people show their skills, what to help employers know that these skills have been shown. One massive part of the jobs gap was we discovered that a lot of employers didn't know how to find qualified AI people and didn't know the right questions to ask them. So we created the certificate program with that in mind.


Impact. State of Employment Globally


[18:25] And then just to wrap up I alluded to it a little bit earlier on, but part of the impact on this one is, you know, we really want to help seed the economy. We really want to help train developers to be successful in the new economy. I don't want to share some statistics about the state of employment globally.

First of all, this chart this is a genuine, nice upwards trend chart, but there's a big dip at the bottom. And this is the chart of GDP of averaged out across several countries UK, USA, and others. And you can see the general trend over about the past 35 years is that it's been mostly upwards. There's been a couple of little dips, but the problem when you chart something like this, as you see a general upward trend and you think everything is fine, and you might miss some of the important nuances in this. So I rechartered it as a derivative chart where instead of just showing the overall growth, I wanted to show the growth in any given year relative to the previous two years.

[19:17] And as a result, the same 35 year charts showed this. This to me is really interesting because there are a couple of big peaks in this chart. One of them is here and this one, I don't have the years on this one, but this one was from about 1993 through about 2007. And this was the how the growth of the web and how the emergence of the web and the tech economy really changed the world economically. And this massive period of sustained growth. And it's no surprise what are now the biggest companies in the world market cap are all companies that came as a result of this growth, the Microsofts, the Apples, the Googles, all of those tech companies that are now giants came from this period of growth.

And the second one here is the app economy. This is really the growth that came about as a result of the emergence of the smartphone. First introduced, I would say you could argue, Apple with the launch of the iPhone in 2007. I know there were other smartphones before that, but this economy where they picked up, then, then the emergence and growth of Android and other platforms. And we've seen this as a really, really huge growth area on wireless.

A big dip on the right, because of COVID. And because of the pandemic, we just want to realize that even like Apple, this year released a statistic that said in 2020, the app ecosystem has grown 300,000 jobs in the U. S. alone. So we can see that jobs and employment are still very, very hot because of these two peaks. And th th the next peak, maybe in 10 years time, if you look at a graph like this one, there'll be a new peak, and that's the fourth industrial revolution and the economy that comes as a result of AI.

[20:53] The World Economic Forum, for example, a data report where they wants to look at the jobs of tomorrow, starting in 2020 and ending in 2022. And they had five categories, data and AI engineering and cloud people in culture, product development, sales, and marketing, and far and away. The biggest as you can see here was about a 37% growth in jobs between 2020 and 2022 in data and AI and engineering and cloud computing, and C was plus 34%. A lot of that was around ML stuff in clouds out also. So we can see it's a huge potential growth thing for jobs and unemployment.

There's also a Forbes reports. And I put the link to it here that showed the global ML market will have a compound annual growth rate of about 44% between 2017 and 2024 for 1.5 billion to 20.8 billion. So it is a huge growth area and AI software revenue. They also reported between 2018 and 2025 from 10 to 126 billion, which is a category of 43%. So we can see here that there's definitely massive, massive growth in this area coming.

[21:58] You know, they're all, all the reports that I see tend to put it as a compound annual growth rate in the forties, both of these 43 and 44% support that. And even they, they mind LinkedIn to take a look at jobs and 44,000 jobs in the USA required some kind of ML and almost a hundred thousand globally required some kind of ML. So we can see ML, AI, these kinds of skills are massively, massively important as we go forward. And we wanted to be the people to help train millions of developers to reach billions of people.

And with that, I just want to say thank you. And I'll be able to answer any questions. Yeah. Thank you so much. Thank you.


Questions


[22:41] Laurence Moroney: Can I answer that question with all of the above?

Ajay Halthor: Yes, please. Do you, do you have an answer right now? Let's do it.

Laurence Moroney: Yeah. So where I want to deploy my models. Absolutely. Everywhere,

Ajay Halthor: Everywhere. You know what? Yeah. Everywhere. It's like all the options, what's the correct option. Everything, everything models everywhere. That's awesome. Well, right now let's actually start with some amazing questions because we kind of have some amazing questions right here. So actually this is a Hossein. I hope I'm pronouncing your name right. But which do you think is a better option for the industry TensorFlow or PI torch?

[23:24] Laurence Moroney: Well, first of all, I'll start saying I'm completely biased given that I work on the TensorFlow team, I'm wearing a TensorFlow jacket and I'm drinking coffee from a TensorFlow mug. So I think my answer there is going to be TensorFlow. And but so if I take off my jacket and take off my mug, I think it's always good for everybody to learn as much as possible about you know, we welcome the great competition that we're getting with PI torch, I think is brilliant for the industry to have competition amongst the vendors. But I will answer that, you know, despite that, I still think TensorFlow, just because of the number of different places that you can deploy TensorFlow models, right. From the massive cloud-scale infrastructure with multi-node TPS all the way down to the tiny microcontrollers and embedded systems that your TensorFlow skills, regardless of what type of model you want to produce, there's a place for you to put it. And so, like from the industry perspective, I think, you know, I would strongly recommend despite my jacket, I would still strongly recommend TensorFlow.

[24:27] Ajay Halthor: Yeah, definitely. I do know that, you know, PI torch does abstract a lot of well hidden mechanisms that TensorFlow does provide. And like, while on one it's kind of a trade off because on one side, TensorFlow is definitely more adaptive point controllable and flexible. Whereas with PI towards some people just might be an easier interface to just use in general.

[24:47] Laurence Moroney: Yep. I, I won't claim any kind of expertise in PI it's, it's something that I need to learn more of to be perfectly frank. But yeah, my focus is always been on enabling people to succeed with the technology and, you know, it's one level is being able to build models, but the other level and more important level is having somewhere to deploy them. And I think, you know, that deployment surface is a massive, massive strength of TensorFlow. And it's, you know, where I think we distinguish ourselves across in the ML and that's why I'm particularly passionate. 

[25:20] Ajay Halthor: Yeah, that's a great answer. And also, here's another question from Arjun Kumar. How do you recommend continuing to build your career after, you know, completely like courses like yours or Andrew's for example?

[25:37] Laurence Moroney: Yeah. Good question. I think practice, practice practice is very important. I think like with any kind of coding, the more that you do it, the better you get at it, you begin to realize the pitfalls, you begin to realize like those tips, those tricks, those things that make you better at doing it. So ML is no exception in that regard. It's really practice, practice, practice. It doesn't have to be public with competitions on Kaggle or anything like that. And one thing that I always recommend is you find something that you're passionate about, and then there's gotta be some data out there around something that you're passionate about and start building models in that regard and, you know, just continue to practice and become a better model creator. And then once you have those models that to practice, practice, practice, then in deploying those models, be it like a large scale global scalable infrastructure is, or maybe just the single cell phone for yourself to use, but you know, the key with anything is, you know, constantly practice to improve yourself.

[26:34] Ajay Halthor: Yeah. That's, that's true. You know, I've also about practicing too and everything that, that is associated with this too. Like there are also like a number of people I'm also like looking at a bunch of questions where there is that mathematical aspect that is deep learning and machine learning. Whereas, you know, TensorFlow is more of like, okay, let's kind of break things apart from a more practical aspect and then, you know, dig deep into that way. So a question would be like, you know, how do you kind of balance these two, you know, these two completely different notions of approaching machine learning and deep learning, would you prefer one or the other or somehow like a symbiotic relationship between like theory and practical applications when going on your machine learning journey?

[27:18] Laurence Moroney: I like the term that you just use symbiotic because I think that's exactly right, but symbiotic, like how do you start? And so they should be meshed together, but how do you get started? And I mean, I can say just for me personally, I hadn't done calculus in 30 years, but I started looking into modern machine learning. And when the initial material that I was looking at was kind of teaching you how an optimize it works through gradient descent and tons of calculus, or, you know, teaching you how a convolution works using mathematical and Greek notation. I'll admit I was lost. It was harder for me to follow it. But when it was I started looking into this, like using code I'm primarily code it's much easier for me to read and understand code. So then my symbiotic relationship began definitely on the practical coding side of it, so that now I can understand what a model is.

Now I can understand how a model learns from parameters. Now I can see how backpropagation works and things like that. And then with all of that context around it, then to start going into the math and then seeing, ah, now I see why I'm using calculus for gradient, the sands and all of those kinds of things. So to really build that context around it so that I could understand it better. And that's why now it's very much that symbiotic relationship that, you know, I could start building models without knowing any math. But once I started building models, if I wanted to build better models, then to start understanding what's going under the hood was necessary.

[28:46] Ajay Halthor: Yeah. That's completely true because like, I know there's a lot of people who are in school, especially like, based on our poles and you know, maybe like in school you'll be like learning some very complex mathematics, like, like calculus, for example, like that you mentioned. And you're like, well, if you're just, if it's just thrust at you, it's like, okay, but where do I use this? What am I going to do with it? I don't care. As long as I just pass this course, I'm good. Then when you're like, but I am interested in machine learning and deep learning. Hmm. What do I need to do for that incoming calculus? Just surreptitiously it's like hi again. And then you're like, oh, so this is how you can use calculus to do, to create like all of these machine learning, you know, projects that, you know, you see everywhere just being deployed. It's like, oh my God, there's self-driving cars. There's like this system that can detect your fraud, whether your activity is fraudulent. Yeah. Just like the, the applications are endless and all with calculus. So that's definitely very interesting to hear.

[29:47] Laurence Moroney: Yeah. But it's like, don't let that be. That don't let that be the gatekeeper that prevents you from getting in that they feel like me that once you start reading calculus and your mind, your brain goes to sleep, you know, that kind of stuff, you can really get started and start doing all of this stuff and be productive before you start getting into the math. But it's like, once you're at that productive level, then it becomes easier to get into math because you build that context around it and it makes it much easier for you to do so. Sorry, you said you had another question.

[30:13] Ajay Halthor: Never. We always have so many questions for you. They just keep pouring in. So you've seen TensorFlow, just like, you know, build, just become this huge thing right in front of you. But do you have any reservations about TensorFlow? Like maybe somewhere along the way TensorFlow could have done something better and it's probably still something that's a big problem. Is there anything like that?

[30:36] Laurence Moroney: I don't think there's anything that I would consider to be a big problem now. I think one thing that would have been really nice to have earlier, but we then eventually had with TensorFlow too, was to think about it in a more polyphonic way. Because early versions of TensorFlow were very much graph oriented, so you'd load all your stuff into a graph. And then you execute like a session within a graph. And for me as a coder, as opposed to an AI person that was a little bit alien to begin with, it was very hard to like, cause you can't do step--step debugging and things like that. When we released TensorFlow who became like with eager default became more polyphonic. So it made it much easier for the people who are coders to get up and running. And if there was any one thing that I had at a time machine that I would change to do sooner, it would be that.

[31:27] Ajay Halthor: That's very fascinating. And I think we have a final question over here. So I so there are people with like some experience in JavaScript and Python and they always like asking, so, which is the best language to start my machine learning journey. Do you have any specific thoughts on that?

[31:48] Laurence Moroney: Yeah, I think the best language to start your machine learning journey of those two is the one that you're most passionate about using. I know a lot of people that love JavaScript. I know a lot of people who hate JavaScript and I know a lot of people that love Python. I know a lot of people who hate it. If you don't have any particular affinity to either of them, my recommendation would be to start with Python, just because of the number of support libraries that are out there for the beyond machine learning, part of it, the data science part of it, they're drawing graphs and all of that kind of thing. So like I said, if you were going to pick one from scratch, I would definitely say Python, but there's certainly nothing wrong with Javascript. The things you can do with TensorFlow JS are fabulous. And yeah, I think Jason's going to be talking about that at this conference later today. And he always shows amazing things.

[32:30] Ajay Halthor: Exactly. I'm completely looking forward to that too. By the way, Laurence, I think we are out of time for some questions, but thank you so much for all of our questions until now and for our lovely audience here. Of course, we're always happy to have you. And if you guys, you, anyone here just did not want to get a chance to ask Laurence your questions or they didn't come up right now, feel free to reach out to him in his speaker room, even starting like right now after this call. So just be there, Laurence, it was great having you right here and we look forward to hear from you soon.

[33:07] Laurence Moroney: Yep. Sounds good. Thanks. And I'll see you all in the speaker room.



Transcription


Hi, everybody. I'm Lawrence Moroney, and I lead AI advocacy here at Google. And it's really my job to educate the world around AI and to try and make the world a better place through AI. A little bit about me is I'm also the author of this book, AI and machine learning for Coders. It was just released, so it's quite a new release. And it was actually the number one bestseller in a number of AI categories in its week of release on Amazon. And if you're interested in the book on Amazon, this is the URL. And the book itself is actually based very much on the syllabus that I'm going to be talking about during this talk. So first of all, I want to talk a little bit about AI and where AI is at. And this curve I like to use, and this is the Gartner's life cycle curve. And the life cycle curve of any technology usually begins with the technology being introduced, and then it reaches this peak of inflated expectations. And the peak of inflated expectations is the kind of thing that when you see massive hype around the technology, but that hype isn't really based around anything real on the technology. It's based around speculation on the technology itself. And then often the life cycle curve drops us into the trough of disillusionment. And despite the negative sounding name, it's actually a very positive thing because that's when we blow through the inflated expectations, we blow through the hype, and we understand what the product and what the technology is really all about. And then once we reach that point, from there on, we can reach productivity. Unfortunately, AI right now is probably somewhere about here on the curve. There's still many inflated expectations, and just to call this out, sometimes those inflated expectations can be positive, where everybody's thinking about the amazing things that can be done with the technology. And sometimes they can be negative, where people are terrified and they're afraid of the technology, but their expectations about its capabilities are what are inflated because of this hype cycle curve. But my job is generally, I'm trying to get people down here into the trough of disillusionment. And I sometimes joke that I'm a professional disillusioner, but really with the idea of having the world understand what AI really is all about, what you can do with AI, how you can build with AI, and from there, then you can grow up into the productivity. So like I said, we're here right now. And my question then becomes, why do you think we're here right now? Well, there's a number of reasons behind this. The first one I will show is this number, and this number is 300,000. And 300,000 is the number of AI practitioners in the world, according to a survey done by a company in China. And so they wanted to just take a look at why is there such a global skills shortage around AI? This was about two and a half years ago. Why was there such a global skills shortage around them? And they wanted to say, well, how many qualified people are out there, and it was 300,000 AI practitioners. Now, I like to compare this with this number, which is 30 million. And there are a number of estimates out there as to the number of software developers globally. And they vary wildly. I've seen some around 22 million. I've seen some around 35 million. I'm just picking a number roughly in the middle of that, and it was 30 million. And I could argue, actually, that the number is far greater than this. So for example, at WWDC this year, Tim Cook mentioned that there are 28 million developers in the Apple ecosystem alone. So if I did a rule of thumb that half of the developers in the world are in the Apple ecosystem, we could be actually closer to 60 million developers globally. Well, let's work with this number of 30 million. Now, remember, there are 300,000 AI practitioners, according to the survey. There are 30 million software developers, according to my estimate. So I made it our vision at Google is that what if we could train 10% of the world's developers to be effective in machine learning and in artificial intelligence? And if we did that, we'd have 3 million AI and ML developers, which is 10 times this number. So we made that our goal. Can we increase the number of AI practitioners globally by 10? By a factor of 10, not by a number of 10. So we said, we'd set that, and we'd make this our goal. We started this journey about 18 months ago, a little over 18 months ago. And today, I just want to share the strategies that we used and the results that we've gotten. But first of all, when working with software developers, and when I talk with them, and when I would look at how they were being trained, I got a lot of feedback around why they thought AI was difficult and why AI was something that was... While it was something that was of interest to them, it was something that it was going to be too difficult for them to give up a lot of their time and a lot of their study time to be able to pick it up. And I started seeing a lot of terminology, like I've put in this chart here, people saying it was difficult. There was a lot of math. There were a lot of terms that they weren't familiar with, like unsupervised learning or supervised learning. They really, like me, they hadn't done things like calculus and probability in 25 years. And as a result, the number of concepts that were being thrown at them just to get started made it like there was a massive road bump that you'd have to get across to be able to get started, to be able to transform your career and transform your skillset to be a machine learning or an AI developer. So I saw that as a challenge. And one of the things that we do at Google is that we give ourselves a mission, and our mission is defined by three words. And so the three words that I used were, make AI easy. And so I've made that my mission at Google is really, it's all about making AI easy. And I skipped one slide here for a moment. And then that led me to this overall mission that I wanted to be able to train millions of developers to reach billions of people. And going back to the life cycle curve and the hype cycle curve that I spoke about earlier on, if we can train millions of developers, and I think millions of developers are the key to be able to reaching billions of people with AI solutions and with AI applications, that will burst us through the bubble. That will burst us through the hype bubble so that we can see what's real in AI. So we'll make AI easy. So some of the strategy around this was that the first goal was to come up with a syllabus of material and a set of content that was aimed at developers. So it was by a developer, me, and it was aimed at developers who are coders. You don't need a PhD, didn't need math, any of these kinds of things. I don't have a PhD. And like I mentioned before, I haven't done calculus and probability and statistics in 25 years. So I wanted to come up with a syllabus that was by a coder and it was for coders. And then to build a strategy around this syllabus. And the strategy is twofold. First of all, is the professional strategy. So this is the strategy aimed at educating people who are already developers, people already in careers as coders. And folks in that domain tend to be reached in three different ways. There's massively online open courses or MOOCs. There's being able to do it direct and at scale. And then there's managed training. So let me talk a little bit about each of these. So MOOCs, we partnered with deeplearning.ai, Andrew Ang and the great folks at deeplearning.ai to produce three specializations. We've released two of them, tensorflow in practice and tensorflow data and deployment. And I'm currently hard at work on the third one, which is going to be released in November and December. And we're calling that tensorflow from basics to mastery. And the idea behind this one was to produce 12 courses in three specializations that would take somebody from no knowledge at all, except maybe a little bit of Python programming, to be able to, in tensorflow in practice, to be able to meet the common scenarios that are needed in machine learning careers. And that's natural language processing, computer vision, and sequence modeling. The second specialization, data and deployment, is then taking the work that you've done in building machine learning models and making them real by putting them into mobile apps, by putting them into websites, by using them in javascript and things like that. And then the final one, tensorflow from basics to mastery, is kind of going back a little bit to the model creation from tensorflow in practice, but going deeper, learning how to extend your models, learning how to create custom. We have things in models called layers, learning how to create custom ones, and then looking at some of the advanced algorithms and being able to break them down so that you can understand them a little bit better. In that one, I'm afraid I broke my promise of not doing math, because we had to do math a little bit in that. But we tried to make it as approachable as possible. Direct at scale is then we launched the tensorflow YouTube channel. And we wanted to make that the go-to place where people could have short, quick lessons, either about how to do something in tensorflow or what is happening in tensorflow, new releases, all of those kind of things. We also put together a number of hands-on code labs, again, aimed at people. You don't need a PhD on math. You sit down, you start coding, and you start building out concepts. And it's in line with the three pillars that I spoke about earlier on that we saw as what employers were looking for. And that's computer vision, natural language processing, and sequence modeling. And then we've also been working through a number of different Google training initiatives. And I created a series for Google training. And I created another series as free training when people were at home in lockdown and those kind of things called ML Foundations. And all of that was being done direct at scale via YouTube and via other Google channels. And then, oops. And then finally, sorry, just before I go on to the next one, and just as an example of how the model of being developer focused has been successful, Carmel Allison and I did a talk called machine learning Zero to Hero at Google I-O 2019 to test it out with the developer audience. So I'm trying to break the machine learning audience into two. There are the people who are expert at machine learning and AI. And then there are the software developers who want to use machine learning and AI. And my focus has been on the second group. So Google I-O is the audience for them. And we did a talk at Google I-O that we called machine learning Zero to Hero to take somebody in about 40 minutes to go from basic Python knowledge to be able to do some pretty complex computer vision. In this case, in the screenshots, I was teaching them how to do a computer vision exercise that recognizes rock, paper, or scissors. But again, when it was done very developer focused, one of the nice things we've learned about this one is the viewership of this video. We're approaching 1 million views on it. And when it comes to talks at Google I-O, this was far and away the most successful from the YouTube perspective. So we realized that we have resonance with developers with this type of message. And then the third one I mentioned is managed. So this is where we have high touch managed efforts to train developers. And two of them that I show on the slide here, one of them is an academy in Indonesia called Bankit. And Bankit was an exclusive machine learning academy led by Google working with the government and working with various startups that we brought in 300 people. It was application based. We trained them in machine learning. And the Indonesian government really wanted to be a part of this so that they could start seeding their startup ecosystem with people who are qualified and people who know the technology. And then the one at the bottom as the Google Developers machine learning Bootcamp is running in Korea. It actually started right at the end of October. And the idea behind this one is that Google and a number of companies, and many of whom are startups, got together to sponsor this bootcamp, but to make it a very different bootcamp. And that is the cohort of people coming into this bootcamp. If they pass the content in the bootcamp, they have a guaranteed job. I'm particularly excited about this one, given the economic situation at the moment because of the pandemic, that to be able to work together with industry to create a pipeline of people who are qualified to do a job, but to have the motivation for those folks to go through this bootcamp to have a guaranteed job or internship at the end of it, I find wonderful. And all of this leads to what we call the Employment Oriented Developer Certificate. So we've created a rigorous certificate exam. And this certificate exam is that once you pass that, you're proving that you can be a developer with computer vision, natural language processing, and sequence modeling. And this is also part of what we had with the Bank Hit Academy and with the academy that I mentioned in Korea, the bootcamp I mentioned in Korea. The second half of this is to work with academia. So to work with universities. And so the number one thing that I when I speak with professors at universities that I hear is that they would love to teach new courses. But every time you need to add a new undergraduate course, you have to get rid of an old undergraduate course, because there's only so many that they can teach. And there's a great risk involved in teaching new courses. There's a lot of time involved in developing the curriculum, in training, teaching assistants and all of those kind of things. And as a result, there tends to be a lot of friction about adding new courses. And those of us who have graduated with computer science, I have half of a computer science degree. And I did half physics, half computer science. But I recall when I graduated, the skills that I learned to graduate were the skills of maybe four or five years prior to my graduation date. And the languages and frameworks that I needed to get a job on the date I graduated just weren't taught in academia. So we wanted to help try and solve that problem with machine learning. So we created a program where we help universities in developing courses. We provide them with the same curriculum that's used by the MOOCs, the same curriculum, code first oriented curriculum, that like we're using our own trainings and all of that kind of stuff, with support, and in some cases, some financial assistance to, for example, hire TAs and that kind of thing. And so we've launched that with a number of different universities. And again, the idea was to drive this towards an employment oriented developer certificate and the same one as the professional folks had. So some of the results I'd like to share about this and the global impact of this. So first of all, the MOOCs, Coursera and Deep Learning AI, and the two specializations that we launched. As of now, when I'm recording this, we're over 600,000 learners, and it's growing exponentially. And this new specialization is launching next month. We think we're going to be getting, hopefully, above a million learners by sometime next year. And one thing that the World Economic Forum did a jobs report about the jobs of the future and the jobs of the fourth industrial revolution that AI has been called, and they actually cited these MOOCs as a necessary skill. They listed a number of them, and the top two were Andrew Ang's Deep Learning specialization, which I learned from, a terrific specialization, and my specialization, the tensorflow in Practice one, the ones that we created with each other. So it shows that it's landing well from an employability perspective, even recognized by the World Economic Forum jobs report. Udacity, also a MOOC, they launched an introduction to tensorflow and some tensorflow Lite courses, and they're coming close to 200,000 learners. In China, the syllabus was used to create a course with a company called NetEase, and this launched just last week, or middle of October, by the time I'm filming this, and they have around 3,000 learners already. So we see that it's really beginning to pick up, and Udemy are launching some courses quite soon. So again, the syllabus, it's being used by multiple folks, by multiple providers, and we're seeing really nice numbers. With the universities, we've had dozens of universities globally that we've assisted in teaching the tensorflow syllabus, as I described earlier on, but I want to call out a couple of highlights. Imperial College in London used this to create their teaching syllabus, but they also used this to create an online, massively online open course of their own that they teach on Coursera, which I found really exciting. One of the things we're really committed to with this is widening access as much as possible. So I've been talking about you don't just need a PhD and things like that, but we also want to widen access to people who are traditionally not represented in AI. And Keio University in Japan have launched a course for women, and we know that there is a representation problem in tech, a diversity problem in tech with having women in tech, and in Japan it's particularly sharp because of some historic issues there. So Keio University really wanted to see if they could bring in more women into AI, and they are just launching, right around now, they're launching this course for women, being taught by women, and again, they became part of our university funding, the program that we set up, and this is one of the results that came out of it. And then given the issues globally and what we've been learning globally about racial issues and the fact that black lives really do matter, one of the things that we've been trying to work very hard on is to work with historically black colleges and universities in the USA to help them gain the benefits of the gifts that we can have with AI and to be able to take advantage of what the World Economic Forum was showing about the growth of jobs in AI and be able to train their students for that. Earlier I mentioned the certificate program, we launched that in the middle of 2020, sorry, early 2020, spring of 2020, and it's an employment-oriented certificate, a rigorous exam covering nlp, computer vision, and sequence modeling. And as of now, we have about 1,200 holders globally, and it is increasing nicely. And the vision here is to really help people show their skills, but to help employers know that these skills have been shown. One massive part of the jobs gap was we discovered that a lot of employers didn't know how to find qualified AI people and didn't know the right questions to ask them, so we created the certificate program with that in mind. And then just to wrap up, I alluded to it a little bit earlier on, but part of the impact on this one is we really want to help seed the economy, we really want to help train developers to be successful in the new economy, and I want to share some statistics about the state of employment globally. First of all, this chart, this is a general nice upwards trend chart, but there's a big dip at the bottom, and this is the chart of GDP. I averaged that across several countries, UK, USA, and others, and you can see the general trend over about the past 35 years is that it's been mostly upwards, there's been a couple of little dips. But the problem when you chart something like this is you see a general upward trend and you think everything is fine, and you might miss some of the important nuances in this. So I recharted it as a derivative chart, where instead of just showing the overall growth, I wanted to show the growth in any given year relative to the previous two years. And as a result, the same 35-year chart showed this. This to me is really interesting, because there are a couple of big peaks in this chart. One of them is here, and this one, I don't have the years on this one, but this one was from about 1993 through about 2007, and this was how the growth of the web and how the emergence of the web and the tech economy really changed the world economically in this massive period of sustained growth. And it's no surprise, what are now the biggest companies in the world by market cap are all companies that came as a result of this growth, the Microsofts, the Apples, the Googles, all of those tech companies that are now giants came from this period of growth. The second one here is the app economy. This is really the growth that came about as a result of the emergence of the smartphone. First introduced, I would say, you could argue by Apple with the launch of the iPhone in 2007. I know there were other smartphones before that, but this economy really picked up then. Then the emergence and growth of Android and other platforms. And we've seen this as a really, really huge growth area. And while there's a big dip on the right because of COVID and because of the pandemic, we just want to realize that even like Apple this year released a statistic that said in 2020, the app ecosystem has grown by 300,000 jobs in the US alone. So we can see that jobs and employment are still very, very hot because of these two peaks. And the next peak, maybe in 10 years time, if you look at a graph like this one, there'll be a new peak. And that's the fourth industrial revolution and the economy that comes as a result of AI. The World Economic Forum, for example, did a report where they wanted to look at the jobs of tomorrow, starting in 2020 and ending in 2022. And they had five categories, data and AI, engineering and cloud, people and culture, product development, sales and marketing. And far and away, the biggest, as you can see here, was about a 37% growth in jobs between 2020 and 2022 in data and AI. And engineering and cloud computing, you can see, was plus 34%. A lot of that was around ML stuff in clouds also. So we can see it's a huge potential growth thing for jobs and unemployment. There's also a Forbes report, and I put the link to it here, that showed the global ML market will have a compound annual growth rate of about 44% between 2017 and 2024, from 1.5 billion to 20.8 billion. So it is a huge growth area. And AI software revenue, they also reported between 2018 and 2025, from 10 to 126 billion, which is a CAGR of 43%. So we can see here that there's definitely massive, massive growth in this area coming. All the reports that I see tend to put it as a compound annual growth rate in the 40s. Both of these, 43% and 44%, support that. And even they mined LinkedIn to take a look at jobs, and 44,000 jobs in the USA required some kind of ML, and almost 100,000 globally required some kind of ML. So we can see ML, AI, these kind of skills are massively, massively important as we go forward. And we wanted to be the people to help train millions of developers to reach billions of people. And with that, I just want to say thank you, and I'll be able to answer any questions. Yeah. Thank you so much. Thank you. Yeah. Can I answer that question with all of the above? Yes, please, you do have an answer right now. Let's do it. Yeah. So like, wherever I want to deploy my models, absolutely everywhere. Everywhere. You know what? Yeah, everywhere. It's like all the options. What's the correct option? Everything. Everything. All right. Models everywhere. That's awesome. Well, right now, let's actually start with some amazing questions because we kind of have some amazing questions right here. So actually, this is by Hossain. I hope I'm pronouncing your name right. But which do you think is a better option for the industry, tensorflow or PyTorch? Well, first of all, I'll start by saying I'm completely biased, given that I work on the tensorflow team. I'm wearing a tensorflow jacket and I'm drinking coffee from a tensorflow mug. So I think my answer there is going to be tensorflow. So if I take off my jacket and take off my mug, I think it's always good for everybody to learn as much as possible about everything. We welcome the great competition that we're getting with PyTorch. I think it's brilliant for the industry to have competition amongst the vendors. But I will answer that despite that, I still think tensorflow, just because of the number of different places that you can deploy tensorflow models, right? From the massive cloud scale infrastructure with multi-node TPUs, all the way down to the tiny microcontrollers and embedded systems that your tensorflow skills, regardless of what type of model you want to produce, there's a place for you to put it. And so from the industry perspective, I think I would strongly recommend, despite my jacket, I would still strongly recommend tensorflow. Yeah, definitely. I do know that PyTorch does abstract a lot of hidden mechanisms that tensorflow does provide. And while it's kind of a trade-off, because on one side, tensorflow is definitely more adaptable and controllable and flexible, whereas with PyTorch, some people just find it an easier interface to just use in general. I won't claim any kind of expertise in PyTorch. It's something that I need to learn more of, to be perfectly frank. But yeah, my focus has always been on enabling people to succeed with the technology. And it's one level is being able to build models, but the other level and more important level is having somewhere to deploy them. And I think that deployment surface is a massive, massive strength of tensorflow. And it's where I think we distinguish ourselves across in ML. And that's why I'm particularly passionate about it. Yeah, that's a great answer. And also, here's another question from Arjun Kumar. Do you recommend, how do you recommend continuing to build your career after completing courses like yours or Anjuang's, for example? Yeah, good question. I think practice, practice, practice is very important. I think like with any kind of coding, the more that you do it, the better you get at it. You begin to realize the pitfalls. You begin to realize those tips, those tricks, those things that make you better at doing it. So ML is no exception in that regard. It's really practice, practice, practice. Doesn't have to be public with competitions on Kaggle or anything like that. And one thing that I always recommend is you find something that you're passionate about, and then there's got to be some data out there around something that you're passionate about and start building models in that regard. And just continue to practice and become a better model creator. And then once you have those models to practice, practice, practice, then in deploying those models, be it like a large scale, global scalable infrastructures, or maybe just the single cell phone for yourself to use. But the key with anything is constantly practice to improve yourself. Yeah, that's true. About practicing too and everything that is associated with this too, there are also a number of people. I was looking at a bunch of questions where there is that mathematical aspect that is deep learning and machine learning, whereas tensorflow is more of like, okay, let's break things apart from a more practical aspect and then dig deep into that way. So a question would be, how do you balance these two completely different notions of approaching machine learning and deep learning? Would you prefer one or the other? Or somehow a symbiotic relationship between theory and practical applications when going on your machine learning journey? I like the term that you just used, symbiotic, because I think that's exactly right. But symbiotic, like how do you start? They should be meshed together, but how do you get started? I can say just for me personally, I hadn't done calculus in 30 years when I started looking into modern machine learning. And when the initial material that I was looking at was teaching you how an optimizer works through gradient descent and tons of calculus or teaching you how a convolution works using mathematical and Greek notation, I'll admit I was lost. It was harder for me to follow it. But when it was like I started looking into this using code and primarily code, it's much easier for me to read and understand code. So then my symbiotic relationship began definitely on the practical coding side of it so that now I can understand what a model is, now I can understand how a model learns from parameters, now I can see how back propagation works and things like that. And then with all of that context around it, then to start going into the math and then seeing, ah, now I see why I'm using calculus for gradient descent and all of those kind of things. So to really build that context around it so that I could understand it better. And that's why now it's very much that symbiotic relationship that I could start building models without knowing any math. But once I started building models, if I wanted to build better models, then to start understanding what's going on under the hood was necessary. Yeah, that's completely true. Because I know there's a lot of people who are in school, especially based on our polls. And maybe in school, you'll be learning some very complex mathematics, like calculus, for example, that you mentioned. And you're like, well, if it's just thrust at you, it's like, OK, but where do I use this? I'm good with it. I don't care as long as I just pass this course. I'm good. But then when you're like, but I am interested in machine learning and deep learning. Hmm. What do I need to do for that? Incoming calculus just surreptitiously is like, hi again. And then you're like, oh, so this is how you can use calculus to do to create like all of these machine learning projects that you see everywhere just being deployed. It's like, oh, my God, there's self-driving cars. There's like this system that can detect your fraud, whether your activity is fraudulent. Yeah, just like the applications are endless and all with calculus. So that's definitely very interesting to hear. Yeah, well, it's like, don't let that be the gatekeeper that prevents you from getting in that if you're like me, that once you start reading calculus and your mind, your brain goes to sleep, you know, that kind of thing. You can really get started and start doing all of this stuff and be productive before you start getting into the math. But it's like once you're at that productive level, then it becomes easier to get into the math because you've built that context around it and it makes it much easier for you to do so. Sorry, you said you had another question. No, we always have so many questions for you. They just keep pouring in. So, you know, you've seen like tensorflow just like, you know, build, just become this huge thing right in front of you. But do you have any reservations about tensorflow? Like maybe somewhere along the way, tensorflow could have done something better and it's probably still something that's a big problem. Is there anything like that? I don't think there's anything that I would consider to be a big problem now. I think one thing that would have been really nice to have earlier, but we then eventually had with tensorflow 2 was to think about it in a more Pythonic way. Because like early versions of tensorflow were very much graph oriented. So you'd load all your stuff into a graph and then you execute like a session within a graph. And for me as a coder, as opposed to an AI person, that was a little bit alien to begin with. It was very hard to like, cause you can't do step by step debugging and things like that. When we released tensorflow 2 became like with eager by default became more Pythonic. So it made it much easier for the people who are coders to get up and running. And if there was any one thing that I had the time machine that I would change to do sooner, it would be that. Ah, that's very fascinating. And I think we have like a final question over here. So I, so there are people with like some experience in like javascript and Python and they, they're always like asking like, so which is the best language to start my machine learning journey? Do you have any, any specific thoughts on that? Yeah, I think the best language to start your machine learning journey of those two is the one that you're most passionate about using. I know a lot of people who love javascript. I know a lot of people who hate javascript. I know a lot of people that love Python. I know a lot of people who hate it. If you don't have any particular affinity to either of them, my recommendation would be to start with Python just because of the number of support libraries that are out there for the beyond machine learning part of it, the data science part of it, the drawing graphs and all of that kind of thing. So like I said, if you were going to pick one from scratch, I would say definitely Python, but there's certainly nothing wrong with javascript. The things you can do with tensorflow JS are fabulous. And yeah, I think Jason's going to be talking about that at this conference later today and he always shows amazing things. Exactly. Yeah, I'm completely looking forward to that too. By the way, Lawrence, I think we are out of time for some questions, but thank you so much for answering all of our questions until now. And for our lovely audience here, of course, we're always happy to have you. And if you guys, anyone here just did not get a chance to ask Lawrence your questions or they didn't come up right now, feel free to reach out to him in his speaker room, even starting like right now after this call. So just be there. Lawrence, it was great having you right here and we look forward to hear from you soon. Yep. Sounds good. Thanks. Thanks for having me.
34 min
02 Jul, 2021

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Workshops on related topic