Visualising React: Metaphors, Models, and Spatial Mediums

♪ Hi. I'm really glad I can't really see you all because of the blinding light. That's really useful, but this is going to go well. Good, so, yes, we're up. This talk is called A Picture Worth 1,000 Programs, and it's going to be about making pictures, and specifically about making pictures of programs, and hopefully about making pictures worth 1,000 programs, as the traditional saying goes. So before we dive in, I want to quickly introduce myself and let you know I am actually up here on stage talking to you about pictorial programs. So my name is Maggie. I'm a designer, art director, illustrator, metaphor nerd, and tangentially I also build things with react. I spent the last five years working as the art director at Egghead.io, which is an education platform for web developers, but as of next week, I will be moving on to a new role, leading design at Hash.ai. But I spent my whole career so far creating visual representations of programs. During my time at Egghead, I taught hundreds, I made hundreds of cover illustrations for the courses we taught. Each of these would start with an abstract programming concept, and I had to find a way to visually represent that in a single image. I learned to rely very heavily on visual metaphors for these, so styling with css became painting a house, and organizing types in typescript became suits in a deck of cards. I've also made a lot of illustrated diagrams over the years to explain topics like how javascript prototype inheritance works, or what happens when you flatten an array. One that became quite popular was a visual essay on what APIs are and how they work, which was told through small robotic waiters that bring you the data you ask for. And the goal of all these illustrations is to make complex technical topics relatable and easier to understand, which is a theme you're going to hear a lot about in this talk. I also recently collaborated on a project called Just javascript with Dan Abramov, who some people might have heard of. It's a javascript course that teaches the core mental models of the language through visual diagrams and animations. We worked together to develop this whole visual language that was a bit more formal than some of the other illustrations I showed. Every piece of syntax is correlated to a specific visual shape, and we used the system to explain concepts like assigning properties or object mutation. I'm not running through all this work to show off. I just wanted to give you some context around the kind of programming visuals I've made in the past, so you have a sense for what I mean when I say pictures of programming. In this process of transforming programming concepts into visual images hundreds of times, I've been forced to think a lot about the way we represent and communicate complex programming ideas. And I've come to believe that visual representations have a lot to offer us here in the land of code. In this talk, I'm going to show you how visuals can make programming concepts less abstract, easier to understand, and more accessible to more people. I'm also going to show you why visuals are so special, and it turns out to be relatively simple. And it's because visuals bring invisible, abstract programming concepts down into the embodied world. The embodied world is where you and I live. Everyone watching this talk has a body, and you use it to interact with physical objects around you and move through space and experience events over time. Everything you know about the world is mediated through your body, and this fact is so fundamental that we sometimes completely forget it. And programming concepts are abstract ideas that do not live here in the embodied world with us. They're imaginary objects and functions that exist in the liminal space that feels like it doesn't obey the same laws of physics as we do. And we can only interact with them through this disembodied experience of typing linear text characters into a code editor. This is precisely what makes programming so difficult. We're trying to reason about and work with things we cannot see or touch when we are creatures who are evolutionarily adapted to function in a highly visual, spatial, and physical world. And I think that visuals are a big part of bridging that gap between our embodied human world and the disembodied machine world that we're trying to program in. So here's the plan. We're going to explore this topic through three questions. We're first going to ask what's wrong with text. Then we'll explore what can visuals do that text can't. And finally, we'll go on a very brief history tour to find out haven't we already tried this? After all, there are no new ideas under the sun, and many, many people in the past have explored ways to make programming more visual. So we're going to look at what's already been tried and what opportunities still lie ahead. So first, what's wrong with text? This is an important question to ask because everything we do in programming is expressed in text. Every single app you've ever worked on looks like this, right? It's text arranged in lines going from left to right and top to bottom. Here's every documentation website you've ever used. Here's every blog post you've ever read. All of our current programming languages, tools, interfaces, and documentation are overwhelmingly text-centric. You sometimes get diagrams here and there, but it's really slim pickings. If I had to guess about the balance of text to visuals in our industry, I'd bet we're at 98% text to 2% visuals. This is not based on an official survey, and I couldn't find anyone who has done an official survey, so this is just based off my personal experience in the web development community. But if you take a minute and think over all the code and documentation that you interact with on a daily basis, I'm betting you're going to land on a similar estimate. If we look at the history of programming, it's fairly clear how we ended up in a text-heavy world. This is a computer circa 1970. You'll notice the lack of screen. You had a keyboard and a stack of punch cards, and the only thing you could do was type linear text to create programs. This design constraint meant that all our early programming languages were text-based, and once you establish text as the primary paradigm of a field, it becomes really hard to break away from, especially in an industry where we rely so heavily on lower-level abstractions created by all the programmers who came before us. There are also plenty of logical reasons why we rely so heavily on text in programming. Written words and syntax are an ideal medium for expressing abstract logic. It's quick to create, it's flexible, and it's easy to move between applications through copy and paste without worrying about compatibility. You can pack a dense amount of information into a very small space, and you can be very specific about what you mean, which obviously matters when we're talking to computers who have no sense of nuance. So far, text has been working great for us in programming. But some of text's greatest strengths are also its greatest weaknesses. The abstract nature of text is what removes it from our embodied experiences in space and time. When we code, we're writing a set of hypothetical instructions to run on someone else's machine at some point in the future in a time and place that we'll never know about. This level of abstraction removes the physical, spatial, and embodied qualities that humans rely upon to understand the world around us. This can be good in some ways, right? If we want to write a function like fetch user data, we don't have to define the size, shape, or color of it. It's just a simple function floating in machine land. If you already know what this function is and how it works and where it's located, this level of brevity is great. But imagine someone new to programming who has never written a function to fetch user data and has no pre-existing mental model of how it might work. One of the easiest and most effective ways to make it comprehensible to them is to explain it in familiar terms by using physical qualities they already understand, like size, shape, color, and spatial relationships. Which might give us something like this to help explain what a fetch user data function does. The visual doesn't have to be crazy complex or beautiful. Boxes and arrows work great. We're certainly allowed to use text labels to make the imagery clear. It's just that visuals like this allow us to use our pre-existing embodied knowledge to show how programs work in a way that linear text can't. So I've just started to hint at the question we're going to look at in part two, which is what can visuals offer us that we can't get from linear text? I think visuals reveal three aspects of programming that we're unable to see in linear text. They reveal fundamental metaphors embedded in our programming languages. They reveal spatial mappings that we use to reason about how our programs are structured and how data moves through them. And they reveal how our programs and data behave over time. These things are all implicit in the programs we write, but they're not shown explicitly in the medium of linear text. So let's start with metaphors. Just to make sure we're all on the same page, let's establish that metaphors are thinking tools that allow us to understand one thing in terms of another. So let's say we have thing A here, which we understand, and thing B, which we don't. And in order to help us understand thing B and get a general idea of what it's like and what it can do, we map the qualities of thing A onto it. If we say corruption is a disease, we understand that corruption spreads, is difficult to overcome, and, if left unchecked, can kill. Similarly, we often say life is a journey. There are many paths our lives might take. They vary in length, and they all have a beginning and a final destination. Now, when I talk about metaphor in the context of programming, I don't mean the creative, fanciful metaphors that you find in poetry, like taking the road less traveled and wandering lonely as a cloud. Those are called figurative or poetic metaphor, and they're the sort we're often warned not to use in technical tutorials, since elaborate, poorly chosen metaphors can be more confusing than helpful. I'm talking about a much more fundamental type of metaphor that lies at the heart of all abstract thinking, including programming, and these are called cognitive metaphors, since they enable cognition on a much deeper level. These cognitive metaphors are based in our embodied experiences of the world. We have all these non-physical things we need to communicate to each other, like emotions and thoughts and ideas and programming concepts, and in order to understand them and talk about them, we use our experience of the physical world around us as a metaphor. If we look at the way we talk about abstract things, this becomes obvious. We talk about ideas in terms of light, when we say, that's a really bright idea, or that really illuminated the problem. We talk about emotions, like their objects, we'll say, he hid his jealousy, or she doesn't handle anger very well. We can also use force and motion metaphors to describe experiences. We can say, I found your talk moving, or your talk really touched me. Or we could talk about programming in terms of temperature, like we have hot reloading in react, or the javascript landscape is really heating up. So this isn't a theory I've just come up with. These principles come from the field of cognitive metaphor and embodied cognition. They were first developed in the 1980s by George Lakoff and Mark Johnson, who have since written numerous books on this topic, and it's become a major area of research in cognitive science. These two books, Philosophy in the Flesh and Metaphors We Live By, are two of the major texts, if you're interested and want to dive deeper. But I can't go too deep into it here, so hopefully you can do a PhD on it. And just like every other abstract topic that we can't see or touch, programming relies heavily on these physical embodied metaphors. And we have to, because programming itself is a game of abstractions, right? We write a javascript file, and what we're really doing is telling a microchip to flip a bunch of logic gates using tiny electrical pulses. Programming tiny logic gates is tedious and difficult for humans. So we've developed a stack of elaborate metaphors that make it faster and easier. Some people might prefer to call these abstractions, but for the purposes of this talk, metaphors and abstractions are roughly the same thing, and we can debate the differences on Twitter later. We simplify our binary code into machine code, which we simplify into higher-level languages like javascript, which we simplify into GUIs. And at every step of this process, we're trying to make the abstract machine world resemble our tangible human world. Because the closer we move towards intuitive embodied knowledge in the upper right-hand side of the scale, the easier it becomes for us to understand what's happening in a system. Components in react are a great example of these embodied metaphors in action. Components are essentially containers. They hold sets of UI elements for us. A card component might have an image, a button, and a paragraph inside it. The CPU that's eventually going to render this component on the screen knows nothing about containers. It only knows machine code and how to make the right pixels light up. The container is a metaphor we humans need in order to manage and organize the code we write. The only reason you know what a container is is because as a child, you dumped sand into a bucket and then you dumped it back out again. And through this physical world where you're doing embodied experience and learning, it taught you that containers hold things. They have insides and outsides. They have boundaries. All of these concepts are essential and necessary for you to understand how components work in react. So let's take another one. In react, we structure our components in a hierarchical tree where everything is connected back to a single root component. You know what a tree is from seeing thousands of trees. And you get that they have many branches that spread out from a single root, an understanding that allows you to work with component trees in react. The tree metaphor is also kind of a double metaphor since it's based on the idea of a family tree. We have parent and child components that inherit props in the way children inherit qualities from their parents. And these concepts that we're borrowing from the human world allow us to interact with machines in a way that is easy and natural for us. And you can probably see where I'm going with this. Since your understanding of react is based on your pre-existing knowledge of the physical world around you, it would be helpful if we could make that explicit in the ways you represent and explain react. And visual mediums allow us to do this. You could certainly read about react component structures and passing down props, then form a mental image in your mind based on your understanding of physical trees and family inheritance, and then apply that knowledge to writing linear code all without ever seeing it explicitly visualised. But that's called making it harder than it has to be. So let's move on to the next visual quality, space. As humans with bodies, we inherently understand a large array of spatial concepts like up, down, left, right, in, out, big, small. And we use these metaphors just like we use physical concepts to understand programs, we also use spatial concepts to talk about the structure and behaviour of programs. If you think about the way we talk about the internet, there are very clear physical directions to it. We upload data to the cloud above us and download files to our desktops. We look through a browser window. We browse web pages moving from left to right, so pages you visited in the past are to the left, and pages you're going to in the future are in the right. This format is also based on our Western cultural mapping of time to space. We think that the past is on the left and the future is on the right, but not every culture does that. Specifically in react, we use our understanding of vertical directions to think about how data moves. react has a hierarchical order to components, and parent components are above children, and they pass data downwards, which means we also have gravity in react land. The idea of prop drilling is another one that suggests drilling downwards, so you have to pass data deep into your tree, so it also has depth. And when we talk about side effects, we're using our understanding that there is a centre and a periphery where things happen on the side, so when we run a function like a setTimeout inside a useEffect hook, we understand it's accessing something outside our central component. I could go on with these. Our apps have front and back ends, useShallow merge objects. I mean, you get the point here. We use spatial principles from our embodied experience of the world to talk about and understand react. And when we create visuals that show these spatial principles explicitly, it clarifies what we're already doing in our heads. And our final element here is change over time. When we're working in a linear text editor, time is essentially invisible. We're looking at a static representation that describes a whole array of future events that may or may not happen, depending on what button a user clicks or whether our data request resolves. We're forced to use our imaginations to predict what's going to happen in all those potential futures rather than being able to see it in some form. Our current best technique for trying to see how things change over time is console logging data along the way. But console logging feels like trying to get a program to send signals up to the surface of a dark ocean where everything is being executed out of sight. We can't see anything happening down there, and we just have to keep asking for clues about how the data is changing, which doesn't feel like the best developer experience. This is again where visuals might come in handy because they allow us to see multiple points in time within a single frame. They let us compare things side by side in a way that we can't with linear text. They essentially allow us to play spot the difference. Let's look at the syntax of the useEffectHook as an example of this. Here are four different versions of the useEffectHook which are each going to behave differently when they execute in the browser. The first has no dependency array. The second has an empty one. The third has the value count as a dependency. And the fourth also has count as a dependency, and it runs a cleanup function. By just looking at these four versions of syntax, do you have any way of knowing how this function is going to run once it loads into the browser? Specifically, do you know how often it's going to be called and when? Given the demographics of this audience, I bet you do. But it's because you've memorized the syntactical meaning here rather than knowing because the answer is visible in our syntax. What we need to do here is compare four things that behave differently over time. The best way to do that is to visually show what changes over time. I've made a simple set of diagrams to try to show the difference between these four on a timeline. Here's our first version with no dependency array. The useEffect function is called on every render regardless of whether our count variable updates or not. Here's our second version where the dependency array is empty, so the useEffect function is only called on the initial render, and we still don't care what the count variable is doing. In our third version with count in the dependency array, useEffect is called only when count updates, which then triggers a re-render. In our final version, useEffect is called on count updates and then it runs a cleanup function after. Even if we remove the labels on these, this simple visual representation of how useEffect behaves over time gives us a much better understanding of how it works than the linear syntax is able to. So part three, haven't we already tried this? Obviously, I am not the first person to realize that visual mediums enable us to understand and reason in ways that are worth exploring in programming. The primary way people have tried to incorporate visuals into programming in the past is by sticking graphical user interfaces onto IDEs. These efforts all fall under the umbrella of what gets called visual programming. There have been many, many past attempts at this with varying degrees of success. I'm going to quickly whip through a few examples so you get a sense of what's already been tried. The very first visual programming language was Ivan Sutherland's Sketchpad in 1963. This is Grail from 1968, where we put text into boxes for the first time. This is Pygmalion from 1975, that built on that box and arrow model. This is LabVIEW, which came out in 1986 and is used for systems engineering, and we got a lot more intense about boxes at this stage. Here's a more modern example. This is Blueprint in Unreal Engine, which is used for 3d game development. This is Max MSP, which is widely used for building audio-visual experiences. Similarly, here's TouchDesigner, also for interactive multimedia. This is Origami Studio, which is a prototyping tool built by Facebook, and you'll notice the nodes and wires design pattern is very popular in a lot of these. So there's lots of promising stuff in these examples, but visual programming is still relatively niche. We've also discovered a bunch of really wicked design challenges that are hard to solve. These systems don't scale well. They sometimes use ambiguous symbols and unfamiliar interface patterns. They try to turn everything into a box, which takes up way too much space on the screen, and it can lead to literal spaghetti code. This is a Figma prototype gone very wrong. These issues have led to a lot of skepticism over the feasibility of visual programming, and being an advocate for it often feels like being Gretchen in Mean Girls. Visual programming certainly isn't dead, though. Funnily enough, the new no-and-low code movement really looks like visual programming under a new name. Many of the interface patterns that visual programming helped develop, like nodes and wires or direct manipulation, are visible in tools like IntegraMap and Zapier and Webflow. But rather than trying to build Turing-complete visual programming languages that can reach industrial scales, we've tactically moved on to developing visual interfaces for specific use cases in programming with sensible constraints. I'm a huge fan of the visual programming agenda and finding ways to add more visual affordances into the current developer tools, and there are lots of really smart, impressive people working on the problem. But in many ways, it's taking the harder route. Building a true visual programming medium will require overcoming a ton of design and culture and engineering challenges, and frankly, it's going to take a while. But there are easier ways to advance this effort in the short term. We can simply sprinkle some visuals into our existing textual world, which means adding more diagrams and illustrations into blog posts, documentation, and learning materials, and if we're feeling brave, building plugins for our editors and developer tools that visualize very small, scoped elements of our programs. This is essentially the low-tech paper prototype version of building a fully-fledged visual programming interface. So to wrap this up, here's the thing I want you to take away from this talk. We need more visuals that reveal metaphors, spatial meaning, and change over time to make programming easier for everyone. It will make it easier for you, since you are an embodied human who needs to learn complex, abstract programming concepts in order to do your job well. It will make it easier for all the people who don't currently know how to program but are trying to learn, and it makes it easier for people who aren't developers but need to understand what we do, like product managers and designers who are often in our text-heavy documentation. So, what's a developer to do? I'm sure you are a humble but skilled react developer and want to help advance this goal. First, I highly suggest looking into the history of visual programming and some of the past attempts in this field. There's a lot of good previous art to learn from. If you are a current or future creator of tools for other developers, you should consider ways to build visual affordances into your libraries, plugins, apps, or frameworks. And finally, you should use and advocate for visual explanations in documentation and tutorials. That might mean making them for your own blog posts or collaborating with designers if you work on larger projects with a lot of documentation. We can set the bar low for now. We are trying to budge this fake statistic up to 10%. That was a lot of information packed into one talk. This was definitely a taster session, more than a complete meal. I've put together a list of some things that I think are really important to consider when you're thinking about visual programming. I've included a list of some of the most important things I've covered in this talk. These will be up on my website at maggiealpertone.com slash programmingpictures. George Lakoff and Mark Johnson are the ones that worked on cognitive metaphors. Barbara Javersky's work on embodied cognition is great. Literally everything Brett Victor has ever made is worth your time. There are some really great resources on thefutureofcoding.org. They're a visual programming enthusiast community. Thank you very much for listening. I will post these on my website, maggiealpertone.com, where I have also more writing on metaphors and visual programming and that kind of stuff. You can tweet me at mappletons. Applause Amazing! Thank you, Maggie. Please, Foxy, join me in the office. Applause That was truly so, so incredible. And folks have tons of questions. So let's get right into it. L, the letter, asks, How do you arrive at a visual metaphor? Is it something that forms unconsciously as you engage with the concept, or do you verbally reason through it? I definitely verbally reason through it and research. I rely a lot on linguistics and the kind of techniques linguists use to analyze the language we use around things. When I'm trying to design a metaphor for a certain technology, I'll read the docs and I'll really pay attention to the words they're using. And they're almost always using physical words. They're moving, passing, you know, spatial things. They're talking about how things are arranged in space, even though they're just writing it in linear text. But I pay a lot of attention to language. There's more than one question asking if you have any stickers or merch of your illustrations. The people need more Maggie. I don't sell anything personally. I'm trying to think. Egghead has a swag site where they sell some of the course illustration stickers. But we tend to also just kind of give them out for free when we show up at conferences. But I don't have an organized swag store. Sorry. And similarly, again, the people really are into it. Any chance of a collaboration with the new beta react docs team? I think I'm allowed to say I might have some visuals coming up in the new react docs. Amazing. They're in progress. It's a secret. Yeah, I know, right? What resources would you recommend for someone that wants to explore these concepts further? I know you mentioned a couple, but is there anything else that you'd like to add to that? I think it's best to maybe go to my website where I have that list on the programming pictures slug. I'll add more on there because I have a long list, but it would take a while to read them all out. A couple of folks are interested about tooling. What do you use to draw, to add your illustrations to the decks, et cetera? All the illustrations I draw nowadays are on an iPad with Procreate. I used to draw more in Photoshop on a Cintiq, but to be honest, they're almost interchangeable and the iPad's just better. We just had much better advances in hardware and software over the last five years that the iPad's kind of state of the art. Oh, this is an interesting one. Do you think that, and I'm sure there is, is there potentially an issue of metaphors being interpreted differently by different countries and cultures? How do we think about that? There definitely is, and that metaphors are incredibly cultural. Almost all the metaphors we use are scoped to the country we're in and the language we're speaking. Given the fact that react and the web development community is a fairly unified place and we all have a fairly shared understanding of metaphors and symbols, I find it's not that difficult to find metaphors that relate for everyone, although I'm always aware that if someone in another country who doesn't have a lot of the same associations that we do sees it, there's a good chance they could miss nuances of it. So, yeah, that is something you need to consider on a case-by-case basis. We'll need Maggie's in all countries and all time zones, just in case. Should we, do you think, be leveraging existing metaphors that are already in our programming languages and frameworks? Or trying to optimize for ones that perhaps communicate something well? That's a good question. It almost gets into a programming language design question. I like to assume that the people who designed the programming language thought enough about abstractions and the api language that the metaphors they have chosen, either consciously or unconsciously, are the correct ones. I don't try to invent brand new ones on top of it if I'm trying to draw something that is genuinely meant to explain the point. But sometimes, if it doesn't seem like there's a clear metaphor, that's when you get into the more inventing one that fits the needs. Yeah. That's really cool. There's a couple here that are sort of people wanting to know a little bit more about you and your process. Did you start programming and then go into design, or vice versa? And what's your process a little more to turn programming concepts into? You mentioned reading the docs a little bit. Is there anything else you go through? The docs, and then I do things like browse Reddit and Twitter, and I just look for the language that everyday people are using around whatever the technology is, if that answers the question. And which one came first? Oh, kind of both at the same time. I started playing around with html when I was 12, and there was very much a desire to make pretty things on the web was the impetus there. So I have always done both, and gone through different periods of intensity for each. I think when I started learning react maybe six years ago, I went much more into programming for a few years and was primarily design before that, but I've always gone back and forth. And this is just me thinking, do you do a lot of drawing on paper and pencil ever? Yes. Almost all my drawing starts on Post-it notes with a pencil, and the iPad is really only the final step, but there's a lot of sketches in the middle where I'm just figuring things out, and the polished final version is almost never the thing that takes time or the most effort. Do you ever find a drawing and not remember what it was for? Do I do what? Find a drawing and not remember what it was for. Oh, yeah. No, yeah. I look back, like in the same way you find notes from 10 years ago, and you're like, I don't even understand what this was supposed to be. We have time for one more. Drew asks, help! I'm awful at design and drawing. What can I do? Ooh, I have on my website a whole list of all my favorite courses and books and links and things that are about the technicalities of how do you draw a circle well, and how do you make a human face look like a human face. That sort of technical drawing content, that's, yeah, slash resources on my website will be a good resource for you. Thank you so much, Maggie, for your time. Find Maggie on the internet. Clearly great resources there. Big thanks to Maggie, please. applause