Let’s Build a TV Spatial Navigation

Rate this content
Bookmark

In this talk, I'll take you through my journey as I joined the team supporting our Smart TVs application and share my experience learning one of the most overlooked but essential pieces of functionality we have.

18 min
23 Oct, 2023

Video Summary and Transcription

This talk discusses the challenges of implementing spatial navigation for TV controls and the need for a library. The approach to spatial navigation can be improved by developing extra logic to connect TV control with the application. The demo application showcases the use of a NavigationEngine class and the useFocusRef hook function. The integration logic involves selecting the next node based on key presses and node coordinates. The talk also highlights the challenges of complex arrangements and circular navigations in smart TV application development.

Available in Español

1. Introduction to Spatial Navigation

Short description:

Welcome to the talk on Spatial Navigation. We'll be discussing the challenges of implementing spatial navigation for TV controls and why a library is needed. The market for smart TVs has multiple brands with their own operating systems, making it necessary to have native applications for each. However, to simplify maintenance, we built a web application for the user interface. Unfortunately, this approach resulted in the loss of native platform support for spatial navigation. Although there is a proposal to provide this functionality in browsers, it is still a work in progress.

Welcome, everyone. Thank you very, very much for joining this talk.

My name is Sergio Avalos, and we're going to be talking about Spatial Navigation. But rather than talking, we're going to be building in.

I'm a software engineer at Spotify and I recently joined about a year ago the team working behind the Spotify client that runs on your smart TV. That means that for this talk, we're not going to be talking about mobile, neither we're going to be talking about desktop. And most importantly, we're not going to be talking about mouse. Instead, we're going to be talking about TV control, that gadget that I bet all of you have in your living rooms.

Special navigation is nothing else, just a fancy name for describing what you do with the TV control when you are pressing the directional keys, the arrow keys for selecting the one application or just navigating to one of them. That got me very curious when I joined the recent team where I'm working because I didn't know that, I mean, I was surprised that one had to create a library for that. So I decided to dig into the code and I was fascinated. Not because the code was amazing, I mean, it was fine, the code, but most importantly because I felt that it was a very interesting problem to solve. So that's what this talk is about. I want to share with you my learning of how I learned about this library and what a better way to learn than just building it ourselves.

But in case you wonder, because that was my first impression, why do we need to build a library for spatial navigation? I mean, isn't it a huge utility that should be provided by the platforms? And the answer is yes, totally. If you're building a native application. Let me try to explain.

The market for smart TVs is quite cement, there are many brands and each of them run their own operating system. That means that you need to have your native application running for each of them. But, just to make our lives easier and reduce the maintenance costs, we decided to build a user interface using a web application that can be loaded in each of the native app. That gave us a great interoperability of shipping the same code to all these native applications. But, it came at the cost of losing the support from the native platform. In that case, it's obviously the spatial navigation.

Then, I was thinking, okay, okay. But, the year is 2023. Shouldn't that be provided by the browser? I mean, the browser, nowadays, is a very sophisticated piece of software. And, the answer is not yet. It's a work in progress. There is a proposal. It's still a draft for building this functionality, but it's not there yet.

2. Improving the Approach to Spatial Navigation

Short description:

We need to continue waiting. Are there any open-source projects we could use? Norwegian Media released one in 2019, but our application is older. Let's start building it. Wrap each navigational element with an ID and tell them where to go. This approach has caveats: difficult with dynamic views, prone to mistakes, and adds extra information. Let's improve this approach by developing the extra logic to connect TV control with our application.

We need to continue waiting. Then, I was thinking, okay, okay. But, are there any open-source projects out there that we could use? And, actually, there is. Thank you very much, Norwegian Media, for providing this. Unfortunately, they released it in, no, unfortunately, but they released it in 2019, and our application is a little bit older than that. So, we didn't have any back then.

Having answered that question, let's start. Let's start building it. If I ask you just from top of your head, like your intuition, how would you do it? I don't know about you, but for me, it was, I mean, the simplest that I could come up, and I think I read it on a blog from Norwegian Media, and even from Netflix. It's basically, you just wrap each of what I call navigational elements as the element that the user can interact with it, with just an ID, just identify them, and then you tell them where to go. Take for example, the sidebar of our application, the Spotify application. Each of these elements is just a link for the home view, the search, and so on, and like I've explained before, you wrap them with an ID, and in that wrapper, you tell them where to go, so if you are going to the, if you're in the search and you go to the app, then you tell them, go to this ID that is the home.

That approach actually gets the job done, but it obviously have a few caveats as you, I can imagine, you can anticipate. One is that it is difficult to work with dynamic views. Think for example, recommendations, the developer doesn't know what they are going to get. Also, it's ever prone, because the developer is the developers role to add this ID manually, so mistakes can happen. We're humans. And finally, it just adds extra information that is not related to the application, like I said, this is just a utility that should be invisible to the application layer. So let's improve this approach.

For this presentation, I built a very small application app that basically has just two views. It's a welcome. Click on this one. Then you go to another view that renders you just a surprise for whatever. And then you have the go back link. And then you come back to SMBN. It works perfectly well with the mouse, but it doesn't work with the TV control. So this is exactly what we're going to do. We're going to develop the extra logic that we need to connect TV control with our very simple application. Demo application.

3. Demo Application and Spatial Navigation Logic

Short description:

In the demo application, we have the index page with the router configuration for the welcome page and the surprise page. Each view is a React component, such as the question box and the go back link. We register navigational nodes, listen to events from the TBControl, and select the next element based on the direction. We create a NavigationEngine class to handle this logic and make it available to the app using a context provider. The API for setting the HTML element reference is straightforward, with a focus function.

Demo application. OK. Awesome. So I'm just going to go very briefly to the source code of the demonstration app. We have the index page that you get from the React app and create React app, a script. Inside of it, we have the component for our application that is just the router configuration for going to the welcome page and the surprise page. For this, I'm using React router DOM library.

And each of these components, I'm sorry, each of these views are just another React component that we have. For example, for the welcome page, we have an array of 10 empty elements and we only use it to render 10 different components that is the question box. And for the surprise view, we have already hard coded the links of the images that will display randomly along with another go back link. Finally, these two components are the question box and the go back link. That is nothing else, but it just only uses the link component from the React router. In the case of the question box, it's like rendering the question box image and the other one is basically just rendering the children. That is the text that says, go back.

Okay. So here is this. Let's jump into the logic of the spatial navigation. First, we start registering all the navigational nodes, then we listen to the events coming up from the TBControl. Finally, from there we select the element that should go depending on the direction. And finally, we just update the cursor, meaning what is the next element that should be focused? If I put everything on a diagram so it's crystal clear for you from the steps 1 to 3, you can see that each of the question boxes is going to be registered on a class called NavigationEngine with the method RegisterNode. We add an event listener called OnKeyDown that will call the HandleNavigation method from this class that we just defined. All right, step number one, let's create a class of this NavigationEngine that has a private variable called Node, and then one method for adding nodes to this private variable and another one for removing. Then we go back to the index script where we instantiate this class NavigationEngine for the purpose of this talk we make it available to the whole global Finally, we also make it available inside our app using a context provider. I hope you don't believe that we are writing directly from the Windows. That's only for this presentation. Finally, we go back to the navigation, not finally, but we go back to the navigational element because I wanted to show you first the API that I encountered. I felt it was super simple. It was just a hook function that returns you a callback for like setting the reference of the HTML element that you're rendering. And then there is focus as simple as that. You don't need to think about IDs.

4. UseFocusRef Hook and TV Control Integration

Short description:

To use the useFocusRef hook function, you need to create a reference value with a callback, generate a unique ID, and obtain an instance of the navigation engine. The registerNode method is called when the component is rendered and removed when it's sub-mounted to avoid memory leaks. We can debug the navigational nodes to ensure they are rendered correctly. We add an event listener to the document to listen to key presses and call the handleKeyEvent callback function. We use a map to define directional keys and integrate them with our internal values in the app.

It's only, that's the only thing that you need to do. How do you call this useFocusRef hook function? Well, it first, you create a reference value with the callback to instantiate this reference. Then you generate a unique ID. And finally you obtain one of the instance of the navigation engine using the context provider. And with the help of the useEffect function, every time the component is render, is mounted, excuse me, we're going to call the method called registerNode. And when it's sub-mounted we're going to remove it to avoid having memory leaks.

Cool. We are going to now debug this. So we want to make sure like all the navigational nodes are render and if we look at the nodes variable we see that we have 10. We click in each one of them and then again we have only one, so it's refreshing. We can even inspect inside and if we do see the reference is pointed to the HTML element. We go back and then again we have 10. So it's working. Let's go with the step number two.

Listen to the TV control. In the where we are in the in our application component we add an event listener to the document so that every time the any key is pressed then we just call this a callback function called handleKeyEvent. To generate that callback we basically just are we're using a method that will distinguish whether the key that you are pressing is one of the directional ones, the arrows, and just for this step we're going to console.login so we're able to debug it. To build these directional keys we already have a map where we define what is a directional key and that map is just basically the integration between our internal values in the app what we define as being up next and the values coming from the from the platform, in this case the our platform is a desktop, where it can change depending on if you're actually running on a smart tv or on a gaming console for example.

5. Integration Logic and Node Selection

Short description:

This part focuses on the integration logic for the native component of the application. We explore how to select the next node after a key press by using the getBoundingClientRect method to obtain the dimensions and coordinates of each node. We then filter the nodes by direction and main axis, and select the closest one based on distance. The NavigationEngine class is updated with the handleNavigation method to implement this logic. Finally, we update the course based on the initial diagram.

This is the part of the integration logic that the native component native application needs to know but we're not going to do it for this presentation.

So let's go and see if the events of the tv control are being registered and yeah we can see here I press the key down and it's telling me like it's the arrow key I go to the left and when then we have the left.

Now we can go with the funnest part of this code which is selecting the node after the user press the key. In case you were wondering why do we need the reference it's because we can call this method called getBoundingClientRect that gives you exactly the dimensions and also the coordinates according to the viewport where the element is rendered. That means that if you take all the nodes and you call this method one by one then you get all the information that you need to build that logic so you can forget in this moment about the application or whatever render.

With this information we can exactly decide where this focus should go. So how do you choose this the next node? First you filter all the nodes by the direction, then you filter by the main axis, and finally you pick the closest one just by the distance. Let's go step by step. Imagine that we're talking about another matrix, a bigger one, five by five. You are in the middle and the key that is pressed is going to the right. Then you will filter the first the last two columns, excuse me, but if you are pressing to the top then you pick the first two rows. From those, you choose by the main axis, so if you are going to the right then you choose those nodes that are between the margin top and the margin bottom. The same if you are going up then you choose between the margin right and the margin left. Sorry, it should be like that! Finally, once you narrow to those two, then you pick the one closest to the distance. How does it look in code? We go back to the NavigationEngine class that we defined before, we add the handleNavigation method, and we do this step by step. First we filter by the direction, and we do that with the help of this dictionary that already has the predefined method that you need to filter all those methods. Then from there, you do exactly the same but you are going to do the filter by the main axis, and then you pick the closest element. And we're pretty much done. Now we can go back to the step number two, remove the console.log, and call the method called handleNavigation. Cool. Let's see if this works. So if I click the down, then we can see the I already have it, auto-complete. We can see this is the element class. So we're in this corner. If we press to the left, then we have this one. If we press down, then you know what's going to happen, right? Awesome. So it's working, but it's not selecting the element that is supposed to be selected. Let's do that. Step number four, update the course. So we go back to the diagram that I showed at the beginning and we're going to update it.

6. Adding Subscribers and Final Remarks

Short description:

So every time a note is raised, we add a subscriber and execute callbacks to notify all subscribers. We update the useHook function to keep track of the focused element's state. We demonstrate the functionality and mention the challenges of complex arrangements, annoying pop-ups, and circular navigations. We provide links to a report and a library for further exploration. We encourage getting in touch and building a community around smart TV application development.

So every time a note is raised, we're going to add a subscriber. So the note can say like, Hey, if anything happens, please let me know. Notify me. Let's do this in code. If we go back to the navigation engine class, then we add one method for adding those subscriptions. Another one for executing all those callbacks called notifyAllSubscribers. And we call this method exactly after we found one of the elements.

We update the useHook function. We update the value that we previously defined and is focused with a new hook function that will keep the state whether that element in turn is focused or not. And again, with the help of the useEffect function, every time the component is mounted, first we check it initially like, hey, am I focused? Yeah or not. We update the state. And we also subscribe for whenever the navigation engine calls the handle navigation, then I just check, hey, is it me who is focused? And if so, then we just update the state.

Cool, we're pretty much done. Let's just do a quick demonstration. So it's going to top-down, it's going down, left, left, we click on OK, and again. So perfect, it's working. I was afraid of the demo. So this is just the beginning, because trust me, from here, it just gets a lot more complicated, a lot more funnier. For example, your application is not going to be a perfect shape matrix instead, where you're going to deal with a more complex arrangement where you have the sidebar or the different columns, so literally you need to deal with those corner cases. What about those pop-annoying and inconvenient pop-ups that tell you, hey, do you want to buy premium? This is an interesting case because you need to focus the attention on just two elements, accept or buy. Although all those navigation notes are behind, you just need to focus on those two. And finally, one of my favorite ones is called circular navigations, where you want to solve the constraints in some areas. For example, the sidebar menu, if the user is hitting down, down, down, down, it reaches the bottom, just for convenience, we want to reach it to the top. Trust me, this is a total game-changer. But we're not going to continue because we're out of time. But if you feel curious and you want to continue the party, here is the link of the report that I used for this presentation so we can continue creating. On the other hand, if you feel inspired and say like, I want to create my first smart application for the smart TV, there is the link of the library that you can use so you don't have to build this from scratch. However, if you think you've been thinking like, oh, this is way too complicated, you can do it in a much easier way. Please get in touch. My personal goal of this talk is like to create a small community because building an application for a smart TV is not easy. So let's get in touch. Let's help each other. But that's it for now. Muchas gracias. Thank you very much. Here you have again the links to the QR codes of the links. And I really hope to hear from you.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

TechLead Conference 2023TechLead Conference 2023
35 min
A Framework for Managing Technical Debt
Let’s face it: technical debt is inevitable and rewriting your code every 6 months is not an option. Refactoring is a complex topic that doesn't have a one-size-fits-all solution. Frontend applications are particularly sensitive because of frequent requirements and user flows changes. New abstractions, updated patterns and cleaning up those old functions - it all sounds great on paper, but it often fails in practice: todos accumulate, tickets end up rotting in the backlog and legacy code crops up in every corner of your codebase. So a process of continuous refactoring is the only weapon you have against tech debt.In the past three years, I’ve been exploring different strategies and processes for refactoring code. In this talk I will describe the key components of a framework for tackling refactoring and I will share some of the learnings accumulated along the way. Hopefully, this will help you in your quest of improving the code quality of your codebases.

React Summit 2023React Summit 2023
24 min
Debugging JS
As developers, we spend much of our time debugging apps - often code we didn't even write. Sadly, few developers have ever been taught how to approach debugging - it's something most of us learn through painful experience.  The good news is you _can_ learn how to debug effectively, and there's several key techniques and tools you can use for debugging JS and React apps.
React Advanced Conference 2022React Advanced Conference 2022
22 min
Monolith to Micro-Frontends
Top Content
Many companies worldwide are considering adopting Micro-Frontends to improve business agility and scale, however, there are many unknowns when it comes to what the migration path looks like in practice. In this talk, I will discuss the steps required to successfully migrate a monolithic React Application into a more modular decoupled frontend architecture.
React Advanced Conference 2023React Advanced Conference 2023
22 min
Power Fixing React Performance Woes
Next.js and other wrapping React frameworks provide great power in building larger applications. But with great power comes great performance responsibility - and if you don’t pay attention, it’s easy to add multiple seconds of loading penalty on all of your pages. Eek! Let’s walk through a case study of how a few hours of performance debugging improved both load and parse times for the Centered app by several hundred percent each. We’ll learn not just why those performance problems happen, but how to diagnose and fix them. Hooray, performance! ⚡️
React Summit 2023React Summit 2023
24 min
Video Editing in the Browser
Video editing is a booming market with influencers being all the rage with Reels, TikTok, Youtube. Did you know that browsers now have all the APIs to do video editing in the browser? In this talk I'm going to give you a primer on how video encoding works and how to make it work within the browser. Spoiler, it's not trivial!

Workshops on related topic

React Summit Remote Edition 2021React Summit Remote Edition 2021
87 min
Building a Shopify App with React & Node
Top Content
WorkshopFree
Shopify merchants have a diverse set of needs, and developers have a unique opportunity to meet those needs building apps. Building an app can be tough work but Shopify has created a set of tools and resources to help you build out a seamless app experience as quickly as possible. Get hands on experience building an embedded Shopify app using the Shopify App CLI, Polaris and Shopify App Bridge.We’ll show you how to create an app that accesses information from a development store and can run in your local environment.
JSNation 2022JSNation 2022
41 min
Build a chat room with Appwrite and React
WorkshopFree
API's/Backends are difficult and we need websockets. You will be using VS Code as your editor, Parcel.js, Chakra-ui, React, React Icons, and Appwrite. By the end of this workshop, you will have the knowledge to build a real-time app using Appwrite and zero API development. Follow along and you'll have an awesome chat app to show off!
GraphQL Galaxy 2021GraphQL Galaxy 2021
164 min
Hard GraphQL Problems at Shopify
WorkshopFree
At Shopify scale, we solve some pretty hard problems. In this workshop, five different speakers will outline some of the challenges we’ve faced, and how we’ve overcome them.

Table of contents:
1 - The infamous "N+1" problem: Jonathan Baker - Let's talk about what it is, why it is a problem, and how Shopify handles it at scale across several GraphQL APIs.
2 - Contextualizing GraphQL APIs: Alex Ackerman - How and why we decided to use directives. I’ll share what directives are, which directives are available out of the box, and how to create custom directives.
3 - Faster GraphQL queries for mobile clients: Theo Ben Hassen - As your mobile app grows, so will your GraphQL queries. In this talk, I will go over diverse strategies to make your queries faster and more effective.
4 - Building tomorrow’s product today: Greg MacWilliam - How Shopify adopts future features in today’s code.
5 - Managing large APIs effectively: Rebecca Friedman - We have thousands of developers at Shopify. Let’s take a look at how we’re ensuring the quality and consistency of our GraphQL APIs with so many contributors.
JSNation 2023JSNation 2023
57 min
0 To Auth In An Hour For Your JavaScript App
WorkshopFree
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.js backend + Vanilla JS frontend) to authenticate users with One Time Passwords (email) and OAuth, including:
- User authentication – Managing user interactions, returning session / refresh JWTs- Session management and validation – Storing the session securely for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
JSNation 2023JSNation 2023
87 min
Build a Collaborative Notion-Like Product in 2H
WorkshopFree
You have been tasked with creating a collaborative text editing feature within your company’s product. Something along the lines of Notion or Google Docs.
CK 5 is a feature-rich framework and ecosystem of ready-to-use features targeting a wide range of use cases. It offers a cloud infrastructure to support the real-time collaboration system needs. During this workshop, you will learn how to set up and integrate CK 5. We will go over the very basics of embedding the editor on a page, through configuration, to enabling real-time collaboration features. Key learnings: How to embed, set up, and configure CK 5 to best fit a document editing system supporting real-time collaboration.
Table of contents:- Introduction to the CK 5 ecosystem.- Introduction to a “Notion-like” project template.- Embedding CK 5 on a page.- Basic CK 5 configuration.- Tuning up CK 5 for a specific use case.- Enabling real-time editing features.