Controlling Apps with Your Mind and AI

Rate this content
Bookmark

What is the future of user interactions? Will we continue using web and mobile or will we switch into VR and AR completely? What is our UX today on web and mobile and how it will change when the digital world will bridge into new dimensions. Would we still use keyboard and mouse or gestures or will we use something else? In this talk we will get a glimpse into the future where we will control apps with our thoughts. Literally. It’s not a thought experiment, but a journey into our brainwaves with consumer EEG headset. We will explore how we can use them and AI to create futuristic experiences, which will lay a foundation stone to our future interactions with the digital world.

Vladimir Novick
Vladimir Novick
25 min
02 Aug, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

This Talk explores controlling apps with the mind and the future of UI and UX. It discusses the integration of VR and AR into UI and UX, the understanding of neurons and EEG headsets, connecting to Muse via Bluetooth, measuring brain waves and blink detection, feeding data to machine learning, and mind control with AR. The speaker emphasizes the importance of learning React Native, AR, React, Bluetooth, and drones for those interested in exploring these topics.

1. Introduction to Controlling Apps with Your Mind

Short description:

Hi, everyone. Today, I want to talk about controlling apps with your mind and the future of UI and UX. We are in a transition phase, exploring new horizons and dimensions. We have the ability to adapt and transform things to something totally different. The foundations of UI and UX are based on 2D Medium, even though we have fake 3D elements.

Hi, everyone. I'm really excited to be here at React Summit Remote Edition, streaming from outer space. And today, I want to talk about controlling apps with your mind. My name is Vladimir Novik, I'm a software architect and consultant in Vladimir Novik Labs. I'm Google Developer Expert, author, engineer, and on a daily basis, I work in web, mobile, VR, AR, IoT, and AI fields.

I'm also CTO and co-founder of EventLoop, and we are creating robust, performant, feature-rich online conference experience. Basically, instead of having Zoom for your virtual conference, we bring you a set of conference tools and different plugins and widgets and so on that helps you to organize the conference, attend the conference. So if your organizer or speaker or attendee reaches out to us and signs up for our Alpha product, it will be an open source product. So if you want to collaborate, you are welcome. You can find us at eventloop.ai or on Twitter at eventloopHQ.

Today, I want to talk about the future. And I think we are in kind of transition phase technological-wise. When we are exploring new horizons, we're exploring new medium, we have VR, AR, mixed reality, web, mobile. Everything is constantly changing. And that's why we are sort of the ones who break the rules to make the new ones. Let's think about what medium we will use in the future. What dimensions we will use? Maybe VR? Maybe AR? Maybe something different. I'm just thinking here, but quantum computing is getting traction, VR is getting traction, AR is getting traction. Everything is kind of changing. Which dimensions we will use? Will there be web as it is today or mobile or will we completely switch to different medium? How we should prepare for that transition? How we should adapt? Should we adapt or should we actually shatter foundations that we have? Maybe invent new techniques to manipulate things, to interact differently? Maybe new UX patterns, maybe new best practices?

So, we have this ability right now to change things, to adapt and transform things to something totally different. And let's talk what are the very foundations of UI and what are the very foundations of UX. And I think it's 2D Medium. And if you think about that, we've gone a long way from cave drawings to the mobile apps. But if you think about all the color theory and lines and history of art and how everything kind of created the foundation of design and everything in our screens is basically 2D Medium. Because we're fake 3D. Third dimension. It's not really like third dimension. We have 3D models in the browser or in headsets on wherever, but all the like shapes and depth is based on shaders, which is basically a function of how light is reflected. So it's it's kind of fake, right? And we have things on our phones, we have screens. It's everything is 2D.

2. The Future of UI and UX in VR and AR

Short description:

XR is adding a new dimension to UI and UX in VR and AR. We need to create reality-based interactions in these mediums. Understanding the different dimensions and limitations is crucial. Adaptive UI and mind reading are also emerging trends.

And XR is adding a new dimension to that. So how we've adopted. We took different forms. Let's say you have a sign up form. So we have this form hovering in thin air in VR. Realistic? Not really. Something that we've adopted from 2D medium, right? Or we have in AR, we have arrows pointing to different directions that completely break immersion. But we have it because we've adopted. And we haven't invented something new.

So I think it's more crucial to create some sort of reality based interactions in VR and AR. And if you think about that, if you need to log in inside VR, so currently you have a login form, you have this hovering keyboard, you type on this keyboard, and then you get into, you type your username and password and you get inside. But is it something that you will see in reality? Not really, right? So it's more realistic to have some kind of password or key or whatever that you just put on the spot or turn the key inside the door, and it will let you through. It's a sort of reality-based interactions, right? So we need to understand our reality in order to create these interactions. And the medium is completely different.

Now, in VR there is another dimension, something happening behind the viewer. So, I'm looking at the camera, but something is happening behind. So I cannot use color theory to do this amazing call-to-action button animation. So I need to use different things like haptics, sounds, maybe slowing time, and so on. There is also an adaptive UI. Adaptive UI is something that is used on the web. And the idea is, UI, is kind of learning from what are you doing with it. So, like, forms are learning and adapting. So, you can google that. That's kind of a new trend-ish. And another thing that I propose is actual mind reading. And, yeah, obviously I cannot, like, read your thoughts, right? But, to some degree. And I want to ask you, what is it? And obviously, like, we're online. You can answer in chat. I'll pause. So, it's the known universe.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Don't Solve Problems, Eliminate Them
React Advanced Conference 2021React Advanced Conference 2021
39 min
Don't Solve Problems, Eliminate Them
Top Content
Humans are natural problem solvers and we're good enough at it that we've survived over the centuries and become the dominant species of the planet. Because we're so good at it, we sometimes become problem seekers too–looking for problems we can solve. Those who most successfully accomplish their goals are the problem eliminators. Let's talk about the distinction between solving and eliminating problems with examples from inside and outside the coding world.
Jotai Atoms Are Just Functions
React Day Berlin 2022React Day Berlin 2022
22 min
Jotai Atoms Are Just Functions
Top Content
Jotai is a state management library. We have been developing it primarily for React, but it's conceptually not tied to React. It this talk, we will see how Jotai atoms work and learn about the mental model we should have. Atoms are framework-agnostic abstraction to represent states, and they are basically just functions. Understanding the atom abstraction will help designing and implementing states in your applications with Jotai
Debugging JS
React Summit 2023React Summit 2023
24 min
Debugging JS
Top Content
As developers, we spend much of our time debugging apps - often code we didn't even write. Sadly, few developers have ever been taught how to approach debugging - it's something most of us learn through painful experience.  The good news is you _can_ learn how to debug effectively, and there's several key techniques and tools you can use for debugging JS and React apps.
Fighting Technical Debt With Continuous Refactoring
React Day Berlin 2022React Day Berlin 2022
29 min
Fighting Technical Debt With Continuous Refactoring
Top Content
Let’s face it: technical debt is inevitable and rewriting your code every 6 months is not an option. Refactoring is a complex topic that doesn't have a one-size-fits-all solution. Frontend applications are particularly sensitive because of frequent requirements and user flows changes. New abstractions, updated patterns and cleaning up those old functions - it all sounds great on paper, but it often fails in practice: todos accumulate, tickets end up rotting in the backlog and legacy code crops up in every corner of your codebase. So a process of continuous refactoring is the only weapon you have against tech debt. In the past three years, I’ve been exploring different strategies and processes for refactoring code. In this talk I will describe the key components of a framework for tackling refactoring and I will share some of the learnings accumulated along the way. Hopefully, this will help you in your quest of improving the code quality of your codebases.
The Epic Stack
React Summit US 2023React Summit US 2023
21 min
The Epic Stack
Top Content
Modern web development is fantastic. There are so many great tools available! Modern web development is exhausting. There are so many great tools available! Each of these sentiments is true. What's great is that most of the time, it's hard to make a choice that is wrong. Seriously. The trade-offs of most of the frameworks and tools you could use to build your application fit within the constraints of the vast majority of apps. Despite this, engineers consistently struggle with analysis paralysis.Let's talk about this, and a solution I am working on for it.
AHA Programming
React Summit Remote Edition 2020React Summit Remote Edition 2020
32 min
AHA Programming
Top Content
Are you the kind of programmer who prefers to never see the same code in two places, or do you make liberal use of copy/paste? Many developers swear the Don't Repeat Yourself (DRY) philosophy while others prefer to Write Everything Twice (WET). But which of these produces more maintainable codebases? I've seen both of these approaches lay waste to codebases and I have a new ideology I would like to propose to you: Avoid Hasty Abstractions (AHA). In this keynote, we'll talk about abstraction and how you can improve a codebase applying and creating abstractions more thoughtfully as well as how to get yourself out of a mess of over or under-abstraction.

Workshops on related topic

React, TypeScript, and TDD
React Advanced Conference 2021React Advanced Conference 2021
174 min
React, TypeScript, and TDD
Top Content
Featured WorkshopFree
Paul Everitt
Paul Everitt
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
Web3 Workshop - Building Your First Dapp
React Advanced Conference 2021React Advanced Conference 2021
145 min
Web3 Workshop - Building Your First Dapp
Top Content
Featured WorkshopFree
Nader Dabit
Nader Dabit
In this workshop, you'll learn how to build your first full stack dapp on the Ethereum blockchain, reading and writing data to the network, and connecting a front end application to the contract you've deployed. By the end of the workshop, you'll understand how to set up a full stack development environment, run a local node, and interact with any smart contract using React, HardHat, and Ethers.js.
Remix Fundamentals
React Summit 2022React Summit 2022
136 min
Remix Fundamentals
Top Content
Featured WorkshopFree
Kent C. Dodds
Kent C. Dodds
Building modern web applications is riddled with complexity And that's only if you bother to deal with the problems
Tired of wiring up onSubmit to backend APIs and making sure your client-side cache stays up-to-date? Wouldn't it be cool to be able to use the global nature of CSS to your benefit, rather than find tools or conventions to avoid or work around it? And how would you like nested layouts with intelligent and performance optimized data management that just works™?
Remix solves some of these problems, and completely eliminates the rest. You don't even have to think about server cache management or global CSS namespace clashes. It's not that Remix has APIs to avoid these problems, they simply don't exist when you're using Remix. Oh, and you don't need that huge complex graphql client when you're using Remix. They've got you covered. Ready to build faster apps faster?
At the end of this workshop, you'll know how to:- Create Remix Routes- Style Remix applications- Load data in Remix loaders- Mutate data with forms and actions
AI on Demand: Serverless AI
DevOps.js Conf 2024DevOps.js Conf 2024
163 min
AI on Demand: Serverless AI
Top Content
Featured WorkshopFree
Nathan Disidore
Nathan Disidore
In this workshop, we discuss the merits of serverless architecture and how it can be applied to the AI space. We'll explore options around building serverless RAG applications for a more lambda-esque approach to AI. Next, we'll get hands on and build a sample CRUD app that allows you to store information and query it using an LLM with Workers AI, Vectorize, D1, and Cloudflare Workers.
Vue3: Modern Frontend App Development
Vue.js London Live 2021Vue.js London Live 2021
169 min
Vue3: Modern Frontend App Development
Top Content
Featured WorkshopFree
Mikhail Kuznetcov
Mikhail Kuznetcov
The Vue3 has been released in mid-2020. Besides many improvements and optimizations, the main feature of Vue3 brings is the Composition API – a new way to write and reuse reactive code. Let's learn more about how to use Composition API efficiently.

Besides core Vue3 features we'll explain examples of how to use popular libraries with Vue3.

Table of contents:
- Introduction to Vue3
- Composition API
- Core libraries
- Vue3 ecosystem

Prerequisites:
IDE of choice (Inellij or VSC) installed
Nodejs + NPM
Developing Dynamic Blogs with SvelteKit & Storyblok: A Hands-on Workshop
JSNation 2023JSNation 2023
174 min
Developing Dynamic Blogs with SvelteKit & Storyblok: A Hands-on Workshop
Top Content
Featured WorkshopFree
Alba Silvente Fuentes
Roberto Butti
2 authors
This SvelteKit workshop explores the integration of 3rd party services, such as Storyblok, in a SvelteKit project. Participants will learn how to create a SvelteKit project, leverage Svelte components, and connect to external APIs. The workshop covers important concepts including SSR, CSR, static site generation, and deploying the application using adapters. By the end of the workshop, attendees will have a solid understanding of building SvelteKit applications with API integrations and be prepared for deployment.