Scaling Databases For Global Serverless Applications

Rate this content
Bookmark

This workshop discusses the challenges Enterprises are facing when scaling the data tier to support multi-region deployments and serverless environments. Serverless edge functions and lightweight container orchestration enables applications and business logic to be easily deployed globally, often leaving the database as the latency and scaling bottleneck.


Join us to understand how PolyScale.ai solves these scaling challenges intelligently caching database data at the edge, without sacrificing transactionality or consistency. Get hands on with PolyScale for implementation, query observability and global latency testing with edge functions.


Table of contents

  • - Introduction to PolyScale.ai
  • - Enterprise Data Gravity
  • - Why data scaling is hard
  • - Options for Scaling the data tier
  • - Database Observability
  • - Cache Management AI
  • - Hands on with PolyScale.ai

FAQ

Polyscale is a no-code database edge cache that distributes data and query compute closer to the user. It functions by caching database data at the edge or closer to the user and also offers the compute itself. This means if you have a unique SQL query, it will execute against Polyscale's Points of Presence (POPs) and return the data from Polyscale if it is in the cache.

To connect a database to Polyscale, update your database credentials such as the host name and database username to route through Polyscale. For MySQL and MariaDB, include your cache ID in the database username. For Postgres, pass the cache ID as the application name parameter.

As of the information provided, Polyscale currently supports MySQL, MariaDB, and Postgres. Support for Microsoft SQL Server is planned and forthcoming, while MongoDB support is also under consideration for future updates.

Polyscale can potentially replace Redis or other application-level caching solutions, especially in scenarios where you want to reduce complexity and avoid managing caching logic at the application level. Polyscale manages caching at the data tier, which can simplify operations.

Polyscale caches DQL (Data Query Language) queries, which include 'select' and 'show' queries. It automatically handles and optimizes caching for these queries based on frequency and other factors using AI and machine learning technologies.

Polyscale ensures data security by not storing database usernames or passwords, and all data remains SSL encrypted during transit. Users can also deploy Polyscale within their own networks for enhanced compliance and control, ensuring that cached data never leaves their environment.

Yes, Polyscale allows manual configuration of caching rules where you can specify custom rules for individual SQL queries or set default behaviors for caching. This provides flexibility in managing how data is cached based on specific application needs.

Polyscale offers significant benefits for global applications by reducing latency and improving data access speed across different regions. It caches data closer to users and distributes query compute globally, which helps in maintaining high performance even when scaling applications internationally.

Ben Hagan
Ben Hagan
83 min
11 Feb, 2022

Comments

Sign in or register to post your comment.

Video Summary and Transcription

Polyscale is a no-code database edge cache that aims to make caching and speeding up the data layer effortless for developers. It uses AI and machine learning to adjust caching behavior, selectively choosing what to cache and automatically setting cache duration. Polyscale supports multiple databases and provides detailed observability. It eliminates the need for caching at the application tier and offers significant performance gains.

1. Introduction to Polyscale and Scaling Data

Short description:

Welcome to this Node Congress talk on scaling databases for global serverless applications. Today, I'm going to introduce you to Polyscale and talk about the challenges in managing and scaling data. We'll also explore different options for scaling database systems and dig into how Polyscale works. Feel free to ask questions and get hands-on with the code.

♪♪ Yeah, welcome, everybody. I can still see a few people coming in, but we'll get kicked off anyway, so welcome to this Node Congress talk on scaling databases for global serverless applications. Let's just show the agenda for today. Just a couple of housekeeping rules, everybody feel free to use the chat, you can also unmute, there's quite a lot of people on this, so stay muted if you're not talking, but please unmute yourself, ask questions, let's make this as interactive as possible, because I think there's lots we can learn by collaborating our questions.

So, yeah, feel free to use the chat, but obviously, unmute and introduce yourself and, by all means, jump in at any point if you have questions. Those are very welcome. So, yeah, hopefully, everybody can hear me OK, and this is our agenda for today. A quick bit about myself, I'm the founder of PolyScale and my background's really been in working with sort of large data companies and database companies. And I've sort of been in the solution architecture and sales engineering space for many years. And sort of the last few companies I was at, you know, Alfresco was an open source document management company. So we dealt with very large amounts of data repositories for specifically the documents of web content management. DataSift was one of the early adopters of the Twitter firehose and ingested all of Twitter's data and later on, that worked with Facebook and LinkedIn to ingest all of that data and serve insights in a privacy safe way. And then from there, I moved to Elastic here in the Bay Area, in San Francisco, where I was working with some large enterprises with the Elastic Solution and scaling those and then before founding Polyscale, I was at a company called Rockset, which is a cloud-based Database for realtime analytics. So throughout those last few years, I've always been focused on managing large volumes of data and understanding how to scale those globally. That's really been the challenges I've been working on.

But today, we've got a great talk lined up. I'm going to introduce you to Polyscale and we'll get hands on with that for people who want to actually test it out. And I'll talk a bit about the challenges really in the enterprise and small businesses as well around managing data and scaling data. And why is that hard? Why is it hard to move data to different regions and to keep and maintain your transactionality? And so we're also going to talk about what are your options, what do people do today to scale these database systems and data platforms. And then, as I mentioned, we'll dig into Polyscale and what Polyscale is and how it works. And as I say, everyone's welcome to get hands on. We've got some code to play with and get started with. So, yeah, as I say, everybody who would, you know, any questions, feel free to chime in, unmute yourself, introduce yourself, you know, very welcome to take questions.

2. Challenges of Scaling Data and Data Gravity

Short description:

Polyscale exists to address the challenges of scaling stateful cloud-native data. Scaling databases to different countries or independently presents difficulties, from CAP theorem considerations to maintaining transactionality and managing data sharding. Accessing data across continents in a low latency environment is also complex. Data gravity, the creation of data across multiple systems, poses challenges for enterprises. Applications and users require access to data from different locations, making data movement difficult. Location, data regulations, and global growth further complicate the situation. However, there are options for deploying web apps and replicating static content and business logic locally with companies like Netlify and Vercel. CDNs and containerization tools like Fly and Section provide flexibility in deploying to multiple regions.

So, really, the premise for, you know, why Polyscale exists and why, you know, our focus is really scaling stateful cloud-native data is hard. So, anyone who's been in a role where you've had to, you know, take a database, for example, and scale that to different countries, or even just scale it independently, you know, you'll understand the challenges of doing that. And there's a whole bunch of reasons around like why that is hard, from things along the CAP theorem lines to, you know, maintaining my transactionality, you know, how do I shard my data, where does that reside? And then how do I sort of manage just the laws of physics around accessing that in a low latency environment across continents, for example?

So, great example that we've got people from all parts of the planet dialed in today. And making data fast for everyone is a hard and complex thing, and specifically here, we're talking also about cloud native data. So, what happens when you are in a situation where, you know, you have a data center outage, where one region goes down, or a part or component of your architecture goes down. You know, how do you support those environments? And that's what makes it hard. It's maintaining that state and it's doing that in an unforgiving environment like the cloud.

So, you know, I talk a lot about this concept of enterprise data gravity. And you know, kind of what is that, and how does it affect us? The reality is data is being created everywhere. And it can be across multiple disparate systems. And inside a single enterprise, there are likely many, many different data persistence layers. It could be things from data lakes, databases, warehouses. And that stuff is being created all day every day. It's just an ongoing thing. With sort of AI and machine learning, sensor data and IoT, that's just accelerating massively. So what tends to happen is, within enterprises, more and more applications want to use that data. And people are requesting access to that data from different departments, from different locations. And that then gives us this inability to move the data. We find ourselves kind of shackled to, once somebody is using... Let's say we have a large transactional database that says Postgres, and that's running from a data center in New York. And we now have, you know, large numbers of users actually actively using that. It becomes harder and harder to kind of access that data in different locations and move that data. So location becomes critical depending on where your users are. And then the other sort of big consideration around data and usages, obviously, what are the data regulations around this? Things like privacy and compliance, and PII data, what are you storing in what countries? You know, is it GDPR compliant? And this is becoming more and more what is incredibly critical today, but it's only becoming more critical as we as we progress with these data systems. And then lastly, you know, we've got to consider global growth. So it could be a simple web application. We find that we now have a growing audience that are spanning different countries or parts of countries. And how do we make sure they can access that data and how do we deal with that global growth? And the same within sort of a non-customer facing application. Maybe it's something like a be a environment whereby people are accessing internal intelligence data using tools like click tech and tableau. So same types of problems. We're putting new offices in different locations. People are working from home, working remotely. How do we support that growth? And that's really the essence of the problems that come with sort of data gravity.

I think probably a lot of people on this this talk will be very familiar with the application tier has a lot of agility. And so I think that's all the time I have today. Like if I said to everyone on this call, like, how would you deploy a web app? I think we have a lot of different options. And if I said, how do I deploy a web app to different regions? I think there's a huge number of options as well. And I kind of think about this, the content and the code. So static content and business logic can easily be replicated locally. And when I think about, if I had a server where we have these amazing companies like, you know, Netlify and Vercel, who can, you know, push my application logic and my static content everywhere. And I can run functions at the edge. And I can knit that together with, you know, a simple load balancing plan that will allow me to resolve to my closest application server. So it's really quite achievable now for any size of business or individual. Just so, you know, an indie hacker to be able to set up global load balancing. We can deploy to CDNs for our static content and our functions. And we're also seeing the ability to, you know, these new players pop up like Fly and Section who allow you to containerize your code and push that everywhere into multiple regions. So as far as the business logic and your static content goes, there really is a huge amount of flexibility. And that's a great space to be in. I'll just pause there. Any questions before we move on? Anything in the chat or anyone wants to chime in with a question? Good.

QnA

Watch more workshops on topic

AI on Demand: Serverless AI
DevOps.js Conf 2024DevOps.js Conf 2024
163 min
AI on Demand: Serverless AI
Top Content
Featured WorkshopFree
Nathan Disidore
Nathan Disidore
In this workshop, we discuss the merits of serverless architecture and how it can be applied to the AI space. We'll explore options around building serverless RAG applications for a more lambda-esque approach to AI. Next, we'll get hands on and build a sample CRUD app that allows you to store information and query it using an LLM with Workers AI, Vectorize, D1, and Cloudflare Workers.
Node.js Masterclass
Node Congress 2023Node Congress 2023
109 min
Node.js Masterclass
Top Content
Workshop
Matteo Collina
Matteo Collina
Have you ever struggled with designing and structuring your Node.js applications? Building applications that are well organised, testable and extendable is not always easy. It can often turn out to be a lot more complicated than you expect it to be. In this live event Matteo will show you how he builds Node.js applications from scratch. You’ll learn how he approaches application design, and the philosophies that he applies to create modular, maintainable and effective applications.

Level: intermediate
Build and Deploy a Backend With Fastify & Platformatic
JSNation 2023JSNation 2023
104 min
Build and Deploy a Backend With Fastify & Platformatic
WorkshopFree
Matteo Collina
Matteo Collina
Platformatic allows you to rapidly develop GraphQL and REST APIs with minimal effort. The best part is that it also allows you to unleash the full potential of Node.js and Fastify whenever you need to. You can fully customise a Platformatic application by writing your own additional features and plugins. In the workshop, we’ll cover both our Open Source modules and our Cloud offering:- Platformatic OSS (open-source software) — Tools and libraries for rapidly building robust applications with Node.js (https://oss.platformatic.dev/).- Platformatic Cloud (currently in beta) — Our hosting platform that includes features such as preview apps, built-in metrics and integration with your Git flow (https://platformatic.dev/). 
In this workshop you'll learn how to develop APIs with Fastify and deploy them to the Platformatic Cloud.
0 to Auth in an Hour Using NodeJS SDK
Node Congress 2023Node Congress 2023
63 min
0 to Auth in an Hour Using NodeJS SDK
WorkshopFree
Asaf Shen
Asaf Shen
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.JS backend + React frontend) to authenticate users with OAuth (social login) and One Time Passwords (email), including:- User authentication - Managing user interactions, returning session / refresh JWTs- Session management and validation - Storing the session for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
Table of contents- A quick intro to core authentication concepts- Coding- Why passwordless matters
Prerequisites- IDE for your choice- Node 18 or higher
Building a Hyper Fast Web Server with Deno
JSNation Live 2021JSNation Live 2021
156 min
Building a Hyper Fast Web Server with Deno
WorkshopFree
Matt Landers
Will Johnston
2 authors
Deno 1.9 introduced a new web server API that takes advantage of Hyper, a fast and correct HTTP implementation for Rust. Using this API instead of the std/http implementation increases performance and provides support for HTTP2. In this workshop, learn how to create a web server utilizing Hyper under the hood and boost the performance for your web apps.
GraphQL - From Zero to Hero in 3 hours
React Summit 2022React Summit 2022
164 min
GraphQL - From Zero to Hero in 3 hours
Workshop
Pawel Sawicki
Pawel Sawicki
How to build a fullstack GraphQL application (Postgres + NestJs + React) in the shortest time possible.
All beginnings are hard. Even harder than choosing the technology is often developing a suitable architecture. Especially when it comes to GraphQL.
In this workshop, you will get a variety of best practices that you would normally have to work through over a number of projects - all in just three hours.
If you've always wanted to participate in a hackathon to get something up and running in the shortest amount of time - then take an active part in this workshop, and participate in the thought processes of the trainer.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

It's a Jungle Out There: What's Really Going on Inside Your Node_Modules Folder
Node Congress 2022Node Congress 2022
26 min
It's a Jungle Out There: What's Really Going on Inside Your Node_Modules Folder
Top Content
Do you know what’s really going on in your node_modules folder? Software supply chain attacks have exploded over the past 12 months and they’re only accelerating in 2022 and beyond. We’ll dive into examples of recent supply chain attacks and what concrete steps you can take to protect your team from this emerging threat.
You can check the slides for Feross' talk here.
Towards a Standard Library for JavaScript Runtimes
Node Congress 2022Node Congress 2022
34 min
Towards a Standard Library for JavaScript Runtimes
Top Content
You can check the slides for James' talk here.
ESM Loaders: Enhancing Module Loading in Node.js
JSNation 2023JSNation 2023
22 min
ESM Loaders: Enhancing Module Loading in Node.js
Native ESM support for Node.js was a chance for the Node.js project to release official support for enhancing the module loading experience, to enable use cases such as on the fly transpilation, module stubbing, support for loading modules from HTTP, and monitoring.
While CommonJS has support for all this, it was never officially supported and was done by hacking into the Node.js runtime code. ESM has fixed all this. We will look at the architecture of ESM loading in Node.js, and discuss the loader API that supports enhancing it. We will also look into advanced features such as loader chaining and off thread execution.
Out of the Box Node.js Diagnostics
Node Congress 2022Node Congress 2022
34 min
Out of the Box Node.js Diagnostics
In the early years of Node.js, diagnostics and debugging were considerable pain points. Modern versions of Node have improved considerably in these areas. Features like async stack traces, heap snapshots, and CPU profiling no longer require third party modules or modifications to application source code. This talk explores the various diagnostic features that have recently been built into Node.
You can check the slides for Colin's talk here. 
You Don’t Know How to SSR
DevOps.js Conf 2024DevOps.js Conf 2024
23 min
You Don’t Know How to SSR
A walk-through of the evolution of SSR in the last twelve years. We will cover how techniques changed, typical problems, tools you can use and various solutions, all from the point of view of my personal experience as a consumer and maintainer.