Evaluating GraphQL for Setting Up an Enterprise-Grade Data Mesh

Rate this content
Bookmark

I'll introduce the concept of a data mesh and why people increasingly are gravitating towards that as a solution to their data (both online and analytical) and application modernization problems. I'll talk about the key requirements around 1) domain-oriented decentralized data ownership and architecture, 2) data as a product, 3) self-serve data infrastructure as a platform, and 4) federated computational governance (ref). And then I'll talk about how GraphQL opens up an opportunity for laying the foundation for a data mesh and best-practices in building, operating and maintaining an enterprise grade data mesh.

Tanmai Gopal
Tanmai Gopal
24 min
09 Dec, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

This Talk explores the benefits of using GraphQL APIs for data, including representing domain models, addressing API tooling and infrastructure, and supporting interconnected data. It discusses model mapping, read APIs, filtering, sorting, and aggregating data, as well as mutations for invoking logic. The Talk also highlights the importance of standardizing infrastructure and leveraging GraphQL's automatic model mapping and authorization. Additionally, it touches on the use of specialized databases, the comparison between GraphQL and gRPC for data APIs, and the speaker's experience working on various parts of the stack.

Available in Español

1. Introduction to GraphQL APIs for Data

Short description:

Hi folks, in this talk, I will discuss GraphQL APIs for data. We'll explore the challenges of fragmented data and the need for a comprehensive API. GraphQL offers benefits in representing domain models, providing various APIs, and addressing API tooling and infrastructure. It enables centralized concerns and supports interconnected data. By adopting a CQR-style thinking, we can design APIs with read models and commands. Bringing these ideas to GraphQL allows us to prioritize read models and handle data within the same or across different domains.

Hi folks, I'm going to talk to you a little bit about GraphQL APIs for data. My name is Tanya. I'm the CEO co-founder at Hasura. Hasura is an open source infrastructure project that allows you to create a GraphQL API easily on top of heterogeneous data sources. And a lot of the content of this talk is inspired by the work we do at Hasura and learnings from our users, especially in enterprise, as they mobilize their data and start to use GraphQL increasingly to make the experience around their data APIs much, much better.

So the rough context for this talk is that data is kind of increasingly getting fragmented and spread across multiple types of sources. And we need an API to start to access that. You might have data that is transactional data or data inside the SaaS service, or data that is more analytical. And within the same application context, or within the same user context, all of the data that is kind of split across these different types of sources is increasingly connected and relevant to the end user at the same time. And if you think about a simple ecommerce application, not only are you making transactions and interacting with data, but you're also searching through catalogs. You're also looking at aggregate analyzed information, and these are kinds of things that are all interconnected. And from an API experience point of view, you might need that to all come together into one API experience. When we think about this API, that's kind of the bottleneck because somebody needs to build this API. Somebody needs to talk to multiple types of sources, and you need to make sure that the API is continuously keeping up with the underlying evolution that's happening. And of course, you want kind of an API experience that guarantees certain amount of performance and security, right?

So in this context of having a data API, where data is getting fragmented, the stuff that we want, the properties that we want from a data API, are kind of the properties that we want for any API engine. But specifically, what we want to make sure is that we're able to represent our domain models, well, their evolution, their interconnectedness, right? We want to make sure that we have certain types of APIs that are again, usual for data APIs, whether it's queries or streaming kind of information, right? Certain ways of interacting with that data, methods on that data, and certain NFRs that are important to ensure that this data API is ready and usable for production. Most importantly, we want to make sure that the API tooling, the infrastructure for our API consumers is top-notch, right and keeps evolving. If we're able to kind of solve these problems, well, these types of problems, well, we essentially have a nice data API that is going to be easy to maintain and want to be easy to evolve, right?

So if you think there's lots of topics here, and so we're going to focus on mostly the AI and data modeling aspect of the API for the purpose of this talk, let's take a look at how GraphQL can kind of add value here when we think about the data API, right? Again, most of you would know, but just to kind of reframe some of those properties of GraphQL in the data API context. Our data is interconnected, so GraphQL represents that reality of a semantic graph of interconnected models much better than us thinking about resources that are siloed in, that are evolving kind of independently, right? GraphQL is JSON native, which is again nice from API consumption point of view, and offers us an opportunity to kind of centralize some of that infrastructure around kind of the JSON conversion-ness of when we're kind of interacting with data. GraphQL also has modern primitives around streaming in real time, which is again useful because data is starting to be less at rest and fair amount of our data is also in flight in real time, right? Then of course, GraphQL offers a GraphQL API always has a schema, a built-in artifact, which means that a lot of our challenges around API infrastructure, tooling, documentation, onboarding, are kind of automatically addressed, right? It's not 100% done, but a huge part of it that we would have to do manually if we weren't using GraphQL is kind of addressed, right? So, it offers certain benefits around what we'd want from our data API and offers certain opportunities to centralize some of those concerns and the way that we think about our data that is split across a mesh of sources.

Typically, when we're designing an API, the way that we would think about it is that we'd have models, we'd have getters or accessors of those models, right? So, if I had artists and albums in the music database, I would have a get method on an artist, right? Or where I might specify an ID as a parameter, right? I would have a list of artists that I fetched with a pagination parameter, right? And then I would have certain commands or methods that I would invoke that affect those models, right? And if we were thinking of it more as a CRUD-style thing, then we would have a model. The same model is a read model and a write model. So, we might post, put, patch, delete a model or we might get that model, right? But if you're thinking about it more from a CQRS point of view, then we'd kind of think of it more as, you know, we have commands that we're invoking and then we kind of invoke commands. Those can be create user, create artist, process artist, add album to artist, right? Or add label to artist, or whatever. And those would be kind of commands that we invoke and our models would be read models that we would kind of expect our commands to affect. And then we would read data from those models, right? And so, that might be a slightly different idea. Now, when we kind of bring these ideas to GraphQL, the way that I like to think about this more is kind of bringing that kind of CQR style thinking into our API design. So, we think of read models first. So, we have read models. Read models can be within the same source or within the same domain, data domain, or split across data domains.

2. Model Mapping and Read API

Short description:

And these would have connections, right? We still haven't defined the methods and that's what we'll come to next. The first category of methods that we can have are queries, right? And because it's a data API, there'll be some common expectations and conventions that we can set up. And now, when we think about the right side of our API, especially when we think about graphical mutations, then the way that we think about it is that, hey, we invoke a command. So at this layer, at the GraphQL API layer, we can start to centralize some infrastructure in the way that we map these models, bring them into types, in the way that we create common read APIs across these domains, and the way that we invoke certain commands. So if you're able to centralize the infrastructure here, right, then we can kind of extract all the benefits that people get out of a GraphQL API, we can centralize a lot of the effort that we're putting into building the API. So what we're going to dive into next is just kind of going a little bit deeper into that model mapping into the APIs, the read API, and write API a little bit, just to see what that might look like. The second thing that we want to do to make sure that we want to represent is to make sure that we can model relationships between these models that we have. And because it's finally going to be JSON, the way that relationship will show up in the API is going to be, you know, maybe I have a nested added a nested object, so you have like many to want to be a nested object, or want to be a nested array, right? Or you can even have fields that are actually a relationship that is coming in from a different model. And, and each of those models should also have authorization tools that determine what fields are accessible, and what entities of that model are actually readable by a particular API consumer. So, that is that, this, these three pieces of work, mapping, relationships, and authorizations, policies, right? This is kind of the bare minimum information that we need to bring a model from our data domain, right, from one of our data sources, right, into the API. The second piece is read. So, what we want to do is create conventions around how we're going to read these models, right? So, we can create conventions around pagination, around ordering and filtering, right? We want to create a convention on aggregations, and then we'd want to compose them with a relationship, we'd want to represent those relationships in our queries, right? So, we'd want to be able to read models across relationships. We'd want to also make sure that our queries can represent aggregations, or ordering or filtering that can reference parent and child objects. So, let's take a quick look at, as an example, I'll just show you kind of the way we think about it at Hasura, and what that read API looks like, what that read convention looks like when we think about it with Hasura, the way that we think about it at Hasura, right?

And these would have connections, right? These would have semantic relationships within the domain or across domains. And these would essentially become the types in our GraphQL schema, right? We still haven't defined the methods and that's what we'll come to next.

The first category of methods that we can have are queries, right? And because it's a data API, there'll be some common expectations and conventions that we can set up around, are we fetching a single element? A single model? A list of models, paginating through them, ordering them, filtering through them. Aggregations especially are important for data APIs, right? So, we can start to create kind of a convention that says, once we have a read model, these are the common set of methods that we can have for reading those models, right? And that kind of addresses the query side of our API.

And now, when we think about the right side of our API, especially when we think about graphical mutations, right? Maybe certain queries as well, when we think about graphical mutations, then the way that we think about it is that, hey, we invoke a command. The command will go do something, but eventually it'll return a reference to a read model or to a list of read models, right? And that kind of gives us a way to think about designing the right side of our API, the graphical mutation side especially. Although like I said, some of it might, some commands might also be query commands. But the same thing, same idea, just revisualizing that in the context of multiple domains, right?

We might have multiple domains, each domain has models, commands, right? And we use that to create our GraphQL API, right? So at this layer, at the GraphQL API layer, we can start to centralize some infrastructure in the way that we map these models, bring them into types, in the way that we create common read APIs across these domains, and the way that we invoke certain commands. So if you're able to centralize the infrastructure here, right, then we can kind of extract all the benefits that people get out of a GraphQL API, we can centralize a lot of the effort that we're putting into building the API. And we can focus more within each of those domains, we can focus on the modeling, we can focus on the actual logic that might run and affect those models, right?

So what we're going to dive into next is just kind of going a little bit deeper into that model mapping into the APIs, the read API, and write API a little bit, just to see what that might look like. Right? So let's get the models, when we think about models, the most important thing is to be able to map kind of what we want on the on the model that we want to expose, the models that we have within our domain, right? So if you have a database type domain, right, we have to believe physical models, these data models might be physical models, they might be tables, right? Or logical models, where the exact data in the table doesn't actually correlate the model that you want to have in the API, right? So you might have a physical model or a logical model. But these are basically getting represented to a graphical model that you'd want at the end of the day. If it's not a database, but it's more like a source, you might have a model that's coming in from an API service, right? So the mapping layer that we have in our graphical infrastructure, code framework configuration, whatever it is, should should kind of help solve this mapping problem to bring in certain models from a from a data domain, from a data source as easily as we can.

The second thing that we want to do to make sure that we want to represent, right, that we want to add configuration around or code around is to make sure that we can model relationships between these models that we have, right. And because it's finally going to be JSON, the way that relationship will show up in the API is going to be, you know, maybe I have a nested added a nested object, so you have like many to want to be a nested object, or want to be a nested array, right? Or you can even have fields that are actually a relationship that is coming in from a different model. Right? And, and each of those models should also have authorization tools that determine what fields are accessible, and what entities of that model are actually readable by a particular API consumer, right? So, that is that, this, these three pieces of work, mapping, relationships, and authorizations, policies, right? This is kind of the bare minimum information that we need to bring a model from our data domain, right, from one of our data sources, right, into the API, right, and at this point we now have a nice way to think about the types of our GraphQL schema and the way those types will show up, right? Not all of the types, some of the types, right, the way that that will show up. So, that's kind of the first piece of the data API that we've kind of gotten sorted.

And if you look at this example that I have here where I have type artists, right? I'm seeing that my GraphQL model is id and name. In the physical model it might actually be first name plus last name, my API model, my GraphQL model I would like to be named, that's my read model, right? And then I might have albums, right, of the artist as a label, the artist is signed on label, the label might have its own information, right? So, that's nested object. The artist has multiple albums, so that'd be a nested array. So, albums and label would be kind of different models and have relationships to the albums model and to the label model, right? And here is where I've done the mapping of the artist model itself. So, we're seeing the mapping portion, we're seeing relationships, right? And then of course, we have authorization rules that determine what fields we can actually access. So, this is kind of how it will show up in the API schema, which is nice because now we can see kind of what our different interconnected models are, right?

The second piece is read. So, what we want to do is create conventions around how we're going to read these models, right? So, we can create conventions around pagination, around ordering and filtering, right? We want to create a convention on aggregations, and then we'd want to compose them with a relationship, we'd want to represent those relationships in our queries, right? So, we'd want to be able to read models across relationships. We'd want to also make sure that our queries can represent aggregations, or ordering or filtering that can reference parent and child objects, right? So, let's take a quick look at, as an example, I'll just show you kind of the way we think about it at Hasura, and what that read API looks like, what that read convention looks like when we think about it with Hasura, the way that we think about it at Hasura, right?

So, I have here a demo app that's running on my machine, and I've connected in a Postgres source and a SQL server source. These are kind of my two data domains. And I have a model called artists, which has ID and name, right? So, we're seeing kind of that information coming here. Let's take a look at what the API looks like. So, I have pagination, right? So, offset-based pagination or cursor-based pagination. We have a way of filtering stuff.

QnA

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

From GraphQL Zero to GraphQL Hero with RedwoodJS
GraphQL Galaxy 2021GraphQL Galaxy 2021
32 min
From GraphQL Zero to GraphQL Hero with RedwoodJS
Top Content
We all love GraphQL, but it can be daunting to get a server up and running and keep your code organized, maintainable, and testable over the long term. No more! Come watch as I go from an empty directory to a fully fledged GraphQL API in minutes flat. Plus, see how easy it is to use and create directives to clean up your code even more. You're gonna love GraphQL even more once you make things Redwood Easy!
Local State and Server Cache: Finding a Balance
Vue.js London Live 2021Vue.js London Live 2021
24 min
Local State and Server Cache: Finding a Balance
Top Content
How many times did you implement the same flow in your application: check, if data is already fetched from the server, if yes - render the data, if not - fetch this data and then render it? I think I've done it more than ten times myself and I've seen the question about this flow more than fifty times. Unfortunately, our go-to state management library, Vuex, doesn't provide any solution for this.For GraphQL-based application, there was an alternative to use Apollo client that provided tools for working with the cache. But what if you use REST? Luckily, now we have a Vue alternative to a react-query library that provides a nice solution for working with server cache. In this talk, I will explain the distinction between local application state and local server cache and do some live coding to show how to work with the latter.
Batteries Included Reimagined - The Revival of GraphQL Yoga
GraphQL Galaxy 2021GraphQL Galaxy 2021
33 min
Batteries Included Reimagined - The Revival of GraphQL Yoga
The Guild has recently released Envelop - a new, modern GraphQL Server Framework and plugin system. In this talk I’ll share a brief overview of Envelop and why you should probably upgrade your existing GraphQL server to it.
Rock Solid React and GraphQL Apps for People in a Hurry
GraphQL Galaxy 2022GraphQL Galaxy 2022
29 min
Rock Solid React and GraphQL Apps for People in a Hurry
In this talk, we'll look at some of the modern options for building a full-stack React and GraphQL app with strong conventions and how this can be of enormous benefit to you and your team. We'll focus specifically on RedwoodJS, a full stack React framework that is often called 'Ruby on Rails for React'.
Step aside resolvers: a new approach to GraphQL execution
GraphQL Galaxy 2022GraphQL Galaxy 2022
16 min
Step aside resolvers: a new approach to GraphQL execution
Though GraphQL is declarative, resolvers operate field-by-field, layer-by-layer, often resulting in unnecessary work for your business logic even when using techniques such as DataLoader. In this talk, Benjie will introduce his vision for a new general-purpose GraphQL execution strategy whose holistic approach could lead to significant efficiency and scalability gains for all GraphQL APIs.

Workshops on related topic

Build with SvelteKit and GraphQL
GraphQL Galaxy 2021GraphQL Galaxy 2021
140 min
Build with SvelteKit and GraphQL
Top Content
Featured WorkshopFree
Scott Spence
Scott Spence
Have you ever thought about building something that doesn't require a lot of boilerplate with a tiny bundle size? In this workshop, Scott Spence will go from hello world to covering routing and using endpoints in SvelteKit. You'll set up a backend GraphQL API then use GraphQL queries with SvelteKit to display the GraphQL API data. You'll build a fast secure project that uses SvelteKit's features, then deploy it as a fully static site. This course is for the Svelte curious who haven't had extensive experience with SvelteKit and want a deeper understanding of how to use it in practical applications.

Table of contents:
- Kick-off and Svelte introduction
- Initialise frontend project
- Tour of the SvelteKit skeleton project
- Configure backend project
- Query Data with GraphQL
- Fetching data to the frontend with GraphQL
- Styling
- Svelte directives
- Routing in SvelteKit
- Endpoints in SvelteKit
- Deploying to Netlify
- Navigation
- Mutations in GraphCMS
- Sending GraphQL Mutations via SvelteKit
- Q&A
End-To-End Type Safety with React, GraphQL & Prisma
React Advanced Conference 2022React Advanced Conference 2022
95 min
End-To-End Type Safety with React, GraphQL & Prisma
Featured WorkshopFree
Sabin Adams
Sabin Adams
In this workshop, you will get a first-hand look at what end-to-end type safety is and why it is important. To accomplish this, you’ll be building a GraphQL API using modern, relevant tools which will be consumed by a React client.
Prerequisites: - Node.js installed on your machine (12.2.X / 14.X)- It is recommended (but not required) to use VS Code for the practical tasks- An IDE installed (VSCode recommended)- (Good to have)*A basic understanding of Node.js, React, and TypeScript
GraphQL for React Developers
GraphQL Galaxy 2022GraphQL Galaxy 2022
112 min
GraphQL for React Developers
Featured Workshop
Roy Derks
Roy Derks
There are many advantages to using GraphQL as a datasource for frontend development, compared to REST APIs. We developers in example need to write a lot of imperative code to retrieve data to display in our applications and handle state. With GraphQL you cannot only decrease the amount of code needed around data fetching and state-management you'll also get increased flexibility, better performance and most of all an improved developer experience. In this workshop you'll learn how GraphQL can improve your work as a frontend developer and how to handle GraphQL in your frontend React application.
Build a Headless WordPress App with Next.js and WPGraphQL
React Summit 2022React Summit 2022
173 min
Build a Headless WordPress App with Next.js and WPGraphQL
Top Content
WorkshopFree
Kellen Mace
Kellen Mace
In this workshop, you’ll learn how to build a Next.js app that uses Apollo Client to fetch data from a headless WordPress backend and use it to render the pages of your app. You’ll learn when you should consider a headless WordPress architecture, how to turn a WordPress backend into a GraphQL server, how to compose queries using the GraphiQL IDE, how to colocate GraphQL fragments with your components, and more.
Relational Database Modeling for GraphQL
GraphQL Galaxy 2020GraphQL Galaxy 2020
106 min
Relational Database Modeling for GraphQL
Top Content
WorkshopFree
Adron Hall
Adron Hall
In this workshop we'll dig deeper into data modeling. We'll start with a discussion about various database types and how they map to GraphQL. Once that groundwork is laid out, the focus will shift to specific types of databases and how to build data models that work best for GraphQL within various scenarios.
Table of contentsPart 1 - Hour 1      a. Relational Database Data Modeling      b. Comparing Relational and NoSQL Databases      c. GraphQL with the Database in mindPart 2 - Hour 2      a. Designing Relational Data Models      b. Relationship, Building MultijoinsTables      c. GraphQL & Relational Data Modeling Query Complexities
Prerequisites      a. Data modeling tool. The trainer will be using dbdiagram      b. Postgres, albeit no need to install this locally, as I'll be using a Postgres Dicker image, from Docker Hub for all examples      c. Hasura
Building GraphQL APIs on top of Ethereum with The Graph
GraphQL Galaxy 2021GraphQL Galaxy 2021
48 min
Building GraphQL APIs on top of Ethereum with The Graph
WorkshopFree
Nader Dabit
Nader Dabit
The Graph is an indexing protocol for querying networks like Ethereum, IPFS, and other blockchains. Anyone can build and publish open APIs, called subgraphs, making data easily accessible.

In this workshop you’ll learn how to build a subgraph that indexes NFT blockchain data from the Foundation smart contract. We’ll deploy the API, and learn how to perform queries to retrieve data using various types of data access patterns, implementing filters and sorting.

By the end of the workshop, you should understand how to build and deploy performant APIs to The Graph to index data from any smart contract deployed to Ethereum.