So, hello everybody. Thank you for taking the time to listen to this talk. We are going to talk about a few things. So first of all, we are going to talk about our
database, our search engine, Weaviate. And we are going to use a demo dataset, which is the complete Wikipedia, to show how you can query it. And most importantly, of course, we are going to talk about the
graphql api that it has. So Weaviate is a vector search engine or
database. It has a
graphql api. And we are going to use it to show you the demo dataset of the complete Wikipedia. So first, I will give a little bit of context about what the vector search engine is, so that you understand what we are talking about, if it's new to you. Then we will look at the
design of the
graphql api. Then we'll go into a demo of the
api on the dataset. And last but not least, I'll show you how you can start it with Weaviate and its
graphql api yourself. So again, thanks for listening. So first of all, what is Weaviate and what is a vector search engine? So at the core, we are dealing with the problem of unstructured
data. And if you ever use a
database or if you ever use a search engine, then you know that the
data that you're storing, for example, if it is text, that you can only find it if you use keywords. So for example, in a traditional search engine, if you search for this
data object for wine, for seafood, you will probably not find it. Because except for the key here, there's nowhere where you find the word wine in the
data. The word for is not in there either, and seafood is not in there either. So using a vector search engine, and you would search wine for seafood, it would actually find the
data object. And the reason why it is able to do that is because every
data object that you add to the search engine is run through a
machine learning model. The
machine learning model creates vector representations, and that's what you use to search to the
database. Now if this is new to you, then let me give you a little bit of context so that you know what's happening there. So most
machine learning models output vectors, and the easiest way to think about vectors are coordinates. So for example, our first model had 300 dimensions, and yet all these kinds of words in there. So the bulbs here represent words like meat, chicken, fish, etc. What you can do if you add a new
data object, for example, the chardonnay that's good with seafood, is that all these individual words that you see here highlighted in green are found in the vector space, and they're placed in that same vector space. And what you can do is that you can give a unique centroid position to the
data object. So now you can say in the vector space, the
data object of the, in this case, the chardonnay, sits exactly here in the middle of where all these words sit. So now if you search for wine related to seafood or those kinds of things, you will actually be able to find that
data object. It is not 100% match, but it's an approximation of what you're searching for. But in a bit, you will see what actually the value is of this. So as you see here, we have the class wine with the property Covi run 2005 chardonnay. It might be related to a beacon and it might have certain vector weights. So this is what the
data object looks like when you store it in a Weaviate instance. Well, to help you work with this, we have the
database, which you see in the middle to store your objects, to do vector search and to do filtering. But of course, there are many, many
machine learning models that you can use to actually or vectorize the
data or search through the
data. The demo that I'm going to give today is purely focusing on text. However, you could also do this for images or videos or any other
data type. If you go a little bit deeper under the hood, you see how that works from an architectural point of view. So for example, we have text to VAC
modules or we have Q&A
modules. They often run on a GPU. That's all running on your infrastructure. These
modules sit in the Weaviate core. Then there's a persistence layer that's taking care of storing the vectors, being able to search through the vectors and to store the
data object. But most important, there is an
api on top of it. And of course, what we're going to focus on today is the
graphql api and how you can leverage to search through your
data. So first, before we do that, I want to talk a little bit about the
design of the
graphql api, because you have to know when we created the
database that we didn't have an interface yet. So we had to choose like what language will we choose to query
data? Will we just have a pure RESTful
api? Will we adopt some kind of query language? Will we invent something of our own? And then we decided that the best for us was actually to use
graphql. And this is in a tiny nutshell, our
design. So at the top, you have a core function within Weaviate. We'll look at that in a bit. You have a class that you can add and add your
data to. A class can be anything. So whatever
data you can have, for example, if you have documents, you can just have a class document. Or if you have products, you can have a class product. Then you have the properties. So a property can be also anything. So for example, if we stay with the class product, then you might have the property name or the property price. You can of course make a cross-reference, hence it's a graph-like
data model. And then we have these underscore additional properties. Those are properties that you get as part of searching for classes. But those are baked in in the
modules or into Weaviate itself. So we have three core functions. So get, explore, and aggregate. Get is the one that we'll be using the most, because that's how we find stuff in our dataset. Explore is to search through the complete vector space. This is often done if you don't know what your classes or properties are called. And aggregate is just, for example, how many
data objects do I have, et cetera. So of course, what you can also do is that for the class name, you can add search filters. So here, for example, if you have the class article, the property title, you can add a near-text filter that comes from the text-to-vect
modules. And you say, OK, I want to search for the concepts, in this case, housing prices. So that is at the root what the
graphql design is like from Weaviate. And I think it's best to actually start to look at the demo, because what's better than actually looking at it in action? So for all the demo datasets that we have, we have a console that you can go to, console.semideattechnology. And embedded in that console is, among other things,
graphql. That's something that we'll be using here. Of course, it's a
database, so I need to select a
database, a dataset. And the dataset that is here is actually Wikipedia. So this is open source. It's on GitHub. And it currently contains a little over 11 million articles, a little over 27 million paragraphs, and a little over 125 million cross-references. And this is the machine that it's running on. In this GitHub repo, you'll find all the information if you want to run this dataset yourself. But we also have a live demo that you can use. So let's start very simple first. So I can say get. I can say paragraph. And I can say title, content, and order. And the reason that we structured it like this is because a paragraph is part of an article in Wikipedia. And a paragraph can have a title, doesn't have to. It has content. And it has an order. So it's like, for example, the fourth paragraph. Now, if I run this query, very boring query, it says as much as like, just get me the first 25 paragraphs and show me the title, content, and order. Now, if we limit this, oh, apologies. If we limit this to the first result, I can do something like this. So I can say limit one, first result. And what's going to be interesting is how the
machine learning model represents this
data object. So what I'm now doing with this additional property is that I want to see the vector for this specific
data object. So if we run it, here it returns the vector representation that is coming from the
machine learning model. So this
data object is run through the
machine learning model. This is the vector representation that it gives us, and it's representing this. Now, if we go one step back, so I just remove the vector representation. We can also make a graph relation. So we can say in article, I can say on article, an article has a title. And now if I run this, you see where it's coming from. So it's coming from a file on Wikipedia. So this is how the
data is structured, or sorry, how the
graphql api is structured. But of course, where it becomes very interesting is now if you're going to use
graphql to enable the
machine learning models to search through the dataset. So let's do something like this. Let's start from the perspective of asking a question. So we can say ask question, and I could say, how many people fit in on Airbus A380? I'm going to tell it that I wanted to find the answer in the content. So this is the content. And I'm going to limit that just to the first result, because I'm looking for a specific answer to a question. And then again, I can add the additional property to show me the answer. Yeah, I need a result. So this query says like search through the paragraphs, try to answer the question, how many people fit in an Airbus A380? Use the content property to find it limited to the first result. Show me the result of the answer. I also want to see the title of the paragraph, the content of the paragraph itself, the order in which this paragraph is showing on the page. And I want to see the actual Wikipedia page where this is coming from. So now I'm running this query. And you see how fast it was there. So it says like 656. And then you see here at launch in December 2006, 156 seat A380-200, et cetera, et cetera. So that's how you see how that works and how it gets these results. You can find anything. So anything that's in Wikipedia can search for. So I'm a big music fan. So I could say, for example, what was the name of Frank Zeppa's first band? So same type of query. Let's keep everything the same here. Let's run this query. And here you see it's the Mothers of Invention, which happens to be the correct answer. I know. We can not only do that for Q&A, but we can also do that for more generic questions. So let's now let's remove the ask and the limit and say near text. I want to search the concepts. And let's go, for example, Italian food. Now, I do not have an answer, but I can ask for certainty. Okay. Let's see what happens when we run this query. So what you see here is, well, of course, it comes from the article list of Italian dishes. That kind of makes sense. It is the first paragraph, so there's no text. And here you see all kinds of things about Italian dishes. And it's like almost 90% certain that this is the right result. But if we scroll down, you see that the number goes slightly down. It's still about Italian cuisine. But Italy cuisine. But the further we go down, you see the more we get removed from the actual topic. So here the culture of Italy, the cuisine, pasta, of course, pasta. So one of the things that we could do, for example, is that we can say, well, we want to have at least an 85% certainty of the result. So if I run this query again, and let me scroll all the way down, then let's see what's the result, so here we see Lombard cuisine from Milan, which is like 85% certain. So what you see, it's still about Italian food. But the further we go down, the further we scroll down, the further it is removed from the actual query that we have. And you can use any natural language query that you want to use. Important to mention, this demo
data set is Wikipedia, but you can use any
data that you have. Last one I want to show is something related to the graph relations. So let's just limit this to the first result. So I'm now going to get Italian food from the list of Italian dishes. Now I can even say in article, on article title, I can say has, oh no, sorry, links to article on article, and then I can say title again. So what this query does is it finds the
data object for the paragraph. So in this case, that's the first paragraph of the list of Italian dishes, which is the first graph relation. And then we're going to make another graph relation where we're going to say, okay, show me what
data objects this
data set is in turn linking to. So let's run that query and see what happens. So if you now go down, you see that the links to the article are pizza Margherita, DOC, Italian cuisine, et cetera, et cetera, et cetera, et cetera. So that's how at the root, the
graphql api of the Weaviate vector search engine works. What I would like to do is I want to introduce you to the
documentation. The easiest thing you can do is just Google for Weaviate or go to our website, semi.technology. You click developer section and you can miss it. In the installation guide, you can click, for example, customize your Weaviate setup. You can go to the customizer and install it yourself. And of course, last but not least, we have the
graphql references. So if you scroll there, you see example queries. You can try them out in real time and you also see the equivalent of these
graphql queries in different programming languages. So thank you so much for listening to my talk. I hope you like it. I hope you'll give Weaviate a try. If you come to our website, you will find our Slack, you will find the software
documentation. Weaviate itself is on GitHub. If you like what you see, then of course, and GitHub stars always appreciate it. And you can also, of course, simply Google for Weaviate and you will find other videos, blog posts, software documentations, demo
data sets, whatever you can think of. Thank you so much for listening. And I hope you'll enjoy the other talks as well. Bye. Bye.
