1
00:00:00,510 --> 00:00:03,820
In this video, we'll talk about vector databases.

2
00:00:04,730 --> 00:00:07,160
The AI revolution is here, and it is

3
00:00:07,170 --> 00:00:08,540
about to change the world.

4
00:00:09,190 --> 00:00:11,460
AI is being used in a variety of

5
00:00:11,470 --> 00:00:16,680
industries, and has the potential to
improve our lives in many ways, but it

6
00:00:16,690 --> 00:00:18,680
also introduces new challenges.

7
00:00:20,080 --> 00:00:22,180
One of the biggest challenges is

8
00:00:22,190 --> 00:00:24,060
efficient data processing.

9
00:00:25,010 --> 00:00:29,320
AI applications such as LLMs, generative

10
00:00:29,330 --> 00:00:35,660
AI, and semantic search require large
amounts of data to train and operate.

11
00:00:36,530 --> 00:00:42,680
Efficient data processing is essential
for making AI applications successful.

12
00:00:44,010 --> 00:00:51,300
Many of the latest AI applications rely
on vector embeddings, chatbots, question

13
00:00:51,310 --> 00:00:56,660
-answering systems, and machine
translation rely on vector embeddings,

14
00:00:57,230 --> 00:01:03,220
which, as you know, means converting text
to numbers that carry within themselves

15
00:01:03,230 --> 00:01:04,700
semantic information.

16
00:01:05,430 --> 00:01:07,580
Vector embeddings are a way of

17
00:01:07,590 --> 00:01:13,260
representing text as a set of numbers in
a high -dimensional space, and the

18
00:01:13,270 --> 00:01:17,260
numbers represent the meaning of the
words in the text.

19
00:01:18,080 --> 00:01:24,500
This is critical for the AI to gain
understanding and maintain long -term memory.

20
00:01:26,180 --> 00:01:31,700
If you store text embeddings in a CSV
file or in another format that is not

21
00:01:31,710 --> 00:01:37,980
dedicated for embeddings, the size of the
file will increase dramatically, and the

22
00:01:37,990 --> 00:01:39,280
performance will drop.

23
00:01:40,250 --> 00:01:42,320
Consequently, we need a specialized

24
00:01:42,330 --> 00:01:48,860
database or data store specifically
designed to manage such large quantities

25
00:01:48,870 --> 00:01:51,820
of data in a numeric representation.

26
00:01:52,570 --> 00:01:54,420
There are many vector databases

27
00:01:54,430 --> 00:01:57,340
available, both free and commercial.

28
00:01:57,930 --> 00:02:00,820
Some of the most popular ones include

29
00:02:00,830 --> 00:02:06,600
Pinecone, Chroma, Milvus, or Quadrant.

30
00:02:09,100 --> 00:02:11,770
In this project, we'll use Pinecone.

31
00:02:12,040 --> 00:02:16,770
Pinecone is a vector database designed
for storing and querying high

32
00:02:16,780 --> 00:02:17,710
-dimensional vectors.

33
00:02:18,180 --> 00:02:21,630
It provides fast and efficient semantic

34
00:02:21,640 --> 00:02:24,190
search over these vector embeddings.

35
00:02:24,760 --> 00:02:28,750
By integrating OpenAI's LLMs with

36
00:02:28,760 --> 00:02:33,910
Pinecone, we combine deep learning
capabilities for embedding generation

37
00:02:33,920 --> 00:02:37,570
with efficient vector storage and retrieval.

38
00:02:39,370 --> 00:02:41,910
Vector databases are a new type of

39
00:02:41,920 --> 00:02:46,090
database designed to store and query
unstructured data.

40
00:02:47,500 --> 00:02:53,670
Unstructured data is data that does not
have a fixed schema, such as text,

41
00:02:54,100 --> 00:02:55,830
images, and audio.

42
00:02:56,640 --> 00:03:00,010
SQL databases, on the other hand, are

43
00:03:00,020 --> 00:03:04,370
designed to store and query structured
data, which has a fixed schema.

44
00:03:05,080 --> 00:03:11,830
In a traditional SQL database, we usually
query for rows where the values match

45
00:03:11,840 --> 00:03:13,150
specific criteria.

46
00:03:13,780 --> 00:03:16,490
Think of the SQL SELECT statement.

47
00:03:17,120 --> 00:03:22,750
In vector databases, we apply a
similarity metric to find a vector that

48
00:03:22,760 --> 00:03:25,070
is the most similar to our query.

49
00:03:26,920 --> 00:03:30,170
What is the common pipeline for vector databases?

50
00:03:31,060 --> 00:03:37,090
Vector databases use a combination of
different optimized algorithms that all

51
00:03:37,100 --> 00:03:41,290
participate in approximate nearest
neighbor search.

52
00:03:42,660 --> 00:03:44,330
Take a look at this image.

53
00:03:47,180 --> 00:03:49,970
To understand how vector databases work,

54
00:03:50,500 --> 00:03:53,590
let's break down the image into three steps.

55
00:03:54,440 --> 00:03:57,050
Embedding, Indexing, and Query.

56
00:03:58,040 --> 00:04:04,970
The first step, embedding, is to create
vector embeddings for the content we want

57
00:04:04,980 --> 00:04:05,590
to index.

58
00:04:06,760 --> 00:04:10,010
This is done by using an embedding model

59
00:04:10,020 --> 00:04:14,150
such as OpenAI's Text Embedding Ada 0 .0 .2.

60
00:04:16,740 --> 00:04:20,870
The second step, indexing, is to insert

61
00:04:20,880 --> 00:04:24,330
the vector embeddings into the vector database.

62
00:04:24,340 --> 00:04:27,470
This is done by associating each vector

63
00:04:27,480 --> 00:04:32,990
embedding with a reference to the
original content that was used to create it.

64
00:04:34,420 --> 00:04:36,770
And the third step is query.

65
00:04:37,960 --> 00:04:40,430
The third step is to query the vector

66
00:04:40,440 --> 00:04:42,430
database for similar content.

67
00:04:43,220 --> 00:04:45,470
This is done by using the same embedding

68
00:04:45,480 --> 00:04:48,170
model used to create the vector embeddings.

69
00:04:48,520 --> 00:04:51,190
The embeddings model is used to create a

70
00:04:51,200 --> 00:04:53,310
vector embedding for the query.

71
00:04:53,320 --> 00:04:56,870
And this vector embedding is then used to

72
00:04:56,880 --> 00:04:59,690
query the database for similar vector embeddings.

73
00:05:01,240 --> 00:05:03,810
The similar vector embeddings are then

74
00:05:03,820 --> 00:05:08,190
associated with the original content that
was used to create them.

75
00:05:08,800 --> 00:05:14,390
To illustrate, let's imagine a company
that wants to store and query its private documents.

76
00:05:14,920 --> 00:05:19,190
The company would first use an embedding
model to create vector embeddings for

77
00:05:19,200 --> 00:05:19,870
each document.

78
00:05:20,580 --> 00:05:23,030
These vector embeddings would then be

79
00:05:23,040 --> 00:05:26,630
inserted into a vector database such as Pinecone.

80
00:05:27,440 --> 00:05:29,610
Each vector embedding would be associated

81
00:05:29,620 --> 00:05:32,130
with a reference to the original document.

82
00:05:32,800 --> 00:05:35,050
Finally, the company can use the vector

83
00:05:35,060 --> 00:05:41,110
database to query for information that is
similar to a given query or question.

84
00:05:42,360 --> 00:05:48,010
For instance, the company could query the
documents or to a product description.

85
00:05:48,960 --> 00:05:55,110
Vector databases are much more efficient
at storing and querying unstructured data

86
00:05:55,120 --> 00:05:58,690
and are heavily used in LLMs applications.

87
00:06:00,480 --> 00:06:01,050
All right.

88
00:06:01,140 --> 00:06:01,630
We are done.

89
00:06:02,060 --> 00:06:04,810
That was a short introduction to vector databases.

90
00:06:05,780 --> 00:06:11,670
In this lecture, you learned what a
vector database is, why we need them, and

91
00:06:11,680 --> 00:06:16,090
how they leverage the power of the next
generation of AI applications.

92
00:06:16,100 --> 00:06:20,830
We will take a quick break and in the
next video, we'll move on to splitting

93
00:06:20,840 --> 00:06:23,530
and embedding text using Lanchain.

94
00:06:24,040 --> 00:06:26,810
You'll also learn about how to upload the

95
00:06:26,820 --> 00:06:29,950
vector embeddings to a vector store like Pinecone.

96
00:06:30,100 --> 00:06:33,490
This is the first step in working with

97
00:06:33,500 --> 00:06:37,730
OPLstack and developing LLMs applications.

98
00:06:38,600 --> 00:06:41,970
The OPLstack is a set of open -source

99
00:06:41,980 --> 00:06:47,030
tools that can be used to build
applications that use large -language models.

100
00:06:47,560 --> 00:06:53,930
The stack consists of three main
components, OpenAI, Pinecone, and Lanchain.

101
00:06:54,380 --> 00:06:55,290
I'll see you soon.