1
00:00:00,240 --> 00:00:01,230
So now let's talk about

2
00:00:01,230 --> 00:00:03,510
the Amazon OpenSearch Service.

3
00:00:03,510 --> 00:00:06,390
And Amazon OpenSearch is a successor

4
00:00:06,390 --> 00:00:07,890
to something you may have heard before

5
00:00:07,890 --> 00:00:10,170
called Amazon ElasticSearch.

6
00:00:10,170 --> 00:00:13,530
So the name change was due to some licensing issues.

7
00:00:13,530 --> 00:00:16,950
So in DynamoDB, just to do a comparison,

8
00:00:16,950 --> 00:00:19,740
you can only query the data by primary key,

9
00:00:19,740 --> 00:00:22,560
or if you have indexes on your database.

10
00:00:22,560 --> 00:00:23,790
But with OpenSearch,

11
00:00:23,790 --> 00:00:25,560
you can actually, as the name indicate,

12
00:00:25,560 --> 00:00:29,280
search any fields, even for partial matches.

13
00:00:29,280 --> 00:00:31,200
So it's very common to use OpenSearch

14
00:00:31,200 --> 00:00:34,500
to provide search to your application.

15
00:00:34,500 --> 00:00:35,850
And so you would use OpenSearch

16
00:00:35,850 --> 00:00:38,640
as a compliment to another database.

17
00:00:38,640 --> 00:00:40,920
So OpenSearch can be used for search

18
00:00:40,920 --> 00:00:44,340
but also, as the name doesn't indicate,

19
00:00:44,340 --> 00:00:47,700
you can also do analytic queries on top of OpenSearch.

20
00:00:47,700 --> 00:00:50,010
So to create OpenSearch and to use it,

21
00:00:50,010 --> 00:00:52,320
you need to create a cluster of instances.

22
00:00:52,320 --> 00:00:54,420
So it's not a serverless service.

23
00:00:54,420 --> 00:00:56,460
And OpenSearch has its own query language.

24
00:00:56,460 --> 00:00:58,470
It does not support SQL.

25
00:00:58,470 --> 00:01:01,320
So you can ingest data from different places,

26
00:01:01,320 --> 00:01:03,480
such as Kinesis Data Firehose,

27
00:01:03,480 --> 00:01:05,069
IoT, CloudWatch Logs,

28
00:01:05,069 --> 00:01:08,130
or any of your custom-built application.

29
00:01:08,130 --> 00:01:09,180
You have security

30
00:01:09,180 --> 00:01:12,150
provided through integration with Cognito, IAM.

31
00:01:12,150 --> 00:01:15,180
You get at-risk encryption and in-flight encryption.

32
00:01:15,180 --> 00:01:16,890
And as I said, you can do analytics

33
00:01:16,890 --> 00:01:18,900
on top of the OpenSearch service.

34
00:01:18,900 --> 00:01:22,260
So you can use something called OpenSearch Dashboards

35
00:01:22,260 --> 00:01:26,340
to create visualizations on top of your OpenSearch data.

36
00:01:26,340 --> 00:01:29,160
So here are some common patterns to use OpenSearch.

37
00:01:29,160 --> 00:01:31,950
So you would have DynamoDB and it will contain your data.

38
00:01:31,950 --> 00:01:34,020
This is where your users will insert,

39
00:01:34,020 --> 00:01:35,970
and delete, and update data.

40
00:01:35,970 --> 00:01:39,780
And then you send all the streams in a DynamoDB stream,

41
00:01:39,780 --> 00:01:42,090
which is then picked up by Lambda Function.

42
00:01:42,090 --> 00:01:44,490
And that Lambda Function will insert the data

43
00:01:44,490 --> 00:01:47,460
into Amazon OpenSearch in real time.

44
00:01:47,460 --> 00:01:49,140
And through this process,

45
00:01:49,140 --> 00:01:50,970
what happens that your application

46
00:01:50,970 --> 00:01:54,270
now has the ability to search for a specific item,

47
00:01:54,270 --> 00:01:58,290
for example, to do a partial search with the item name

48
00:01:58,290 --> 00:02:00,660
and then find the item ID out of it.

49
00:02:00,660 --> 00:02:02,880
And then once the item ID is obtained,

50
00:02:02,880 --> 00:02:05,400
then it will call DynamoDB

51
00:02:05,400 --> 00:02:07,230
to actually retrieve the full item

52
00:02:07,230 --> 00:02:09,030
from your DynamoDB table.

53
00:02:09,030 --> 00:02:10,410
So that's a common pattern

54
00:02:10,410 --> 00:02:13,470
in which OpenSearch provides the search capability.

55
00:02:13,470 --> 00:02:15,180
Whereas your main source of data

56
00:02:15,180 --> 00:02:17,073
still remains your DynamoDB table.

57
00:02:17,910 --> 00:02:18,930
There's also other ways.

58
00:02:18,930 --> 00:02:21,780
So you can ingest CloudWatch Logs into OpenSearch.

59
00:02:21,780 --> 00:02:22,920
So the first one is to use

60
00:02:22,920 --> 00:02:26,310
what's called a CloudWatch Log subscription filter,

61
00:02:26,310 --> 00:02:29,580
sending data in real time to Lambda Function

62
00:02:29,580 --> 00:02:31,140
that is managed by AWS.

63
00:02:31,140 --> 00:02:32,340
And then the Lambda Function

64
00:02:32,340 --> 00:02:36,360
in real time sends all the data into Amazon OpenSearch.

65
00:02:36,360 --> 00:02:38,820
Or you can also use a CloudWatch Logs

66
00:02:38,820 --> 00:02:40,440
and then Subscription Filter.

67
00:02:40,440 --> 00:02:42,720
But this time, Kinesis Data Firehose

68
00:02:42,720 --> 00:02:45,180
can read from the subscription filter.

69
00:02:45,180 --> 00:02:46,500
And then near real time,

70
00:02:46,500 --> 00:02:47,970
because this is Data Firehose,

71
00:02:47,970 --> 00:02:50,193
data will be inserted in Amazon OpenSearch.

72
00:02:51,300 --> 00:02:52,560
Other pattern around Kinesis.

73
00:02:52,560 --> 00:02:57,300
So to send Kinesis data streams into Amazon OpenSearch,

74
00:02:57,300 --> 00:02:58,890
you have two strategies.

75
00:02:58,890 --> 00:03:01,440
The first one is to use Kinesis Data Firehose.

76
00:03:01,440 --> 00:03:04,560
This is a near real time, again, type of service.

77
00:03:04,560 --> 00:03:07,020
You can optionally do some data transformation

78
00:03:07,020 --> 00:03:08,400
using a Lambda Function

79
00:03:08,400 --> 00:03:11,370
and then send data into Amazon OpenSearch.

80
00:03:11,370 --> 00:03:13,980
Or you can use Kinesis Data Streams again.

81
00:03:13,980 --> 00:03:16,380
But this time you would create a Lambda Function

82
00:03:16,380 --> 00:03:19,320
that would read the data stream in real time,

83
00:03:19,320 --> 00:03:21,120
and then you would write custom code

84
00:03:21,120 --> 00:03:22,770
to have the Lambda Function write

85
00:03:22,770 --> 00:03:25,170
to Amazon OpenSearch in real time.

86
00:03:25,170 --> 00:03:26,790
So all these patterns are valid.

87
00:03:26,790 --> 00:03:29,610
And now, you know pretty much all the possible architectures

88
00:03:29,610 --> 00:03:31,860
for using Amazon OpenSearch.

89
00:03:31,860 --> 00:03:32,970
So that's it for this lecture.

90
00:03:32,970 --> 00:03:34,020
I hope you liked it.

91
00:03:34,020 --> 00:03:35,970
And I will see you in the next lecture.