1
00:00:00,290 --> 00:00:02,450
So let's go ahead and get some practice

2
00:00:02,450 --> 00:00:03,780
for Kinesis Data Streams.

3
00:00:03,780 --> 00:00:06,710
So I'm going to open the Kinesis service,

4
00:00:06,710 --> 00:00:08,560
and create our first Kinesis Data Streams.

5
00:00:08,560 --> 00:00:10,620
As we can see, we have three options here.

6
00:00:10,620 --> 00:00:13,150
We can use Data Streams, Data Firehose, or Data Analytics,

7
00:00:13,150 --> 00:00:14,870
but we only know about Data Streams so far,

8
00:00:14,870 --> 00:00:15,900
so let's go ahead.

9
00:00:15,900 --> 00:00:17,960
We get some information around the pricing,

10
00:00:17,960 --> 00:00:22,960
which is at per shard, we pay $0.05 per hour.

11
00:00:23,640 --> 00:00:24,473
Okay.

12
00:00:24,473 --> 00:00:26,700
And then there's a cost for making PUTs or for sending data

13
00:00:26,700 --> 00:00:29,000
into Kinesis Data Streams.

14
00:00:29,000 --> 00:00:30,910
So we'll create a Data Stream,

15
00:00:30,910 --> 00:00:33,290
let's name our stream, DemoStream,

16
00:00:33,290 --> 00:00:35,960
and then we have to define the data stream capacity.

17
00:00:35,960 --> 00:00:37,210
So as you can see, we have two modes,

18
00:00:37,210 --> 00:00:40,370
we have the On-demand mode and the Provision mode.

19
00:00:40,370 --> 00:00:41,640
And in the On-demand mode,

20
00:00:41,640 --> 00:00:44,130
you don't have to think about the capacity,

21
00:00:44,130 --> 00:00:46,070
it's automatically going to scale for you,

22
00:00:46,070 --> 00:00:49,390
so there's a maximum throughput of 200 megabytes per second,

23
00:00:49,390 --> 00:00:51,160
and 200,000 records per second,

24
00:00:51,160 --> 00:00:54,980
and the maximum read capacity of 400 megabytes per second,

25
00:00:54,980 --> 00:00:58,030
per consumer, if you're using the enhanced Fan-Out option.

26
00:00:58,030 --> 00:01:00,950
So the on demand as a pay-per throughput pricing model,

27
00:01:00,950 --> 00:01:03,030
but there is no free tier. Okay.

28
00:01:03,030 --> 00:01:06,300
And the Provision mode also doesn't have a free tier.

29
00:01:06,300 --> 00:01:08,750
And in the provision mode, you need to provision shards.

30
00:01:08,750 --> 00:01:10,990
And so there's a Shard estimator tool,

31
00:01:10,990 --> 00:01:13,210
if you wanted to understand how many shards you needed,

32
00:01:13,210 --> 00:01:16,760
based on like how many records you send per second,

33
00:01:16,760 --> 00:01:18,520
and what is the size of your record,

34
00:01:18,520 --> 00:01:21,010
and how many consumers you have. Okay.

35
00:01:21,010 --> 00:01:24,220
But in a example, we're going to just set up one shard.

36
00:01:24,220 --> 00:01:27,160
One shard gives us one megabyte per second for write

37
00:01:27,160 --> 00:01:29,730
and two megabytes per second for read.

38
00:01:29,730 --> 00:01:30,990
And obviously, if you put 10 shards,

39
00:01:30,990 --> 00:01:33,640
then everything is multiplied by 10. Okay,

40
00:01:33,640 --> 00:01:36,010
And so one shard is enough for us to do the demo.

41
00:01:36,010 --> 00:01:37,730
It's also the cheapest option we can get.

42
00:01:37,730 --> 00:01:40,360
If you do not want to pay any money in this course,

43
00:01:40,360 --> 00:01:41,580
then do not do this hands on,

44
00:01:41,580 --> 00:01:43,610
because you will pay some money for your shard,

45
00:01:43,610 --> 00:01:45,070
although we will deal with it,

46
00:01:45,070 --> 00:01:47,260
and then delete it fast enough.

47
00:01:47,260 --> 00:01:50,980
So when you're ready, you just click on Create data stream,

48
00:01:50,980 --> 00:01:53,810
and you wait for the data stream to be created.

49
00:01:53,810 --> 00:01:56,230
And our stream is now successfully created.

50
00:01:56,230 --> 00:01:58,290
So in terms of applications, we can see,

51
00:01:58,290 --> 00:02:01,360
we have producers, and we are recommended three options.

52
00:02:01,360 --> 00:02:04,750
The Kinesis Agents, the SDK,

53
00:02:04,750 --> 00:02:06,280
or the Kinesis Producer Library.

54
00:02:06,280 --> 00:02:09,729
So they're all available on GitHub, for you to check it out.

55
00:02:09,729 --> 00:02:13,700
This is a way to stream data from application servers

56
00:02:13,700 --> 00:02:15,980
into Kinesis Data Streams.

57
00:02:15,980 --> 00:02:18,970
The SDK is for you to develop producers at a very low level.

58
00:02:18,970 --> 00:02:21,230
And the KPL is for you to develop producers

59
00:02:21,230 --> 00:02:23,910
at a very high level, with a better API.

60
00:02:23,910 --> 00:02:24,860
And in terms of consumers,

61
00:02:24,860 --> 00:02:28,090
we get Kinesis Data Analytics, Kinesis Data Firehose,

62
00:02:28,090 --> 00:02:31,150
or the Kinesis Client Library, or Lambda, as well,

63
00:02:31,150 --> 00:02:33,370
that is not shown as an option right here.

64
00:02:33,370 --> 00:02:35,450
Okay. We get some monitoring information

65
00:02:35,450 --> 00:02:36,760
about our Kinesis Data Streams.

66
00:02:36,760 --> 00:02:39,110
So how many records we are sending into it?

67
00:02:39,110 --> 00:02:40,380
We could look at the configuration.

68
00:02:40,380 --> 00:02:42,230
So if you wanted to scale the stream,

69
00:02:42,230 --> 00:02:44,170
we can say how many shards we want.

70
00:02:44,170 --> 00:02:46,610
We can go from one, to say, five,

71
00:02:46,610 --> 00:02:49,470
and scale our Kinesis Data Stream.

72
00:02:49,470 --> 00:02:50,670
We could add some tags,

73
00:02:50,670 --> 00:02:53,420
and then we could use enhanced Fan-Out, and configure it,

74
00:02:53,420 --> 00:02:55,610
if we wanted to have some consumer applications

75
00:02:55,610 --> 00:02:58,660
leveraging the enhanced Fan-Out capability.

76
00:02:58,660 --> 00:02:59,557
But for now, let's keep it simple.

77
00:02:59,557 --> 00:03:03,230
We just want to write and read from our stream,

78
00:03:03,230 --> 00:03:04,063
and so therefore,

79
00:03:04,063 --> 00:03:07,850
we're going to use the SDK for producing and for consuming.

80
00:03:07,850 --> 00:03:10,370
So to do so, we want to open a CLI,

81
00:03:10,370 --> 00:03:12,590
and let's use CloudShell because it's fun.

82
00:03:12,590 --> 00:03:14,350
So I'm going to click on CloudShell right here,

83
00:03:14,350 --> 00:03:16,400
which is the icon, next to the bell icon.

84
00:03:16,400 --> 00:03:18,180
And this is going to open, for me,.

85
00:03:18,180 --> 00:03:20,773
a command line interface in AWS.

86
00:03:21,870 --> 00:03:23,970
As an alternative, you could be using your own terminal

87
00:03:23,970 --> 00:03:25,360
or CLI if it's preconfigured,

88
00:03:25,360 --> 00:03:27,370
but I like to switch things up,

89
00:03:27,370 --> 00:03:28,380
and this one, I really like,

90
00:03:28,380 --> 00:03:30,640
because there's less configuration involved at least.

91
00:03:30,640 --> 00:03:31,910
Creating the environment, the first time,

92
00:03:31,910 --> 00:03:35,250
can take a bit of time, and the CloudShell is free on AWS,

93
00:03:35,250 --> 00:03:36,640
so no worries here.

94
00:03:36,640 --> 00:03:38,570
In the meantime, on Kinesis,

95
00:03:38,570 --> 00:03:41,080
open the kinesis-data-streams.sh file,

96
00:03:41,080 --> 00:03:43,330
and we're going to use that overall.

97
00:03:43,330 --> 00:03:44,720
So there are two types of commands

98
00:03:44,720 --> 00:03:48,500
to write to Kinesis Data Stream based on your CLI version

99
00:03:48,500 --> 00:03:50,170
for the AWS CLI.

100
00:03:50,170 --> 00:03:52,340
So usually, you have version two installed

101
00:03:52,340 --> 00:03:54,160
but it is possible for you to somehow,

102
00:03:54,160 --> 00:03:55,720
have version one installed.

103
00:03:55,720 --> 00:03:56,920
And so you get the version,

104
00:03:56,920 --> 00:04:00,480
you just type aws --version, and paste,

105
00:04:00,480 --> 00:04:02,070
and then, you will get some information

106
00:04:02,070 --> 00:04:04,280
around the version of AWS CLI.

107
00:04:04,280 --> 00:04:05,370
So in CloudShell,

108
00:04:05,370 --> 00:04:07,890
the CLI version that's going to be installed is version two,

109
00:04:07,890 --> 00:04:11,440
as you can see here, it was CLI version 2.1.16,

110
00:04:11,440 --> 00:04:13,570
so we're going to use the CLI commands version two.

111
00:04:13,570 --> 00:04:15,580
But if you wanted to have the version one,

112
00:04:15,580 --> 00:04:17,320
then you would use these comments.

113
00:04:17,320 --> 00:04:19,300
Okay. So now we're good to go.

114
00:04:19,300 --> 00:04:21,630
The first we want to do is send a record

115
00:04:21,630 --> 00:04:23,600
into our Kinesis Data Stream.

116
00:04:23,600 --> 00:04:27,020
And to do so, there is an API called put-record.

117
00:04:27,020 --> 00:04:29,560
And put-record, we need to specify a stream name.

118
00:04:29,560 --> 00:04:32,450
So in this example, I didn't name my stream test,

119
00:04:32,450 --> 00:04:34,020
I named it DemoStream,

120
00:04:35,330 --> 00:04:37,930
so we'll have to change this in the CLI command,

121
00:04:37,930 --> 00:04:39,030
but you get the idea.

122
00:04:39,030 --> 00:04:41,630
Then you specify a partition key for the data settings.

123
00:04:41,630 --> 00:04:44,240
So user one, and remember the data

124
00:04:44,240 --> 00:04:45,570
that shares the same partition key

125
00:04:45,570 --> 00:04:47,840
will go to the same shard, but we only have one shard,

126
00:04:47,840 --> 00:04:49,990
so that doesn't matter in this case.

127
00:04:49,990 --> 00:04:52,820
Then the data itself, so users signup.

128
00:04:52,820 --> 00:04:55,810
And finally, because we are writing some text data,

129
00:04:55,810 --> 00:04:58,570
we need to say this option cli-binary-format

130
00:04:58,570 --> 00:05:01,180
raw-in-base64-out.

131
00:05:01,180 --> 00:05:04,720
Okay. So let it paste this command, so copy and paste,

132
00:05:04,720 --> 00:05:07,480
but let me just edit the stream name,

133
00:05:07,480 --> 00:05:09,950
to make sure that it is DemoStream.

134
00:05:10,920 --> 00:05:12,900
And the cloud channel is automatically configured

135
00:05:12,900 --> 00:05:14,450
with your own IM credentials,

136
00:05:14,450 --> 00:05:17,000
so it will inherit whatever you have for IM credentials,

137
00:05:17,000 --> 00:05:19,370
and also, we'll use the region by default,

138
00:05:19,370 --> 00:05:20,240
in which it was launched.

139
00:05:20,240 --> 00:05:21,683
So us-east-1.

140
00:05:22,640 --> 00:05:23,860
I press Enter.

141
00:05:23,860 --> 00:05:25,690
And now, we get a successful message.

142
00:05:25,690 --> 00:05:29,930
So the message was sent shardId-0000000000000.

143
00:05:29,930 --> 00:05:32,490
So our first shard, and the sequence number

144
00:05:32,490 --> 00:05:34,010
of the message is here.

145
00:05:34,010 --> 00:05:38,690
If I do it again, then I'm going to get a second message

146
00:05:38,690 --> 00:05:41,040
with a successful, so we can do user signup.

147
00:05:41,040 --> 00:05:42,530
We can mix up the message,

148
00:05:42,530 --> 00:05:47,450
then user login, and then, maybe user logout.

149
00:05:47,450 --> 00:05:49,350
So we're just, we're just setting a few messages

150
00:05:49,350 --> 00:05:51,260
into our Kinesis Data Stream.

151
00:05:51,260 --> 00:05:53,040
Perfect. So I'm going to clear this,

152
00:05:53,040 --> 00:05:54,620
and if you waited a little bit,

153
00:05:54,620 --> 00:05:58,480
and went into monitoring, and look at the stream metrics,

154
00:05:58,480 --> 00:05:59,900
I will have it on one hour,

155
00:05:59,900 --> 00:06:02,430
you would see a put record right here,

156
00:06:02,430 --> 00:06:04,040
but it takes a bit of time for CloudWatch metrics

157
00:06:04,040 --> 00:06:08,010
to be updated, but you would see it here. Okay.

158
00:06:08,010 --> 00:06:10,270
So next we want to be able to consume

159
00:06:10,270 --> 00:06:11,290
from Kinesis Data Stream.

160
00:06:11,290 --> 00:06:15,400
So to do so, we're going to first describe the stream,

161
00:06:15,400 --> 00:06:18,620
to get some information around what this stream is made of,

162
00:06:18,620 --> 00:06:21,540
because we need to be able to consume from a specific shard.

163
00:06:21,540 --> 00:06:24,930
So it's DemoStream, I'll press Enter,

164
00:06:24,930 --> 00:06:27,260
and as you can see here, we have this StreamDescription.

165
00:06:27,260 --> 00:06:30,650
We have one shard called shardId-0000000000000,

166
00:06:30,650 --> 00:06:33,920
and so we need to keep this in our mind. Okay.

167
00:06:33,920 --> 00:06:36,540
To be able to read from this stream.

168
00:06:36,540 --> 00:06:38,570
And so when you use the CLI,

169
00:06:38,570 --> 00:06:40,990
the SDK at a very, very low level,

170
00:06:40,990 --> 00:06:43,630
you need to specify from which shard you are reading from.

171
00:06:43,630 --> 00:06:46,580
But if you are using Kinesis Client Library,

172
00:06:46,580 --> 00:06:49,260
all of this is handled for you by the library itself.

173
00:06:49,260 --> 00:06:50,650
But we are using the CLI,

174
00:06:50,650 --> 00:06:52,890
so we have to specify the shard ID.

175
00:06:52,890 --> 00:06:55,040
So I press Q to quit this,

176
00:06:55,040 --> 00:06:56,341
and I'm going to consume some data.

177
00:06:56,341 --> 00:06:59,060
So I'm going to run this command, right here,

178
00:06:59,060 --> 00:07:00,563
and let me clear this.

179
00:07:01,730 --> 00:07:03,170
And there's two things to note,

180
00:07:03,170 --> 00:07:05,500
so number one, I need to change the name of the stream

181
00:07:05,500 --> 00:07:06,333
I'm consuming from.

182
00:07:06,333 --> 00:07:09,310
So DemoStream is a stream,

183
00:07:09,310 --> 00:07:12,450
and then the shard-iterator-type is TRIM-HORIZON.

184
00:07:12,450 --> 00:07:14,410
And this means that you're going to read

185
00:07:15,293 --> 00:07:16,126
from the very beginning of the stream,

186
00:07:16,126 --> 00:07:17,800
so it will read all the records

187
00:07:17,800 --> 00:07:19,560
that were sent for from the beginning.

188
00:07:19,560 --> 00:07:22,210
The other option, just make sure to only receive the records

189
00:07:22,210 --> 00:07:24,260
from that very moment onwards when from a new launched

190
00:07:24,260 --> 00:07:25,740
to CLI command.

191
00:07:25,740 --> 00:07:29,130
Anyway, so I'm going to press Enter.

192
00:07:29,130 --> 00:07:31,665
And this is going to give me a ShardIterator.

193
00:07:31,665 --> 00:07:36,040
And this ShardIterator can be reused to consume records,

194
00:07:36,040 --> 00:07:37,730
so the next API comment

195
00:07:37,730 --> 00:07:42,000
is kinesis get-records --shard-iterator.

196
00:07:42,000 --> 00:07:45,490
And then, we just specify this entire string, right here.

197
00:07:45,490 --> 00:07:47,380
So this consumption mode I'm doing right now,

198
00:07:47,380 --> 00:07:49,320
by using the low level API,

199
00:07:49,320 --> 00:07:51,330
describing the stream, getting a ShardIterator,

200
00:07:51,330 --> 00:07:54,340
and getting records is using the shared consumption mode.

201
00:07:54,340 --> 00:07:55,980
This is not using enhanced Fan-Out,

202
00:07:55,980 --> 00:07:57,240
which in my opinion,

203
00:07:57,240 --> 00:08:00,610
should be using the Kinesis Client Library,

204
00:08:00,610 --> 00:08:03,520
Consumer Library for you to really leverage,

205
00:08:03,520 --> 00:08:04,750
and have a nice API to do so.

206
00:08:04,750 --> 00:08:06,340
But this is low level.

207
00:08:06,340 --> 00:08:08,700
So let's click, and let's press Enter

208
00:08:08,700 --> 00:08:09,930
on the Kinesis get-records,

209
00:08:09,930 --> 00:08:12,330
and we get a batch of records out of it.

210
00:08:12,330 --> 00:08:14,930
So we used to have record one right here,

211
00:08:14,930 --> 00:08:16,970
which is PartitionKey user1.

212
00:08:16,970 --> 00:08:20,380
And we have some data right here, but it's base64-encoded.

213
00:08:20,380 --> 00:08:23,470
We have, again, another data, base64-encoded.

214
00:08:23,470 --> 00:08:25,330
We get some time stamp information,

215
00:08:25,330 --> 00:08:26,960
another data, base64-encoded.

216
00:08:26,960 --> 00:08:28,620
And then if I press Enter,

217
00:08:28,620 --> 00:08:30,240
it's going to go a little bit down.

218
00:08:30,240 --> 00:08:32,780
Some more data that is base64-encoded.

219
00:08:32,780 --> 00:08:34,940
So to just make sure we can read the data,

220
00:08:34,940 --> 00:08:39,222
I can go to websites, base 64 decode online,

221
00:08:40,299 --> 00:08:44,120
and I'm going to paste this data right here,

222
00:08:44,120 --> 00:08:47,400
into base 64 decode, and click on DECODE,

223
00:08:47,400 --> 00:08:51,170
and this gives us user signup.

224
00:08:51,170 --> 00:08:53,923
And if I paste the second type of data, this one,

225
00:08:55,190 --> 00:09:00,190
copy and paste it in here, is going to give us user login.

226
00:09:01,070 --> 00:09:02,380
So we which is that we sent.

227
00:09:02,380 --> 00:09:03,213
So that's perfect.

228
00:09:03,213 --> 00:09:04,660
Everything is working.

229
00:09:04,660 --> 00:09:05,870
And then, as you can see,

230
00:09:05,870 --> 00:09:08,940
there's a NextShardIterator argument right here.

231
00:09:08,940 --> 00:09:10,560
So the next time we consume,

232
00:09:10,560 --> 00:09:14,080
we need to specify this NextShardIterator argument

233
00:09:14,080 --> 00:09:16,670
to consume from where we stop consuming from.

234
00:09:16,670 --> 00:09:19,230
So this is something you have to iterate from in your code.

235
00:09:19,230 --> 00:09:20,530
But at a low level,

236
00:09:20,530 --> 00:09:22,970
we've produced data to Kinesis Data Stream,

237
00:09:22,970 --> 00:09:26,400
and consume data from Kinesis Data Stream, which is awesome.

238
00:09:26,400 --> 00:09:28,500
And we've also used CloudShell in the meantime,

239
00:09:28,500 --> 00:09:30,100
which I think is very handy.

240
00:09:30,100 --> 00:09:31,150
So that's it for this demo,

241
00:09:31,150 --> 00:09:33,820
just keep this stream open as we will be using it

242
00:09:33,820 --> 00:09:36,170
for Kinesis Data Firehose in a second.

243
00:09:36,170 --> 00:09:38,270
And I will see you in the next lecture.

