1
00:00:00,660 --> 00:00:02,740
So let's about SQS,

2
00:00:02,740 --> 00:00:05,020
and at the core of SQS is a queue

3
00:00:05,020 --> 00:00:07,330
because SQS is a simple queuing service.

4
00:00:07,330 --> 00:00:08,970
So we have an SQS queue,

5
00:00:08,970 --> 00:00:11,150
and it is going to contain messages.

6
00:00:11,150 --> 00:00:12,610
And to contain messages,

7
00:00:12,610 --> 00:00:16,070
well, something needs to send messages into our SQS queue

8
00:00:16,070 --> 00:00:19,140
and whatever sends a message into our SQS queue

9
00:00:19,140 --> 00:00:21,110
is called a producer.

10
00:00:21,110 --> 00:00:23,490
So it's possible for us to have one producer,

11
00:00:23,490 --> 00:00:24,580
but also have more.

12
00:00:24,580 --> 00:00:26,380
You can have multiple producers

13
00:00:26,380 --> 00:00:29,370
sending many messages into an SQS queue.

14
00:00:29,370 --> 00:00:31,120
And the message could be whatever he wants.

15
00:00:31,120 --> 00:00:33,670
For example, it could be process this order,

16
00:00:33,670 --> 00:00:35,180
or process this video.

17
00:00:35,180 --> 00:00:38,350
Whatever message you create goes into the queue.

18
00:00:38,350 --> 00:00:41,680
Then something needs to process the messages from the queue

19
00:00:41,680 --> 00:00:44,610
and receive them, and it's called consumers.

20
00:00:44,610 --> 00:00:47,420
So consumers will poll the messages from the queue,

21
00:00:47,420 --> 00:00:49,620
so that means that they will ask the queue,

22
00:00:49,620 --> 00:00:50,970
do you have any message for me?

23
00:00:50,970 --> 00:00:52,880
And the queue says, yes, here it is.

24
00:00:52,880 --> 00:00:54,600
And the consumer will poll these messages

25
00:00:54,600 --> 00:00:56,210
and get some information.

26
00:00:56,210 --> 00:00:57,540
And then with that message,

27
00:00:57,540 --> 00:01:00,640
it will process it and then delete it back from the queue.

28
00:01:00,640 --> 00:01:02,240
And you may have multiple consumers

29
00:01:02,240 --> 00:01:04,670
consuming messages from an SQS queue.

30
00:01:04,670 --> 00:01:07,770
So a queuing service is here to be a buffer

31
00:01:07,770 --> 00:01:11,980
to decouple between your producers and your consumers.

32
00:01:11,980 --> 00:01:14,190
Now, SQS is a complicated service

33
00:01:14,190 --> 00:01:15,320
and we'll see that in-depth,

34
00:01:15,320 --> 00:01:18,670
but the very first offering is called Amazon SQS

35
00:01:18,670 --> 00:01:20,430
for standard queues.

36
00:01:20,430 --> 00:01:24,590
And SQS, historically on AWS, it is the oldest offering.

37
00:01:24,590 --> 00:01:27,470
It was one of the first services on AWS.

38
00:01:27,470 --> 00:01:29,540
It's over 10 years old so it's quite,

39
00:01:29,540 --> 00:01:31,670
quite set in stone about how it works.

40
00:01:31,670 --> 00:01:33,020
It's a fully managed service,

41
00:01:33,020 --> 00:01:35,510
and it will be used to decouple applications.

42
00:01:35,510 --> 00:01:39,310
So anytime you see application decoupling in your exam,

43
00:01:39,310 --> 00:01:41,850
think about Amazon SQS.

44
00:01:41,850 --> 00:01:45,250
Now, why is SQS so special?

45
00:01:45,250 --> 00:01:47,380
Well, we get an unlimited throughputs.

46
00:01:47,380 --> 00:01:48,213
That means you can send

47
00:01:48,213 --> 00:01:50,620
as many messages per second as you want,

48
00:01:50,620 --> 00:01:53,670
and the queue can have as many messages as you want as well.

49
00:01:53,670 --> 00:01:55,390
So there's no limit on throughputs

50
00:01:55,390 --> 00:01:58,710
and no limits in the number of messages in the queue.

51
00:01:58,710 --> 00:02:00,920
Now, each message is short-lived.

52
00:02:00,920 --> 00:02:01,770
What does that mean?

53
00:02:01,770 --> 00:02:03,310
That means that by default,

54
00:02:03,310 --> 00:02:06,860
the message will stay in the queue for four days

55
00:02:06,860 --> 00:02:08,530
and the maximum amount of time

56
00:02:08,530 --> 00:02:11,860
that a message can be in a queue is 14 days.

57
00:02:11,860 --> 00:02:14,380
That means that as soon as you send a message to a queue,

58
00:02:14,380 --> 00:02:18,250
it has to be read by consumer and deleted from the queue

59
00:02:18,250 --> 00:02:21,410
after being processed within that retention period,

60
00:02:21,410 --> 00:02:23,340
otherwise, it will be lost.

61
00:02:23,340 --> 00:02:24,990
Then we have low-latency.

62
00:02:24,990 --> 00:02:27,560
So SQS means that, for them,

63
00:02:27,560 --> 00:02:28,930
is that whenever you send a message

64
00:02:28,930 --> 00:02:30,330
or read a message from SQS,

65
00:02:30,330 --> 00:02:32,010
you will get a response very quickly,

66
00:02:32,010 --> 00:02:34,900
less than 10 milliseconds on publish and receive.

67
00:02:34,900 --> 00:02:37,330
And the messages in SQS have to be small.

68
00:02:37,330 --> 00:02:41,880
They have to be less than 256 kilobytes per messages sent.

69
00:02:41,880 --> 00:02:43,030
Now SQS is a queuing service,

70
00:02:43,030 --> 00:02:45,820
so you can see high throughput, high volume, and so on,

71
00:02:45,820 --> 00:02:49,450
and so, it is possible to have duplicates messages.

72
00:02:49,450 --> 00:02:50,760
That means that, for example,

73
00:02:50,760 --> 00:02:53,250
sometimes a message will be delivered twice,

74
00:02:53,250 --> 00:02:56,070
and so this is why it's called at least once delivery.

75
00:02:56,070 --> 00:02:58,860
And if you go ahead with writing an application,

76
00:02:58,860 --> 00:03:00,870
you need to take that into account.

77
00:03:00,870 --> 00:03:03,690
It can also have out of the order messages,

78
00:03:03,690 --> 00:03:06,450
which means it's best effort ordering,

79
00:03:06,450 --> 00:03:09,640
and we'll see there is another type of offering from SQS

80
00:03:09,640 --> 00:03:11,450
that can deal with that limitation,

81
00:03:11,450 --> 00:03:13,740
but we'll see this later on in this section.

82
00:03:13,740 --> 00:03:16,810
So, let's go back to our message producers.

83
00:03:16,810 --> 00:03:20,410
So, messages that are up to 256 kilobytes

84
00:03:20,410 --> 00:03:23,900
are sent into SQS by producers.

85
00:03:23,900 --> 00:03:25,950
And how does it happen?

86
00:03:25,950 --> 00:03:28,290
Well, the producers will send the messages to SQS

87
00:03:28,290 --> 00:03:31,310
using an SDK, software development kits.

88
00:03:31,310 --> 00:03:35,710
And the API to send a message to SQS is called SendMessage.

89
00:03:35,710 --> 00:03:36,840
Very simple.

90
00:03:36,840 --> 00:03:38,790
Now the message, it will be written,

91
00:03:38,790 --> 00:03:41,270
it will be persisted into the SQS queue

92
00:03:41,270 --> 00:03:44,920
until a consumer reads that message and deletes it,

93
00:03:44,920 --> 00:03:47,940
which signifies that the message has been processed.

94
00:03:47,940 --> 00:03:49,110
We know the retention.

95
00:03:49,110 --> 00:03:51,220
And so, what's a use case for producing messages?

96
00:03:51,220 --> 00:03:53,760
For example, you want to process an order,

97
00:03:53,760 --> 00:03:56,920
for example, packets, and then ship it to the center.

98
00:03:56,920 --> 00:03:58,780
And so you wanna do this in your own time

99
00:03:58,780 --> 00:04:01,530
so you will send a message into the SQS queue

100
00:04:01,530 --> 00:04:02,890
with maybe some information,

101
00:04:02,890 --> 00:04:05,870
such as the order ID, the customer ID,

102
00:04:05,870 --> 00:04:07,180
and any attributes you may want.

103
00:04:07,180 --> 00:04:08,840
For example, the address, and so on.

104
00:04:08,840 --> 00:04:11,230
And then your consumer, that is in application rights,

105
00:04:11,230 --> 00:04:14,260
will have to deal with that message itself.

106
00:04:14,260 --> 00:04:15,490
So, again, to confirm,

107
00:04:15,490 --> 00:04:19,329
SQS standard has unlimited throughputs.

108
00:04:19,329 --> 00:04:21,610
So we've seen about producers. Very simple.

109
00:04:21,610 --> 00:04:23,750
Now let's look about consumers.

110
00:04:23,750 --> 00:04:25,663
So consumers, they are applications

111
00:04:25,663 --> 00:04:28,150
that we have to write with some code

112
00:04:28,150 --> 00:04:31,910
and these applications can be running on EC2 instances,

113
00:04:31,910 --> 00:04:34,550
so your virtual servers on AWS,

114
00:04:34,550 --> 00:04:37,930
but also they can be running on your own on-premises servers

115
00:04:37,930 --> 00:04:39,350
if you wanted to,

116
00:04:39,350 --> 00:04:41,000
or, we haven't seen it yet,

117
00:04:41,000 --> 00:04:42,930
but they can also be running

118
00:04:42,930 --> 00:04:45,620
on Lambda functions on AWS Lambda.

119
00:04:45,620 --> 00:04:46,453
We'll see in this course,

120
00:04:46,453 --> 00:04:49,220
it is a serverless compute type of service.

121
00:04:49,220 --> 00:04:51,550
What this means is that you can just read messages

122
00:04:51,550 --> 00:04:52,660
directly from them as well.

123
00:04:52,660 --> 00:04:55,040
We'll see this later on this course, do not worry.

124
00:04:55,040 --> 00:04:59,700
So, going back to our simple use case about EC2 instances,

125
00:04:59,700 --> 00:05:02,200
our queue has a consumer

126
00:05:02,200 --> 00:05:05,767
and the consumer polls for SQS for messages.

127
00:05:05,767 --> 00:05:08,280
And that means that the consumer will ask the queue,

128
00:05:08,280 --> 00:05:10,360
do you have messages for me?

129
00:05:10,360 --> 00:05:13,630
And the consumer may receive up to 10 messages at a time.

130
00:05:13,630 --> 00:05:16,420
So if there are messages in the SQS queue,

131
00:05:16,420 --> 00:05:17,940
it will receive a valid response saying

132
00:05:17,940 --> 00:05:21,140
here are the messages that are waiting for you.

133
00:05:21,140 --> 00:05:23,140
Then the consumer, it's your code,

134
00:05:23,140 --> 00:05:25,620
has a responsibility to process these messages.

135
00:05:25,620 --> 00:05:29,110
For example, insert some orders into an RDS database.

136
00:05:29,110 --> 00:05:31,670
So you will go ahead and for each order,

137
00:05:31,670 --> 00:05:33,910
you will insert it into your RDS database,

138
00:05:33,910 --> 00:05:36,350
something that you have to write, obviously with your code,

139
00:05:36,350 --> 00:05:39,210
and then because these messages have been processed

140
00:05:39,210 --> 00:05:40,500
because they have been received

141
00:05:40,500 --> 00:05:43,320
and inserted into an Amazon RDS database,

142
00:05:43,320 --> 00:05:44,660
your consumer will go ahead

143
00:05:44,660 --> 00:05:47,030
and delete these messages from the queue

144
00:05:47,030 --> 00:05:49,330
using the DeleteMessage API.

145
00:05:49,330 --> 00:05:51,990
And this will guarantee that no other consumer

146
00:05:51,990 --> 00:05:53,590
will be able to see these messages

147
00:05:53,590 --> 00:05:56,973
and therefore, the message processing is complete.

148
00:05:57,870 --> 00:05:59,830
So now we can scale this up.

149
00:05:59,830 --> 00:06:02,000
We can have multiple consumers at a time.

150
00:06:02,000 --> 00:06:04,293
So our SQS queue can have multiple consumers

151
00:06:04,293 --> 00:06:08,270
that will receive and process these messages in parallel.

152
00:06:08,270 --> 00:06:10,660
So here we have three on EC2 instances,

153
00:06:10,660 --> 00:06:12,920
and so each consumer will receive

154
00:06:12,920 --> 00:06:16,560
a different set of messages by calling the poll function.

155
00:06:16,560 --> 00:06:18,560
And so, if somehow a message

156
00:06:18,560 --> 00:06:20,260
is not processed fast enough by a consumer,

157
00:06:20,260 --> 00:06:22,660
it will be received by other consumers,

158
00:06:22,660 --> 00:06:26,080
and so this is why we have at least once delivery.

159
00:06:26,080 --> 00:06:29,520
And this is also why we have best-effort message ordering.

160
00:06:29,520 --> 00:06:31,030
Now, as I said, when the consumers

161
00:06:31,030 --> 00:06:33,040
are done with the messages, they will have to delete them,

162
00:06:33,040 --> 00:06:36,090
otherwise, other consumers will see these messages.

163
00:06:36,090 --> 00:06:38,300
And so what this means is that with SQS queues,

164
00:06:38,300 --> 00:06:41,480
if we need to increase the throughputs

165
00:06:41,480 --> 00:06:43,000
because we have more messages,

166
00:06:43,000 --> 00:06:46,500
then we can add consumers and do horizontal scaling

167
00:06:46,500 --> 00:06:49,410
to improve the throughput of processing.

168
00:06:49,410 --> 00:06:51,820
And so, if you remember what we said,

169
00:06:51,820 --> 00:06:54,530
this is a perfect use case for using SQS

170
00:06:54,530 --> 00:06:56,860
with your Auto Scaling groups, or ASG.

171
00:06:56,860 --> 00:06:58,170
So, what does that mean?

172
00:06:58,170 --> 00:07:00,310
Well, that means that your consumers

173
00:07:00,310 --> 00:07:02,190
will be running on EC2 instances

174
00:07:02,190 --> 00:07:04,170
inside of an Auto Scaling group

175
00:07:04,170 --> 00:07:07,530
and they will be polling for messages from the SQS queue.

176
00:07:07,530 --> 00:07:10,110
But now your Auto Scaling group has to be scaling

177
00:07:10,110 --> 00:07:11,850
on some kind of metric,

178
00:07:11,850 --> 00:07:15,240
and a metric that is available to us is the Queue Length.

179
00:07:15,240 --> 00:07:17,850
It's called ApproximateNumberOfMessages.

180
00:07:17,850 --> 00:07:21,650
It is a CloudWatch Metric that's available in any SQS queue.

181
00:07:21,650 --> 00:07:23,190
And we could set up an alarm,

182
00:07:23,190 --> 00:07:28,110
such as whenever the queue length go over a certain level,

183
00:07:28,110 --> 00:07:30,310
then please set up a CloudWatch Alarm,

184
00:07:30,310 --> 00:07:33,670
and this alarm should increase the capacity

185
00:07:33,670 --> 00:07:36,750
of my Auto Scaling group by X amount.

186
00:07:36,750 --> 00:07:38,330
And this will guarantee that

187
00:07:38,330 --> 00:07:40,220
the more messages you have in your SQS queue,

188
00:07:40,220 --> 00:07:43,260
maybe because there's a surge of orders on your websites,

189
00:07:43,260 --> 00:07:45,670
the more EC2 instances will be provided

190
00:07:45,670 --> 00:07:47,140
by your Auto Scaling group,

191
00:07:47,140 --> 00:07:49,520
and you will accordingly process these messages

192
00:07:49,520 --> 00:07:51,160
at a higher throughputs.

193
00:07:51,160 --> 00:07:52,880
So this is a very common integration

194
00:07:52,880 --> 00:07:55,190
that you will see in the exam.

195
00:07:55,190 --> 00:07:57,240
Now SQS, again, the use case

196
00:07:57,240 --> 00:08:02,040
is to decouple between applications, so application tiers.

197
00:08:02,040 --> 00:08:04,660
So, for example, let's take an example of an application

198
00:08:04,660 --> 00:08:06,520
that processes videos.

199
00:08:06,520 --> 00:08:09,560
We could have just one big application

200
00:08:09,560 --> 00:08:12,220
that's called a front-end that will take the request

201
00:08:12,220 --> 00:08:14,370
and whenever a video needs to be processed,

202
00:08:14,370 --> 00:08:15,370
it will do the processing

203
00:08:15,370 --> 00:08:17,850
and then insert that into an S3 bucket.

204
00:08:17,850 --> 00:08:19,150
But the problem is that processing

205
00:08:19,150 --> 00:08:21,210
may be very, very long to do

206
00:08:21,210 --> 00:08:23,680
and it may just slow down your websites

207
00:08:23,680 --> 00:08:26,330
if you do this in the front-end here.

208
00:08:26,330 --> 00:08:29,210
So instead, you can decouple your application here and say,

209
00:08:29,210 --> 00:08:32,309
wait a minute, the request of processing a file

210
00:08:32,309 --> 00:08:34,120
and the actual processing of a file

211
00:08:34,120 --> 00:08:36,230
can happen in two different applications.

212
00:08:36,230 --> 00:08:38,200
And therefore, whenever you take a request

213
00:08:38,200 --> 00:08:39,419
to process a file,

214
00:08:39,419 --> 00:08:42,659
you will send a message into an SQS queue.

215
00:08:42,659 --> 00:08:45,210
Now, when you do the request to process,

216
00:08:45,210 --> 00:08:46,950
that file will be in the SQS queue

217
00:08:46,950 --> 00:08:50,100
and you can create a second processing tier

218
00:08:50,100 --> 00:08:52,160
called the back-end processing application

219
00:08:52,160 --> 00:08:54,260
that will be in its own Auto-Scaling group

220
00:08:54,260 --> 00:08:57,180
to receive these messages, process these videos,

221
00:08:57,180 --> 00:08:59,770
and insert them into an S3 bucket.

222
00:08:59,770 --> 00:09:01,990
So as we can see here with this architecture,

223
00:09:01,990 --> 00:09:04,590
we can scale the front-end accordingly,

224
00:09:04,590 --> 00:09:07,010
and we can scale the back-end accordingly as well,

225
00:09:07,010 --> 00:09:08,410
but independently.

226
00:09:08,410 --> 00:09:11,430
And because the SQS queue has unlimited throughputs

227
00:09:11,430 --> 00:09:13,490
and it has unlimited number of messages

228
00:09:13,490 --> 00:09:16,370
in terms of the queue, then you are really safe,

229
00:09:16,370 --> 00:09:20,310
and this is a robust and scalable type of architecture.

230
00:09:20,310 --> 00:09:22,030
And also, for your front-end,

231
00:09:22,030 --> 00:09:25,030
you can use the optimal type of EC2 instances

232
00:09:25,030 --> 00:09:26,520
or architecture for your front-end.

233
00:09:26,520 --> 00:09:27,530
And for the back-end,

234
00:09:27,530 --> 00:09:29,120
maybe if you're doing some video processing,

235
00:09:29,120 --> 00:09:31,150
you can use some EC2 instances

236
00:09:31,150 --> 00:09:34,350
that have a GPU, a graphical processing unit,

237
00:09:34,350 --> 00:09:36,170
because you know that these type of instances

238
00:09:36,170 --> 00:09:38,490
will be optimal for doing this kind of workload.

239
00:09:38,490 --> 00:09:40,680
So this is the kind of architecture

240
00:09:40,680 --> 00:09:41,830
that will come up in the exam

241
00:09:41,830 --> 00:09:42,920
and that you're expected to know,

242
00:09:42,920 --> 00:09:45,170
and this is an amazing and tremendous use case

243
00:09:45,170 --> 00:09:47,110
for SQS queues.

244
00:09:47,110 --> 00:09:48,980
Finally, SQS security.

245
00:09:48,980 --> 00:09:50,910
So we have encryption in-flight

246
00:09:50,910 --> 00:09:54,320
by sending and producing messages using the HTTPS API,

247
00:09:54,320 --> 00:09:56,920
we get at-rest encryption using KMS keys,

248
00:09:56,920 --> 00:09:59,440
and if we wanted to, we can do client-side encryption,

249
00:09:59,440 --> 00:10:01,160
but that means that the client has to perform

250
00:10:01,160 --> 00:10:02,950
the encryption and the decryption itself.

251
00:10:02,950 --> 00:10:06,270
It's not something that's supported by SQS out of the box.

252
00:10:06,270 --> 00:10:09,420
For access controls, IAM policies are going to be able to

253
00:10:09,420 --> 00:10:11,860
regulate access to the SQS API,

254
00:10:11,860 --> 00:10:14,800
but we also have SQS access policies,

255
00:10:14,800 --> 00:10:17,160
which are similar to S3 bucket policies,

256
00:10:17,160 --> 00:10:19,150
and they're very helpful when you want to do

257
00:10:19,150 --> 00:10:21,290
cross-account access to SQS queue,

258
00:10:21,290 --> 00:10:23,150
or when you want you to allow other services,

259
00:10:23,150 --> 00:10:26,240
such as SNS that we'll see very soon, or Amazon S3,

260
00:10:26,240 --> 00:10:27,870
to write to an SQS queue,

261
00:10:27,870 --> 00:10:29,980
for example, with S3 events.

262
00:10:29,980 --> 00:10:31,840
So that's it for SQS for an overview.

263
00:10:31,840 --> 00:10:32,673
I hope you liked it,

264
00:10:32,673 --> 00:10:35,480
and I will see you in the next lecture for some practice.

