1
00:00:00,240 --> 00:00:01,073
(narrator) Now let's talk

2
00:00:01,073 --> 00:00:05,330
about the SNS plus SQS Fan Out pattern.

3
00:00:05,330 --> 00:00:07,730
The idea is that you want a message to be sent

4
00:00:07,730 --> 00:00:09,720
to multiple SQS queues,

5
00:00:09,720 --> 00:00:11,530
but if you send them individually

6
00:00:11,530 --> 00:00:14,960
to every SQS queue, they can be problems associated with it.

7
00:00:14,960 --> 00:00:17,350
For example, if your application crashes in between,

8
00:00:17,350 --> 00:00:19,020
if they're delivery failures,

9
00:00:19,020 --> 00:00:21,960
or if you add more SQS queues down the road.

10
00:00:21,960 --> 00:00:23,910
Therefore, we want to use the Fan Out Pattern.

11
00:00:23,910 --> 00:00:27,190
The idea is that you're going to push once in SNS topic

12
00:00:27,190 --> 00:00:28,560
and then you're going to subscribe

13
00:00:28,560 --> 00:00:31,960
as many SQS queues as you want to the SNS topic.

14
00:00:31,960 --> 00:00:33,400
These queues are subscribers

15
00:00:33,400 --> 00:00:36,920
and they will all receive the messages sent into SNS.

16
00:00:36,920 --> 00:00:38,870
So for example, we have a buying service

17
00:00:38,870 --> 00:00:39,950
that wants to send messages

18
00:00:39,950 --> 00:00:41,770
into two SQS queues.

19
00:00:41,770 --> 00:00:43,830
What it will do instead, it will send one message

20
00:00:43,830 --> 00:00:46,710
into an SNS topic, and the queues are subscribers

21
00:00:46,710 --> 00:00:49,220
of that SNS topic so that the fraud service

22
00:00:49,220 --> 00:00:51,650
and the shipping service can read all messages

23
00:00:51,650 --> 00:00:53,780
from their own SQS queues.

24
00:00:53,780 --> 00:00:54,920
The idea with this as well is

25
00:00:54,920 --> 00:00:57,100
that it's a fully decoupled model

26
00:00:57,100 --> 00:00:58,990
and there is no data loss.

27
00:00:58,990 --> 00:01:02,200
SQS will give you data persistence, delayed processing,

28
00:01:02,200 --> 00:01:03,890
as well as reties of work.

29
00:01:03,890 --> 00:01:05,800
And with this pattern, we can add more

30
00:01:05,800 --> 00:01:10,310
SQS queues as a subscriber of the SNS topic over time.

31
00:01:10,310 --> 00:01:11,890
For this, we need to make sure

32
00:01:11,890 --> 00:01:15,350
that you SQS queue access policy, as we've seen from before

33
00:01:15,350 --> 00:01:17,630
allows your SNS topic to write

34
00:01:17,630 --> 00:01:20,110
to your SNS, to your SQS queue.

35
00:01:20,110 --> 00:01:22,610
So again yet, another use case

36
00:01:22,610 --> 00:01:25,233
for using Q Access Policies.

37
00:01:26,100 --> 00:01:27,960
Next. So how do we use this pattern

38
00:01:27,960 --> 00:01:29,160
for other purposes?

39
00:01:29,160 --> 00:01:32,150
For example, S3 events into multiple queues.

40
00:01:32,150 --> 00:01:34,860
So there is a limitation of S3 event rules is

41
00:01:34,860 --> 00:01:37,680
that for a combination of event type, for example

42
00:01:37,680 --> 00:01:40,610
an object is being created and a prefix, for example

43
00:01:40,610 --> 00:01:44,780
images/ you can only have one S3 events rule.

44
00:01:44,780 --> 00:01:48,910
But what if you want to send the same S3 event notification

45
00:01:48,910 --> 00:01:51,430
to multiple SQS queues. In that case,

46
00:01:51,430 --> 00:01:52,860
you would use the Fan Out Pattern.

47
00:01:52,860 --> 00:01:55,530
So for example, we have S3 object created

48
00:01:55,530 --> 00:01:58,010
as an event appearing in S3 buckets

49
00:01:58,010 --> 00:02:01,240
and we will send this event into an SNS topic

50
00:02:01,240 --> 00:02:03,880
and we will subscribe many SQS queues

51
00:02:03,880 --> 00:02:05,920
to the SNS topic as a fan out pattern,

52
00:02:05,920 --> 00:02:09,410
but we could also subscribe other type of application,

53
00:02:09,410 --> 00:02:11,770
email, Lambda functions, et cetera, et cetera.

54
00:02:11,770 --> 00:02:14,360
And then what we get from this is that the message

55
00:02:14,360 --> 00:02:15,920
of the events happening

56
00:02:15,920 --> 00:02:17,550
in Amazon S3 is going to go

57
00:02:17,550 --> 00:02:19,490
to many different destinations

58
00:02:19,490 --> 00:02:21,313
thanks to this fan out pattern.

59
00:02:22,170 --> 00:02:23,370
Another architecture is

60
00:02:23,370 --> 00:02:25,110
that you can send data directly

61
00:02:25,110 --> 00:02:28,990
from SNS to Amazon S3 Through Kinesis Data Firehose.

62
00:02:28,990 --> 00:02:32,410
So because SNS has a direct integration with KDF

63
00:02:32,410 --> 00:02:34,030
then your buying service can send data

64
00:02:34,030 --> 00:02:35,650
into an SNS topic,

65
00:02:35,650 --> 00:02:38,010
then Kinesis Data Firehose, KDF,

66
00:02:38,010 --> 00:02:40,310
will receive that information.

67
00:02:40,310 --> 00:02:41,640
And then from Kinesis Data Firehose,

68
00:02:41,640 --> 00:02:44,017
you can send it into your Amazon S3 bucket.

69
00:02:44,017 --> 00:02:45,760
Or for of that matter,

70
00:02:45,760 --> 00:02:49,150
any supported KDF Specific destination

71
00:02:49,150 --> 00:02:51,210
which allows you to be really extensible

72
00:02:51,210 --> 00:02:54,020
in the way you want to maybe persist your messages

73
00:02:54,020 --> 00:02:55,870
from your SNS topic.

74
00:02:55,870 --> 00:02:57,600
So we can apply the final pattern

75
00:02:57,600 --> 00:02:58,840
to FIFO topics as well.

76
00:02:58,840 --> 00:03:03,290
So Amazon SNS has a FIFO or FIFO capability

77
00:03:03,290 --> 00:03:04,570
which is first in, first out

78
00:03:04,570 --> 00:03:06,930
which gives ordering messages in the topic.

79
00:03:06,930 --> 00:03:10,200
So the producers send the messages 1, 2, 3, 4

80
00:03:10,200 --> 00:03:14,140
and the subscribers can only be an SQS FIFO queue for now

81
00:03:14,140 --> 00:03:17,470
which is receiving messages 1, 2, 3, 4 in order.

82
00:03:17,470 --> 00:03:19,540
So the idea is that with SNS FIFO

83
00:03:19,540 --> 00:03:21,460
we get the same features as SQS FIFO.

84
00:03:21,460 --> 00:03:24,030
We get Ordering Message Group ID.

85
00:03:24,030 --> 00:03:27,050
We get Deduplication using a Deduplication ID

86
00:03:27,050 --> 00:03:29,030
or Content Based Deduplication,

87
00:03:29,030 --> 00:03:31,200
and we can have SQS FIFO queues

88
00:03:31,200 --> 00:03:34,180
as subscribers of the FIFO SNS topic.

89
00:03:34,180 --> 00:03:35,930
In terms of throughputs, you're limited,

90
00:03:35,930 --> 00:03:38,800
you get the same throughputs as the SQS FIFO queue

91
00:03:38,800 --> 00:03:41,180
because right now only SQS FIFO queues can read

92
00:03:41,180 --> 00:03:42,830
from SNS FIFO topics.

93
00:03:42,830 --> 00:03:44,260
So why do we need this?

94
00:03:44,260 --> 00:03:47,160
Well, in case you wanna do a fan out using SQS FIFO.

95
00:03:47,160 --> 00:03:49,990
So you need fan out, ordering, and deduplication.

96
00:03:49,990 --> 00:03:51,570
So the bank service will send the data

97
00:03:51,570 --> 00:03:54,580
into an SNS FIFO topic, and then it will fan out

98
00:03:54,580 --> 00:03:57,840
to two SQS FIFO queue which can have the fraud service

99
00:03:57,840 --> 00:04:01,210
and the shipping service read from FIFO queues as well.

100
00:04:01,210 --> 00:04:02,860
One last feature of SNS

101
00:04:02,860 --> 00:04:06,070
that can be really handy regarding the fan out pattern is

102
00:04:06,070 --> 00:04:08,950
that you can do message filtering in SNS.

103
00:04:08,950 --> 00:04:10,840
So what is message filtering?

104
00:04:10,840 --> 00:04:12,920
Well it's JSON policy that is used

105
00:04:12,920 --> 00:04:14,600
to filter messages sent

106
00:04:14,600 --> 00:04:17,140
to the SNS topics as subscription.

107
00:04:17,140 --> 00:04:19,399
So if a subscription doesn't have a filter policy

108
00:04:19,399 --> 00:04:20,709
it will receive a every message

109
00:04:20,709 --> 00:04:22,170
and that's the default behavior,

110
00:04:22,170 --> 00:04:23,720
but let's take an example of what happens

111
00:04:23,720 --> 00:04:26,610
when we set up a message filtering policy.

112
00:04:26,610 --> 00:04:28,130
So we have a buying service,

113
00:04:28,130 --> 00:04:30,910
and it sends transactions into an SNS topic.

114
00:04:30,910 --> 00:04:32,330
For example, the transaction look

115
00:04:32,330 --> 00:04:33,790
like do there's an order number.

116
00:04:33,790 --> 00:04:35,510
There's a product, for example, the pencil

117
00:04:35,510 --> 00:04:39,550
the quantity for, and the states, which is placed.

118
00:04:39,550 --> 00:04:43,640
Now, we want to rate an SQS queue just for placed orders

119
00:04:43,640 --> 00:04:45,910
not all the orders, but just for place orders.

120
00:04:45,910 --> 00:04:48,250
So for this, we will subscribe the SQS queue

121
00:04:48,250 --> 00:04:52,370
into the SNS topic and we'll apply a filter policy in JSON.

122
00:04:52,370 --> 00:04:54,010
And we will specify in a policy

123
00:04:54,010 --> 00:04:56,680
that we want to have state equals placed.

124
00:04:56,680 --> 00:04:59,480
And so only the message is matching the policy will go

125
00:04:59,480 --> 00:05:00,930
into the SQS queue.

126
00:05:00,930 --> 00:05:03,880
But then we can have an SQS queue for canceled orders.

127
00:05:03,880 --> 00:05:06,100
And so we can create our own filter policy

128
00:05:06,100 --> 00:05:08,480
for canceled orders and have those coming

129
00:05:08,480 --> 00:05:11,210
from the same SNS topic into the SQS queue.

130
00:05:11,210 --> 00:05:12,950
And so the placed orders and the cancel orders

131
00:05:12,950 --> 00:05:15,830
in SQS queue will not have the same messages.

132
00:05:15,830 --> 00:05:17,620
We can also use the same filter policy

133
00:05:17,620 --> 00:05:20,500
the canceled one to create an email subscription

134
00:05:20,500 --> 00:05:22,230
for canceled orders.

135
00:05:22,230 --> 00:05:25,160
We could have a filter policy for declined orders

136
00:05:25,160 --> 00:05:27,920
for example, and as another SQS queue

137
00:05:27,920 --> 00:05:29,480
or we can create an SQS queue

138
00:05:29,480 --> 00:05:32,520
without a filter policy to have all messages

139
00:05:32,520 --> 00:05:34,120
from that SNS topic.

140
00:05:34,120 --> 00:05:36,660
So using all these fan out patterns

141
00:05:36,660 --> 00:05:39,550
and message filtering FIFO queues, and FIFO topics,

142
00:05:39,550 --> 00:05:41,290
we get lots of different possibilities

143
00:05:41,290 --> 00:05:44,180
and the exam will try to test you on all of those.

144
00:05:44,180 --> 00:05:45,350
So that's it for this lecture.

145
00:05:45,350 --> 00:05:46,183
I hope you liked it.

146
00:05:46,183 --> 00:05:47,960
And I will see you in the next lecture.

