1 00:00:00,360 --> 00:00:05,310 ‫Now let's talk about the SNS + SQS Fan Out pattern. 2 00:00:05,310 --> 00:00:07,740 ‫The idea is that you want a message to be sent 3 00:00:07,740 --> 00:00:09,720 ‫to multiple SQS Queues. 4 00:00:09,720 --> 00:00:13,290 ‫But if you send them individually to every SQS Queue, 5 00:00:13,290 --> 00:00:14,970 ‫they can be problems associated with it. 6 00:00:14,970 --> 00:00:17,340 ‫For example, if your application crashes in between, 7 00:00:17,340 --> 00:00:19,020 ‫if their delivery failures, 8 00:00:19,020 --> 00:00:21,960 ‫or if you add more SQS Queues down the road. 9 00:00:21,960 --> 00:00:23,910 ‫Therefore we want to use the final pattern. 10 00:00:23,910 --> 00:00:27,180 ‫The idea is that you're going to push once in SNS topic 11 00:00:27,180 --> 00:00:29,700 ‫and then you're going to subscribe as many SQS Queues 12 00:00:29,700 --> 00:00:31,980 ‫as you want to the SNS topic. 13 00:00:31,980 --> 00:00:33,390 ‫These queues are subscribers 14 00:00:33,390 --> 00:00:36,930 ‫and they will all receive the messages sent into SNS. 15 00:00:36,930 --> 00:00:37,980 ‫So for example, 16 00:00:37,980 --> 00:00:39,960 ‫we have a buying service that wants to send messages 17 00:00:39,960 --> 00:00:41,790 ‫into two SQS Queues. 18 00:00:41,790 --> 00:00:43,860 ‫What it will do instead, it will send one message 19 00:00:43,860 --> 00:00:46,710 ‫into an SNS Topic, and the queues are subscribers 20 00:00:46,710 --> 00:00:49,230 ‫of that SNS Topic so that the Fraud Service 21 00:00:49,230 --> 00:00:51,660 ‫and the Shipping Service can read all messages 22 00:00:51,660 --> 00:00:53,790 ‫from their own SQS Queues. 23 00:00:53,790 --> 00:00:55,170 ‫The idea with this as well is that 24 00:00:55,170 --> 00:00:59,010 ‫it's a fully decoupled model and there is no data loss. 25 00:00:59,010 --> 00:01:01,200 ‫SQS will give you data persistence, 26 00:01:01,200 --> 00:01:03,900 ‫delayed processing, as well as retries of work. 27 00:01:03,900 --> 00:01:07,080 ‫And with this pattern, we can add more SQS Queues 28 00:01:07,080 --> 00:01:10,320 ‫as a subscriber of the SNS Topic over time. 29 00:01:10,320 --> 00:01:11,910 ‫For this, we need to make sure 30 00:01:11,910 --> 00:01:15,330 ‫that your SQS Queue access policy, as we've seen from before 31 00:01:15,330 --> 00:01:19,260 ‫allows your SNS Topic to write to your SNS-- 32 00:01:19,260 --> 00:01:20,100 ‫To your SQS Queue. 33 00:01:20,100 --> 00:01:22,620 ‫So again, yet another use case 34 00:01:22,620 --> 00:01:26,010 ‫for using queue access policies. 35 00:01:26,010 --> 00:01:27,990 ‫And we have Cross-Region Delivery. 36 00:01:27,990 --> 00:01:30,540 ‫So, it's definitely possible for an SNS Topic 37 00:01:30,540 --> 00:01:34,320 ‫in one region to send messages to SQS Queues 38 00:01:34,320 --> 00:01:37,863 ‫in other regions if the security allows it. 39 00:01:38,910 --> 00:01:39,743 ‫Next. 40 00:01:39,743 --> 00:01:41,940 ‫So, how do we use this pattern for other purposes? 41 00:01:41,940 --> 00:01:44,940 ‫For example, S3 Events into multiple queues. 42 00:01:44,940 --> 00:01:47,490 ‫So, there is a limitation of S3 Event rules. 43 00:01:47,490 --> 00:01:49,740 ‫And that's for a combination of event type, 44 00:01:49,740 --> 00:01:52,980 ‫for example, an object is being created and a prefix, 45 00:01:52,980 --> 00:01:57,570 ‫for example, images/ you can only have one S3 Event rule. 46 00:01:57,570 --> 00:02:01,710 ‫But what if you want to send the same S3 Event notification 47 00:02:01,710 --> 00:02:03,690 ‫to multiple SQS Queues? 48 00:02:03,690 --> 00:02:05,670 ‫In that case, you would use the fan-out pattern. 49 00:02:05,670 --> 00:02:08,310 ‫So for example, we have S3 Object created 50 00:02:08,310 --> 00:02:10,800 ‫as an event appearing in your S3 buckets. 51 00:02:10,800 --> 00:02:14,040 ‫And we will send this event into an SNS Topic, 52 00:02:14,040 --> 00:02:17,670 ‫and we will subscribe many SQS Queues to the SNS Topic 53 00:02:17,670 --> 00:02:18,720 ‫as a Fan-out pattern. 54 00:02:18,720 --> 00:02:22,200 ‫But we could also subscribe other type of application, 55 00:02:22,200 --> 00:02:24,270 ‫email, Lambda functions, et cetera, et cetera. 56 00:02:24,270 --> 00:02:27,150 ‫And then what we get from this, is that the message 57 00:02:27,150 --> 00:02:29,700 ‫of the events happening in Amazon S3 58 00:02:29,700 --> 00:02:32,280 ‫is going to go to many different destinations 59 00:02:32,280 --> 00:02:34,113 ‫thanks to this fan out pattern. 60 00:02:34,980 --> 00:02:37,830 ‫Another architecture is that you can send data directly 61 00:02:37,830 --> 00:02:41,790 ‫from SNS to Amazon S3 through Kinesis Data Firehose. 62 00:02:41,790 --> 00:02:45,210 ‫So, because SNS has a direct integration with KDF, 63 00:02:45,210 --> 00:02:48,450 ‫then your buying service can send data into an SNS topic. 64 00:02:48,450 --> 00:02:50,100 ‫Then Kinesis Data Firehose, 65 00:02:50,100 --> 00:02:53,100 ‫KDF will receive that information. 66 00:02:53,100 --> 00:02:54,420 ‫And then from Kinesis Data Firehose, 67 00:02:54,420 --> 00:02:57,150 ‫you can send it into your Amazon S3 bucket. 68 00:02:57,150 --> 00:02:58,560 ‫Or, for that matter, 69 00:02:58,560 --> 00:03:01,950 ‫any supported KDF specific destination 70 00:03:01,950 --> 00:03:04,020 ‫which allows you to be really extensible 71 00:03:04,020 --> 00:03:06,810 ‫in the way you want to maybe persist your messages 72 00:03:06,810 --> 00:03:08,670 ‫from your SNS Topic. 73 00:03:08,670 --> 00:03:11,640 ‫So, we can apply the fan-out pattern to FIFO Topics as well. 74 00:03:11,640 --> 00:03:16,080 ‫So, Amazon SNS has a FIFO a FIFO capability, 75 00:03:16,080 --> 00:03:17,370 ‫which is first, first out, 76 00:03:17,370 --> 00:03:19,710 ‫which gives ordering a messages in the topic. 77 00:03:19,710 --> 00:03:23,010 ‫So, the producers send the messages 1, 2, 3, 4. 78 00:03:23,010 --> 00:03:26,910 ‫And the subscribers can only be an SQS FIFO Queue for now 79 00:03:26,910 --> 00:03:30,270 ‫which is receiving messages 1, 2, 3, 4, in order. 80 00:03:30,270 --> 00:03:32,340 ‫So, the idea is that with SNS FIFO, 81 00:03:32,340 --> 00:03:34,260 ‫we get the same features as SQS FIFO, 82 00:03:34,260 --> 00:03:36,810 ‫we get Ordering by Message Group ID. 83 00:03:36,810 --> 00:03:39,840 ‫We get Deduplication using a Deduplication ID 84 00:03:39,840 --> 00:03:41,820 ‫or Content Based Deduplication, 85 00:03:41,820 --> 00:03:45,000 ‫and we can have SQS FIFO Queues as subscribers 86 00:03:45,000 --> 00:03:46,980 ‫of the FIFO SNS Topic. 87 00:03:46,980 --> 00:03:48,720 ‫In terms of throughputs, you're limited. 88 00:03:48,720 --> 00:03:51,600 ‫You get the same throughputs as the SQS FIFO Queue 89 00:03:51,600 --> 00:03:53,640 ‫because right now only SQS FIFO Queues 90 00:03:53,640 --> 00:03:55,620 ‫can read from SNS FIFO Topics. 91 00:03:55,620 --> 00:03:57,060 ‫So, why do we need this? 92 00:03:57,060 --> 00:03:59,940 ‫Well, in case you wanna do a fan out using SQS FIFO. 93 00:03:59,940 --> 00:04:02,790 ‫So, you need fan out, ordering, and deduplication. 94 00:04:02,790 --> 00:04:04,380 ‫So, the buying service will send the data 95 00:04:04,380 --> 00:04:07,380 ‫into an SNS FIFO Topic, and then it will fan out 96 00:04:07,380 --> 00:04:10,590 ‫to two SQS FIFO Queue, which can have the Fraud Service 97 00:04:10,590 --> 00:04:14,010 ‫and a Shipping Service read from FIFO queues as well. 98 00:04:14,010 --> 00:04:16,950 ‫One last feature of SNS that can be really handy 99 00:04:16,950 --> 00:04:18,720 ‫regarding the fan-out pattern, 100 00:04:18,720 --> 00:04:21,750 ‫is that you can do message filtering in SNS. 101 00:04:21,750 --> 00:04:23,640 ‫So, what is message filtering? 102 00:04:23,640 --> 00:04:26,730 ‫Well, it's a JSON policy that is used to filter messages 103 00:04:26,730 --> 00:04:29,940 ‫sent to the SNS Topics subscription. 104 00:04:29,940 --> 00:04:32,190 ‫So, if a subscription doesn't have a filter policy, 105 00:04:32,190 --> 00:04:34,980 ‫it will receive every message and that's a default behavior. 106 00:04:34,980 --> 00:04:36,510 ‫But, let's take an example of what happens 107 00:04:36,510 --> 00:04:39,390 ‫when we set up a message filtering policy. 108 00:04:39,390 --> 00:04:40,920 ‫So, we have a Buying Service 109 00:04:40,920 --> 00:04:43,710 ‫and it sends transactions into an SNS Topic. 110 00:04:43,710 --> 00:04:45,330 ‫For example, the transactions look like 111 00:04:45,330 --> 00:04:46,590 ‫there is an order number. 112 00:04:46,590 --> 00:04:48,300 ‫There's a product, for example, a pencil. 113 00:04:48,300 --> 00:04:49,650 ‫The quantity, four, 114 00:04:49,650 --> 00:04:52,350 ‫and the state, which is Placed. 115 00:04:52,350 --> 00:04:54,840 ‫Now, we want to create an SQS Queue 116 00:04:54,840 --> 00:04:56,460 ‫just for Placed orders. 117 00:04:56,460 --> 00:04:58,710 ‫Not all the orders, but just for Placed orders. 118 00:04:58,710 --> 00:05:01,050 ‫So for this, we will subscribe the SQS Queue 119 00:05:01,050 --> 00:05:05,160 ‫into the SNS Topic, and we'll apply a filter policy in JSON 120 00:05:05,160 --> 00:05:06,810 ‫and we will specify in a policy 121 00:05:06,810 --> 00:05:09,450 ‫that we want to have States equals Placed. 122 00:05:09,450 --> 00:05:11,850 ‫And so only the messages matching the policy 123 00:05:11,850 --> 00:05:13,710 ‫will go into the SQS Queue. 124 00:05:13,710 --> 00:05:16,710 ‫But then, we can have an SQS Queue for canceled orders. 125 00:05:16,710 --> 00:05:18,900 ‫And so, we can create our own filter policy 126 00:05:18,900 --> 00:05:21,270 ‫for canceled orders and have those coming 127 00:05:21,270 --> 00:05:24,000 ‫from the same SNS Topic into the SQS Queue. 128 00:05:24,000 --> 00:05:25,080 ‫And so the place orders 129 00:05:25,080 --> 00:05:26,700 ‫and the cancel orders SQS Queue 130 00:05:26,700 --> 00:05:28,620 ‫will not have the same messages. 131 00:05:28,620 --> 00:05:31,470 ‫We can also use the same filter policy, the canceled one, 132 00:05:31,470 --> 00:05:35,010 ‫to create an email subscription for canceled orders. 133 00:05:35,010 --> 00:05:37,950 ‫We could have a filter policy for declined orders 134 00:05:37,950 --> 00:05:40,710 ‫for example, and as another SQS Queue. 135 00:05:40,710 --> 00:05:44,040 ‫Or we can create an SQS Queue without a filter policy 136 00:05:44,040 --> 00:05:46,920 ‫to have all messages from that SNS Topic. 137 00:05:46,920 --> 00:05:50,670 ‫So, using all these fan-out patterns and message filtering, 138 00:05:50,670 --> 00:05:52,320 ‫FIFO Queues, and FIFO Topics, 139 00:05:52,320 --> 00:05:54,090 ‫we get lots of different possibilities 140 00:05:54,090 --> 00:05:56,970 ‫and the exam will try to test you on all of those. 141 00:05:56,970 --> 00:05:58,140 ‫So, that's it for this lecture. 142 00:05:58,140 --> 00:06:00,923 ‫I hope you liked it, and I will see you in the next lecture.