‫Now let's talk about the SNS + SQS Fan Out pattern.
‫The idea is that you want a message to be sent
‫to multiple SQS Queues.
‫But if you send them individually to every SQS Queue,
‫they can be problems associated with it.
‫For example, if your application crashes in between,
‫if their delivery failures,
‫or if you add more SQS Queues down the road.
‫Therefore we want to use the final pattern.
‫The idea is that you're going to push once in SNS topic
‫and then you're going to subscribe as many SQS Queues
‫as you want to the SNS topic.
‫These queues are subscribers
‫and they will all receive the messages sent into SNS.
‫So for example,
‫we have a buying service that wants to send messages
‫into two SQS Queues.
‫What it will do instead, it will send one message
‫into an SNS Topic, and the queues are subscribers
‫of that SNS Topic so that the Fraud Service
‫and the Shipping Service can read all messages
‫from their own SQS Queues.
‫The idea with this as well is that
‫it's a fully decoupled model and there is no data loss.
‫SQS will give you data persistence,
‫delayed processing, as well as retries of work.
‫And with this pattern, we can add more SQS Queues
‫as a subscriber of the SNS Topic over time.
‫For this, we need to make sure
‫that your SQS Queue access policy, as we've seen from before
‫allows your SNS Topic to write to your SNS--
‫To your SQS Queue.
‫So again, yet another use case
‫for using queue access policies.
‫And we have Cross-Region Delivery.
‫So, it's definitely possible for an SNS Topic
‫in one region to send messages to SQS Queues
‫in other regions if the security allows it.
‫Next.
‫So, how do we use this pattern for other purposes?
‫For example, S3 Events into multiple queues.
‫So, there is a limitation of S3 Event rules.
‫And that's for a combination of event type,
‫for example, an object is being created and a prefix,
‫for example, images/ you can only have one S3 Event rule.
‫But what if you want to send the same S3 Event notification
‫to multiple SQS Queues?
‫In that case, you would use the fan-out pattern.
‫So for example, we have S3 Object created
‫as an event appearing in your S3 buckets.
‫And we will send this event into an SNS Topic,
‫and we will subscribe many SQS Queues to the SNS Topic
‫as a Fan-out pattern.
‫But we could also subscribe other type of application,
‫email, Lambda functions, et cetera, et cetera.
‫And then what we get from this, is that the message
‫of the events happening in Amazon S3
‫is going to go to many different destinations
‫thanks to this fan out pattern.
‫Another architecture is that you can send data directly
‫from SNS to Amazon S3 through Kinesis Data Firehose.
‫So, because SNS has a direct integration with KDF,
‫then your buying service can send data into an SNS topic.
‫Then Kinesis Data Firehose,
‫KDF will receive that information.
‫And then from Kinesis Data Firehose,
‫you can send it into your Amazon S3 bucket.
‫Or, for that matter,
‫any supported KDF specific destination
‫which allows you to be really extensible
‫in the way you want to maybe persist your messages
‫from your SNS Topic.
‫So, we can apply the fan-out pattern to FIFO Topics as well.
‫So, Amazon SNS has a FIFO a FIFO capability,
‫which is first, first out,
‫which gives ordering a messages in the topic.
‫So, the producers send the messages 1, 2, 3, 4.
‫And the subscribers can only be an SQS FIFO Queue for now
‫which is receiving messages 1, 2, 3, 4, in order.
‫So, the idea is that with SNS FIFO,
‫we get the same features as SQS FIFO,
‫we get Ordering by Message Group ID.
‫We get Deduplication using a Deduplication ID
‫or Content Based Deduplication,
‫and we can have SQS FIFO Queues as subscribers
‫of the FIFO SNS Topic.
‫In terms of throughputs, you're limited.
‫You get the same throughputs as the SQS FIFO Queue
‫because right now only SQS FIFO Queues
‫can read from SNS FIFO Topics.
‫So, why do we need this?
‫Well, in case you wanna do a fan out using SQS FIFO.
‫So, you need fan out, ordering, and deduplication.
‫So, the buying service will send the data
‫into an SNS FIFO Topic, and then it will fan out
‫to two SQS FIFO Queue, which can have the Fraud Service
‫and a Shipping Service read from FIFO queues as well.
‫One last feature of SNS that can be really handy
‫regarding the fan-out pattern,
‫is that you can do message filtering in SNS.
‫So, what is message filtering?
‫Well, it's a JSON policy that is used to filter messages
‫sent to the SNS Topics subscription.
‫So, if a subscription doesn't have a filter policy,
‫it will receive every message and that's a default behavior.
‫But, let's take an example of what happens
‫when we set up a message filtering policy.
‫So, we have a Buying Service
‫and it sends transactions into an SNS Topic.
‫For example, the transactions look like
‫there is an order number.
‫There's a product, for example, a pencil.
‫The quantity, four,
‫and the state, which is Placed.
‫Now, we want to create an SQS Queue
‫just for Placed orders.
‫Not all the orders, but just for Placed orders.
‫So for this, we will subscribe the SQS Queue
‫into the SNS Topic, and we'll apply a filter policy in JSON
‫and we will specify in a policy
‫that we want to have States equals Placed.
‫And so only the messages matching the policy
‫will go into the SQS Queue.
‫But then, we can have an SQS Queue for canceled orders.
‫And so, we can create our own filter policy
‫for canceled orders and have those coming
‫from the same SNS Topic into the SQS Queue.
‫And so the place orders
‫and the cancel orders SQS Queue
‫will not have the same messages.
‫We can also use the same filter policy, the canceled one,
‫to create an email subscription for canceled orders.
‫We could have a filter policy for declined orders
‫for example, and as another SQS Queue.
‫Or we can create an SQS Queue without a filter policy
‫to have all messages from that SNS Topic.
‫So, using all these fan-out patterns and message filtering,
‫FIFO Queues, and FIFO Topics,
‫we get lots of different possibilities
‫and the exam will try to test you on all of those.
‫So, that's it for this lecture.
‫I hope you liked it, and I will see you in the next lecture.