‫So we have seen asynchronous processing,
‫we have seen synchronous processing,
‫and now we are going to see Event Source Mapping.
‫So this is the last category of how Lambda
‫can process events in AWS.
‫So it applies to Kinesis Data Streams,
‫SQS and SQS FIFO queue and DynamoDB Streams.
‫So the common denominator of all of these things together
‫is that records need to be polled from the source.
‫So Lambda needs to ask the service to get some records
‫and then the records will be returned.
‫So that means that Lambda needs to poll from these services.
‫So in this case the Lambda function
‫is invoked synchronously.
‫So let's see this,
‫We have Kinesis and the Lambda service,
‫and if we configure Lambda to read from Kinesis,
‫then they will be internally
‫an Event Source Mapping that will be created.
‫And that is responsible for polling Kinesis,
‫and returning and getting the results back from Kinesis.
‫So Kinesis will return a batch to us.
‫And then once this Event Source Mapping
‫has some data for Lambda to process,
‫it's going to invoke our Lambda function synchronously,
‫with an event batch.
‫So at its core, this is how it works.
‫So there are two categories of Event Source Mapper.
‫The first one is streams and the second one is queues.
‫So let's deal with streams,
‫and streams apply to Kinesis Data Streams
‫and DynamoDB Streams,
‫and we'll see what DynamoDB Streams are
‫very, very soon.
‫So in case of streams, they will be an Event Source Mapping.
‫They will create an iterator for each shard,
‫so each Kinesis shard or DynamoDB Stream shard.
‫And the items will be processed in order at the shard level.
‫And so we can configure where to start to read from.
‫You can read it with just the new items,
‫or from the beginning of the shard
‫or from a specific timestamp.
‫Whenever an item is processed from a shard,
‫whether it be from Kinesis or DynamoDB,
‫they are not removed from the streams.
‫That means that other consumers can read the data
‫in Kinesis or DynamoDB,
‫which is how they work in the first place.
‫But I just wanted to reemphasize this.
‫So the use case for this
‫is either low traffic or high traffic.
‫So if you have a low traffic stream,
‫you can use a batch window to accumulate records
‫before processing.
‫So to make sure that
‫you invoke another function efficiently.
‫And then if you have a very high throughput stream
‫and you want to speed up processing,
‫you can set up Lambda to process multiple batches
‫in parallel at the shard level.
‫So here's a diagram from AWS blog.
‫And so we have a shard and there's a record processor and
‫there's a way to have a parallelism
‫to have multiple Lambda functions process the batch
‫within the same shard.
‫So you can have up to 10 batch processors per shard.
‫And for each batch they will be in-order processing,
‫at the partition key level.
‫So if you specify a partition key,
‫it will not be read in order entirely for the shard.
‫But each key within the shard will be read in order.
‫So this is how we can parallelize the processing
‫for Lambda function and your streams.
‫What about errors?
‫By default, if your function returns an error,
‫the entire batch is going to be reprocessed
‫until the function succeeds
‫or the items in the batch expire.
‫So this is very important.
‫Having an error in a batch can block your processing.
‫So to ensure in-order processing,
‫processing for the effected batch is paused
‫until the error is resolved.
‫And so you can manage that in several ways.
‫You can configure the event source mapping
‫to number one, discard old events.
‫or restrict the number of retries
‫or split the batch on errors.
‫This case, this is around Lambda timeout issue.
‫So if your Lambda function doesn't have enough time
‫to process the entire batch,
‫maybe you will have enough time to process half the batch.
‫And then in case you want to discard old events,
‫all the events can go to a destination.
‫And we'll see destinations in the next lectures.
‫So this is all about streams.
‫Now you need to know about queues.
‫So for the queues, it's for SQS and SQS FIFO.
‫And so you have the same idea.
‫The SQS queue will be polled
‫by a Lambda Event Source Mapping.
‫And then whenever a batch is returned,
‫your Lambda function will be invoked synchronously
‫with the event batch.
‫So in the case of SQS,
‫the Event Source Mapping will poll SQS
‫using Long Polling.
‫So it's going to be efficient.
‫And we can specify the batch size
‫from one to 10 messages.
‫And here is the configuration.
‫So just the batch size and the SQS queue.
‫Then there's some recommendations
‫from the websites of AWS,
‫which is to set the queue visibility timeouts
‫to six times the time out of your Lambda function,
‫which is configurable.
‫And then if you want to use a DLQ,
‫so if you want to make sure that
‫if there's a problem reading or processing a message
‫in SQS, it goes to dead-letter queue,
‫then you set up the DLQ on the SQS queue,
‫not on Lambda.
‫So we'd set up a DLQ on SQS
‫but not on Lambda. Why?
‫Because the DLQ for Lambda
‫only works for asynchronous invocations.
‫And this is a synchronous invocations.
‫Or as we'll see, we can also use
‫a Lambda destination for failures.
‫And this is going to be seen in the next lectures.
‫So now the bates of information on queues and Lambda,
‫and I'm sorry this is quite boring but I have to say it.
‫So Lambda supports in-order processing,
‫if you have a FIFO queue, so First-In, First-Out.
‫And the number of Lambda functions
‫that will be scaling to process your queue
‫will be equal to the number of active message groups.
‫So this is the group ID setting.
‫If you use a standard queue,
‫then the items will not be processed in order.
‫And for a standard queue, Lambda will scale
‫as fast as possible to read all the messages
‫in your standard queue.
‫If there is an error happening in your queue,
‫then the batches are going to be returned
‫to the queue as individual items
‫and might be processed in a different grouping
‫than the original batch.
‫Occasionally, the Events Source Mapping
‫might receive the same item twice from the queue,
‫even if no function error occurred.
‫So you need to make sure
‫to have item potent processing
‫for Lambda function in case that happens.
‫Finally, when they're processed by Lambda,
‫Lambda will delete the items from the queue,
‫and then they will never be seen again.
‫And finally, as I said,
‫you can configure the source queue to send the items
‫to a dead-letter queue if they can't be processed.
‫So hopefully all of this makes sense.
‫We'll see in the hands on how we can set it up.
‫And so what about the scaling?
‫So I already said it,
‫but we can go again to summarize the scaling
‫for the Event Mapper.
‫So for Kinesis Data Streams and DynamoDB Streams,
‫you get one Lambda invocation per stream shard.
‫Or if you use parallelization,
‫you can have up to 10 batches processed
‫per shard simultaneously.
‫And for SQS Standard,
‫I said Lambda will scale up pretty quickly.
‫So yes it does.
‫It adds 16 more instances per minute to scale up.
‫So it's pretty quick.
‫And then the maximum amount of batches per second
‫processed simultaneously, is 1000.
‫For SQS FIFO, it's a bit different.
‫So the messaging with the same group ID
‫will be processed in order no matter what.
‫And the Lambda function will scale up
‫to the number of active message groups.
‫Again, they're defined by the group ID.
‫So that's all for Lambda Events Mapping Sourcing,
‫and I know this is a pretty long boring theory lecture.
‫It will make a lot of sense in the next lecture
‫when we do the hands on,
‫but we had to go over it
‫and I would recommend you look again
‫at this lecture before your exam
‫because it's possible the exam will ask you
‫some pretty pointy details about Event Mappers.
‫All right, that's it. I will see you in the next lecture.