1
00:00:00,170 --> 00:00:01,960
Now let's talk about CloudWatch Logs.

2
00:00:01,960 --> 00:00:05,160
So when you want to store your logs in AWS,

3
00:00:05,160 --> 00:00:06,420
the best place is CloudWatch Logs.

4
00:00:06,420 --> 00:00:07,253
So the idea is that you're going

5
00:00:07,253 --> 00:00:09,240
to group these logs into log groups,

6
00:00:09,240 --> 00:00:11,250
and this is a name that you want to give it,

7
00:00:11,250 --> 00:00:13,530
but usually it's representing an application.

8
00:00:13,530 --> 00:00:15,900
And within each log group you have log streams.

9
00:00:15,900 --> 00:00:18,450
And they represents instances within the application

10
00:00:18,450 --> 00:00:19,680
or different log file names

11
00:00:19,680 --> 00:00:21,460
or different containers, and so on,

12
00:00:21,460 --> 00:00:23,480
then you define log expiration policy.

13
00:00:23,480 --> 00:00:25,760
For example, you may never want the log to expire,

14
00:00:25,760 --> 00:00:27,810
or do you want it to be deleted after 30 days and so on,

15
00:00:27,810 --> 00:00:30,700
because you are paying for storage on CloudWatch Logs,

16
00:00:30,700 --> 00:00:31,533
then from Cloudwatch Logs,

17
00:00:31,533 --> 00:00:33,280
you can export the logs through multiple places,

18
00:00:33,280 --> 00:00:35,630
such as Amazon S3, Kinesis Data streams,

19
00:00:35,630 --> 00:00:39,190
Kinesis Data Firehose, Lambda, and ElasticSearch.

20
00:00:39,190 --> 00:00:42,120
Now what types of logs can go into CloudWatch logs.

21
00:00:42,120 --> 00:00:44,162
Well we can send the logs using the SDK

22
00:00:44,162 --> 00:00:46,390
or the CloudWatch Logs Agent

23
00:00:46,390 --> 00:00:48,430
or the CloudWatch Unified Agent.

24
00:00:48,430 --> 00:00:51,220
Now the CloudWatch Unified Agent send logs to Cloudwatch

25
00:00:51,220 --> 00:00:54,260
into the CloudWatch Logs Agent is now sort of deprecated.

26
00:00:54,260 --> 00:00:56,580
You have Elastic Beanstalk, which is used to collect logs

27
00:00:56,580 --> 00:00:58,114
from the application directly into CloudWatch,

28
00:00:58,114 --> 00:01:00,010
ECS will send the logs directly

29
00:01:00,010 --> 00:01:01,867
from the containers into CloudWatch.

30
00:01:01,867 --> 00:01:05,489
Lambda will send logs from the functions themselves,

31
00:01:05,489 --> 00:01:07,090
VPC Flow Logs will send logs

32
00:01:07,090 --> 00:01:09,492
specific to your VPC metadata network traffic,

33
00:01:09,492 --> 00:01:12,550
API gateway will send all the requests

34
00:01:12,550 --> 00:01:14,710
made to the API gateway into CloudWatch logs,

35
00:01:14,710 --> 00:01:16,230
CloudTrail, you can send the logs there

36
00:01:16,230 --> 00:01:17,580
based on the filter

37
00:01:17,580 --> 00:01:19,690
and Route53 will log all the DNS queries

38
00:01:19,690 --> 00:01:22,040
made to its service.

39
00:01:22,040 --> 00:01:23,720
So another thing you can do that's very important

40
00:01:23,720 --> 00:01:26,060
is to define metric filter and insights.

41
00:01:26,060 --> 00:01:27,690
So the idea is that you have your Cloudwatch Logs

42
00:01:27,690 --> 00:01:29,400
and you can use your filter expressions, for example,

43
00:01:29,400 --> 00:01:32,310
to find all the specific IP within, within logs.

44
00:01:32,310 --> 00:01:34,590
So find the log lines where that IP appears

45
00:01:34,590 --> 00:01:36,850
or find every log line in your logs

46
00:01:36,850 --> 00:01:38,480
that contains the word errors.

47
00:01:38,480 --> 00:01:40,140
And then thanks to this metric filter,

48
00:01:40,140 --> 00:01:43,370
you can start counting these occurrences and then these

49
00:01:43,370 --> 00:01:44,980
becomes a metric, okay?

50
00:01:44,980 --> 00:01:48,670
And this metric can be linked into a CloudWatch alarm.

51
00:01:48,670 --> 00:01:50,800
Then the other feature that's really cool to discuss

52
00:01:50,800 --> 00:01:52,510
is CloudWatch Logs Insights.

53
00:01:52,510 --> 00:01:54,140
So the idea is that the CloudWatch Logs Insights,

54
00:01:54,140 --> 00:01:56,350
You can query logs and add these queries

55
00:01:56,350 --> 00:01:58,300
into CloudWatch Dashboards directly.

56
00:01:58,300 --> 00:02:00,570
And some common queries are added directly by AWS,

57
00:02:00,570 --> 00:02:03,100
and this is a very easy language to use.

58
00:02:03,100 --> 00:02:06,330
So first let's talk about the S3 export.

59
00:02:06,330 --> 00:02:08,656
So you send from CloudWatch into Amazon S3,

60
00:02:08,656 --> 00:02:11,130
and this can take up to 12 hours

61
00:02:11,130 --> 00:02:13,420
to become available for export.

62
00:02:13,420 --> 00:02:16,026
And the API call is CreateExportTask,

63
00:02:16,026 --> 00:02:18,056
but this is going to be done in its own time.

64
00:02:18,056 --> 00:02:20,356
And so it's not near real time or real time.

65
00:02:20,356 --> 00:02:23,130
Instead, if you want to stream logs from CloudWatch logs,

66
00:02:23,130 --> 00:02:25,100
you need to use subscriptions,

67
00:02:25,100 --> 00:02:26,047
now subscriptions are what?

68
00:02:26,047 --> 00:02:28,090
Well subscriptions are a filter

69
00:02:28,090 --> 00:02:29,910
that you apply on top of your Cloudwatch logs,

70
00:02:29,910 --> 00:02:31,930
and then you can send it to a destination.

71
00:02:31,930 --> 00:02:33,560
So it could be a Lambda function.

72
00:02:33,560 --> 00:02:37,380
For example, you define a custom or one that is used by AWS.

73
00:02:37,380 --> 00:02:40,400
If you want to send data into Amazon Elasticsearch directly,

74
00:02:40,400 --> 00:02:41,818
or it could be Kinesis data Firehose,

75
00:02:41,818 --> 00:02:43,540
if you want to send it for example,

76
00:02:43,540 --> 00:02:45,380
to Amazon S3 in near real time.

77
00:02:45,380 --> 00:02:47,370
So this is a faster alternative

78
00:02:47,370 --> 00:02:48,203
to the one I just showed you

79
00:02:48,203 --> 00:02:51,710
from before using the exports from CloudWatch to S3,

80
00:02:51,710 --> 00:02:54,190
or you could use Kineses Data Streams, for example,

81
00:02:54,190 --> 00:02:56,658
to send the data into Kinesis Data Firehose

82
00:02:56,658 --> 00:03:00,793
Kinesis Data Analytics, Amazon EC2 or Lambda, and so on.

83
00:03:01,690 --> 00:03:02,523
Finally, with Cloudwatch Logs,

84
00:03:02,523 --> 00:03:04,960
you can do some log aggregation across accounts

85
00:03:04,960 --> 00:03:06,160
and across regions.

86
00:03:06,160 --> 00:03:09,050
So you may have like multiple accounts with,

87
00:03:09,050 --> 00:03:11,330
for example, a region that has a subscription filter.

88
00:03:11,330 --> 00:03:15,110
That's sent into Kinesis Data Streams in a common accounts,

89
00:03:15,110 --> 00:03:15,950
same for account B.

90
00:03:15,950 --> 00:03:18,050
So region two, same architecture and so on.

91
00:03:18,050 --> 00:03:20,780
And so we can centralize all these logs together into

92
00:03:20,780 --> 00:03:23,100
Kinesis Data Streams, for example, and then Data Firehose.

93
00:03:23,100 --> 00:03:25,036
And then for example, Amazon S3.

94
00:03:25,036 --> 00:03:26,390
Okay.

95
00:03:26,390 --> 00:03:28,950
So that's it for the overview of how Cloudwatch Logs work.

96
00:03:28,950 --> 00:03:31,783
I hope you liked it, and I will see you in the next lecture.

