AWS Logging and Analysis - Part 2 - Intro to CloudWatch and Sending events to CloudWatch Logs and S3 bucket

Hello there!!!

Welcome to the AWS Logging and Analysis Quest...

In the previous article, i.e., Part 1 of this quest, we had a discussion about CloudTrail logging in AWS for each and every event. Here, in this article, we will look into another service for log management. The name of the service is CloudWatch. 

CloudWatch is a monitoring service for AWS cloud resources and the applications we run on AWS. A lot of things we can do using CloudWatch which includes:

1. Collection and tracking of events on the basis of Metrics

2. Monitoring resources and triggering Alarms for a pre-decided threshold

3. Automatically reacting to changes in AWS resources

4. Integrate CloudWatch Alarms to other services to take incident specific actions

5. Real-time Logging and Analysis

We will be covering all of these mentioned features of CloudWatch in the upcoming posts but focus mainly on Real-time universal logging of AWS resources. 

Let's start with CloudWatch by selecting, Services --> ManagementTools --> CloudWatch. A new portal will appear as CloudWatch Management Console with a number of options appearing in the left as Dashboard, Alarms, Events, Logs and Metrics.

To get the complete log stream from CloudTrail to CloudWatch, we need to configure a CloudWatch Log Group. This can be done using CloudWatch Management Console or CloudTrail Console. I will be going through the latter one because,

a) A universal Trail can stream all data directly to CloudWatch Log Group
b) Its advantageous to configure the S3 bucket so that all the data will also get dumped to it

Steps to configure CloudTrail (with name - "all-logs-trail123") for sending events to CloudWatch Logs and S3 bucket (with name - "aws-logs-bucket123"):

A) Click Services --> Management Tools --> CloudTrail --> Create Trail
B) Enter "Trail Name" as "all-logs-trail123".
C) Select "Yes" in "Apply trail to all regions" for universal logging. This means logs from all regions will be collected into this trail.

                           

D) In Management events, Select "All" in "Read/Write events".

E) In Data events, for S3 check mark the "Select all S3 buckets in your account" and similarly for Lambda check mark "Log all current and future functions".

F) In Storage location, select "Yes" for creation of new S3 bucket and enter S3 bucket name as "all-logs-bucket123".

G) In the Advanced drop down, let the settings be default as described in the image below.

H) Finally hit the "Create" button.

I) After configuration, the trail will be accessed using "View Trails" option.

J) Now, click on the trail name "all_logs_trail123" to configure CloudWatch Logs Log Group.

K) Go down and click on "Configure" for CloudWatch Log Group. In drop down pane, a default value will appear for "New or existing log group" as "CloudTrail/DefaultLogGroup". Also, we can provide the Tags to this trail. Post that hit the apply button.

L) A new pane will appear which is to inform you that CloudTrail_CloudWatchLogs_Role is required which grants the CloudTrail permissions to CloudWatch API calls. Hit the button "Allow".

M) Finally, we can see the green tick in "Status" of  Trail configured with S3 bucket and CloudWatch Logs Log Group.

Please go through each step thoroughly. We have configured the trail for universal logging, which means in any region if any kind of Data or Management event occurs, it will be logged in both the S3 bucket as well as in the CloudWatch Log Group.