Logging from Java applications in AWS using CloudWatch

When running Java applications on AWS cloud platform using either EC2 or ECS services, one needs to think about the logging solution to be used. There are multiple different solutions available. In case of EC2, it is possible to log the information to a file kept either on EC2 EBS volume or EFS. This is a simple solution. However, when the EC2 instances are used in a disposable (temporary) manner, the attached EBS volume becomes ephemeral, and would not be suitable for long term storage of log files. EFS service provides a better centralized and durable alternative in that case, but you will still need to setup some additional solution in case you want to monitor and analyse the raw log files for errors, incidents and troubleshooting purposes. Additionally, when the application is running on more than one EC2 instance (horizontal scaling), in case of EBS volume the files will be stored on multiple EC2 instances instead of in a centralized location. An additional process for harvesting all the application log files from all the EC2 instances into a central location for analysis will be required. The AWS CloudWatch service provides the capability for applications to send their log files to a central location using either CloudWatch Logs API / SDK or by means of the CloudWatch agent. ECS service for the Fargate execution engine uses the awslogs driver for sending application log information within containers to CloudWatch Logs. For the EC2 execution engine, one can choose to use the CloudWatch agent instead of the awslogs driver.

This blog post explains the basic concepts of CloudWatch Logs service and will show you how to send application log information from an EC2 instance when using the CloudWatch agent. A sample Java application based on Spring Boot framework is provided which makes use of Logback logging framework to log to a file on an EC2 instance.

In this blog post, the following AWS services are used:

• Elastic Compute Cloud (EC2)

• CloudWatch

CloudWatch Logs

CloudWatch Logs service allows for storing, monitoring and accessing your log files from Amazon EC2 instances, AWS CloudTrail, Route 53, and other sources. Using the CloudWatch Logs service, you can centralize the logs from all your systems, applications, and AWS services that you use. The CloudWatch Management Console provides you the means to easily view all your logs, search for specific error codes or patterns within the log files, filter log entries on specific fields, or archive for future analysis. It allows you to see all your logs, regardless of their source, as a single and consistent flow of events ordered by time. It comes with a query language that you can use to create custom computations and visualize log data in dashboards.

To group related log data, you can create a log group. A log group can contain zero or more log streams to where you send the log data. For instance, if your application is named blog-aws-java-logging and you are running it on multiple EC2 instances, you can create one log group named blog-aws-java-logging and one log stream per EC2 instance. In the name of the log stream, you can incorporate the EC2 instance id by using the predefined variable {instance_id}. For instance: blog-aws-java-logging-{instance_id}.log

Both for log groups as well as log streams, as part of the name, you can use {instance_id}, {hostname}, {local_hostname}, and {ip_address} as variables within the name. {hostname} retrieves the hostname from the EC2 metadata, and {local_hostname} uses the hostname from the network configuration file.

In the below diagram you can see one log group and three log streams which receive the log data from EC2 instances where each EC2 instance sends its log data to a unique log stream within the same log group.

By default, the log data in CloudWatch is kept indefinitely. You can however adjust the retention policy for each log group, keeping the indefinite retention period, or choosing a retention period between 1 day and 10 years.

Using CloudWatch Agent

In this section, I am going to demonstrate how to send Java application log files to CloudWatch Logs service from an EC2 instance using the CloudWatch agent. If you decide to send the application log data directly to the CloudWatch Logs by means of CloudWatch Logs API / CloudWatch Logs Java SDK, then the best way would be to integrate the AWS service with your favourite logging framework. For instance, if you use the Logback framework, you will then need to write your own custom log appender. Include your custom log appender within the Logback configuration file to allow the log information to be sent to the AWS CloudWatch Logs service when the application runs on the AWS cloud platform. Use springProfile element within the Logback configuration file to distinguish between the different environments in case you don’t need to log to AWS from your local or the development environment.

Create the log group blog-aws-java-logging using the AWS CloudWatch Management Console.

Create an IAM role with CloudWatchAgentServerPolicy policy and assign this role to the EC2 instance that you are going to start. This will allow the agent to access the CloudWatch Logs API from the EC2 instance. Start an EC2 instance based on Amazon Linux 2 AMI.

Now, let us install the agent using the Command Line. It is also possible to install the agent using the Systems Manager. Refer to the CloudWatch documentation for more information.

Run the below command on EC2 instance to install the agent.

sudo yum install amazon-cloudwatch-agent

Upload the agent configuration file to the server where you are going to run the agent. Upload it int eh following location: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Enter the following command.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

After running the above command, a systemd service will be created for the agent, and the agent will be started. Use the below commands to check the status, stop and start the agent when needed.

sudo /bin/systemctl status amazon-cloudwatch-agent
sudo /bin/systemctl stop amazon-cloudwatch-agent
sudo /bin/systemctl start amazon-cloudwatch-agent

Install the OpenJDK11 to be able to run the sample Java application.

sudo amazon-linux-extras install java-openjdk11

Create the blog-aws-java-logging directory in /home/ec2-user folder of the EC2 instance.

Download and build the aws-java-logging application. Upload the aws-java-logging.jar file on the EC2 instance in the following location: /home/ec2-user/blog-aws-java-logging

Run the below commands to start the application.

cd /home/ec2-user/blog-aws-java-logging
java -jar ./aws-java-logging.jar --spring.profiles.active=prod

After logging some data to a file on the EC2 instance, the application will stop. You can find the data that has been logged in the logs subdirectory in the blog-aws-java-logging.log file.

In the AWS CloudWatch Management Console, you can see that a log stream has been created.

Summary

In this blog post you were introduced to CloudWatch Logs, and how to send application log files to this service using the CloudWatch agent. A sample Java application with a CloudWatch agent configuration file was provided for demonstration purposes. You have also discovered that it is possible to send the application log data by directly using the CloudWatch API / Java SDK from the application, and integrating it with a logging framework. However, this use case is not demonstrated in this blog post.