Logs Agent

The Logs Agent is required for sending operating system and application logs to the Logs service from EC2 instances. It's written in Python and is designed to be cross platform. The agent can use an instance profile to authenticate.

Configuration

The agent is configured in /etc/awslogs. awscli.conf contains AWS CLI configuration for authentication:

[plugins]
logs = cwlogs

[default]
region = us-east-1

Note that the default section should include aws_access_key_id and aws_secret_access_key values if not using EC2 instance profiles for authentication.

Sources for log collection can be added to awslogs.conf:

[/some/file]
file = /some/file
log_group_name = some/name
log_stream_name = {instance_id}
datetime_format = %b %d %H:%M:%S
initial_position = start_of_file
buffer_duration = 5000

The keys:

file contains the filename. Conventionally this should match the section name.
log_group_name specifies the name of the parent log group.
log_stream_name names the individual log stream. Generally this should include the instance ID.
datetime_format is used to parse out message times from the log file.
initial_position specifies where to start processing data; use end_of_file for services that don't truncate their log files on start to avoid ingesting the entire log when the agent starts.
buffer_duration species the ingest interval in milliseconds.

Note the use of the {instance_id} placeholder, which will be expanded to the EC2 instance ID by the agent.

Configuration