CloudWatch Logs: Powerful AWS Logging Tool

Arjun just deployed a new feature to his app.
The team was hyped. The users were excited.
But something broke… again.

“Where do I even see what went wrong?” Arjun muttered.

That’s when he met CloudWatch Logs — AWS’s built-in service for collecting and storing logs.

And everything changed.

---

💡 Why Use CloudWatch Logs?

Amazon CloudWatch Logs is AWS’s centralized log management service.
It’s where your apps, services, and resources can send their log data — securely, flexibly, and at scale.

Arjun learned this the hard way.
Now he uses CloudWatch Logs as his app’s black box recorder — capturing every crash, exception, and debug message in one place.

Before CloudWatch Logs, Arjun was:

• SSH-ing into EC2 just to tail log files
• Manually copying logs to debug
• Missing errors from Lambda functions or API Gateway

Now?

• All logs are in one place: CloudWatch Logs
• Searchable, visual, organized
• No more guesswork

---

📅 Retention: Keep What Matters

Arjun sets log retention policies like:

• ✅ 7 days for debug logs
• ✅ 30 days for API traffic
• ✅ Forever for security logs

You can choose between 1 day to 10 years — or keep them indefinitely.
No more bloated log bills. No more lost critical logs.

---

🔐 Security: Your Logs, Your Keys

By default, AWS encrypts all CloudWatch Logs.
But Arjun takes it further — using KMS to encrypt sensitive logs with his own keys.
Perfect for compliance and audits.

---

📦 How Do Logs Get Into CloudWatch?

Let’s make this crystal clear.

There are 2 ways services send logs into CloudWatch Logs:

Source	Description
EC2	Using CloudWatch Unified Agent to push logs
Lambda	Automatically sends logs for every execution
ECS	Streams container logs
Elastic Beanstalk	App logs routed via Beanstalk
API Gateway	Logs each API request
VPC Flow Logs	Captures network traffic metadata
CloudTrail	Sends filtered logs
Route 53	Logs DNS queries

✅ Some services do it automatically:

AWS Service	Sends Logs to CloudWatch Automatically?
Lambda	✅ Yes, by default
API Gateway	✅ Yes (if logging is enabled)
VPC Flow Logs	✅ Yes (once created)
CloudTrail	✅ Yes (when set to deliver to CloudWatch)
Route 53	✅ Yes (when query logging is enabled)

👉 These need minor configuration, but once done, logs just flow in.

---

⚙️ Some services need manual setup:

AWS Service	How to Send Logs
EC2	Install CloudWatch Agent or Unified Agent
ECS / Docker Containers	Set up log drivers (e.g., awslogs)
Elastic Beanstalk	Configure logging in environment settings

💡 Tip: Use the CloudWatch Unified Agent. It supports both logs and metrics and is the newer option.

---

🧱 How Are Logs Organized Inside CloudWatch?

Let’s break this down simply.

🔹 Log Group = A folder for your logs

You create one per app, service, or environment.

Example:

• /myapp/production
• /payment-service/dev

🔸 Log Stream = A file inside that folder

Each stream holds logs from one instance, container, or Lambda execution.

Example:

• A single EC2 instance’s logs
• One Lambda function’s output
• A specific container’s logs

📌 Think of it like this:

Log Group = Folder
Log Stream = Log file inside that folder

---

🧠 Arjun’s Setup Looks Like This:

For his EC2 App:

• Log Group: /ridego/production
• Log Streams: One for each EC2 instance

For his Lambda Functions:

• Log Group: /aws/lambda/processPayment
• Log Streams: One for each invocation

For his API Gateway:

• Log Group: /aws/api-gateway/ridegoAPI
• Log Streams: One for each API request

And they automatically show up in CloudWatch once logging is enabled.

---

🔍 Querying Logs with CloudWatch Logs Insights

Arjun used to scroll endlessly through raw logs. Not anymore.

Now he uses CloudWatch Logs Insights — a built-in log query engine.

📊 What It Does:

• Search logs by time range, keyword, IP, or error
• Aggregate stats (like how many 500 errors occurred)
• Visualize patterns and trends
• Save and reuse queries in dashboards

Example query:CopyCopy

sqlCopyEditfields @timestamp, @message  
| filter @message like /ERROR/  
| sort @timestamp desc  
| limit 25

He even added it to his CloudWatch Dashboard, so his team sees real-time error counts.

---

🚚 Where Else Can Logs Go?

Arjun doesn’t just store logs — he moves them too:

🟡 Batch Export to S3

• Triggered via CreateExportTask
• Good for archival
• Runs every 12 hours
• ❌ Not real-time

🟢 Real-Time Streaming (Log Subscriptions)

Send logs as they arrive to:

• Kinesis Data Streams (for analytics)
• Kinesis Data Firehose (auto-load to S3, Redshift, OpenSearch)
• AWS Lambda (custom processing)
• Amazon OpenSearch Service (for log search/visualization)

💡 Use Subscription Filters to filter what logs to forward.

---

🌍 Cross-Account Logging Made Simple

Arjun didn’t want to check logs in every AWS account.
He wanted one place to see all logs — across accounts and regions.

Here’s how he did it (in plain English):

1. In the main account, he created a destination (like a Kinesis Stream).
2. Other AWS accounts got permission to send logs there.
3. He connected each source account to the central log stream.

✅ Now all logs from different accounts flow into one central location.

📊 One dashboard. One stream. No more account hopping.

---

🧠 Pro Tips for SAA Learners

• Know the difference between:
  • Log Groups vs Log Streams
  • Batch Export (S3) vs Real-Time Stream (Kinesis)
• Learn CloudWatch Logs Insights syntax
• Understand how Lambda, ECS, API Gateway send logs
• Remember: use CloudWatch Unified Agent now
• Explore cross-account log aggregation

---

🔍 How Arjun Debugs Now

Before:

Log in to server → find the right file → scroll like mad

Now:

Open CloudWatch Logs → Search by keyword or time → Done

He even uses Log Insights to:

• Find all errors from the past hour
• Track how many times a specific error happened
• Visualize log data trends

---

🎯 Final Recap

Concept	Meaning
CloudWatch Logs	Central place for your AWS logs
Log Group	Folder that stores logs for a service or app
Log Stream	Individual log files inside a log group
Automatic logging	Lambda, API Gateway, etc. do this
Manual setup	EC2, ECS need agents or config
CloudWatch Logs Insights	Tool to search and analyze logs

---

🧘 Arjun Now Sleeps Better

Thanks to CloudWatch Logs:

• He sees crashes as they happen
• He knows why things break
• He acts before users complain

CloudWatch Logs didn’t just store data.
It brought clarity to chaos.

---

FAQ

❓ What is AWS CloudWatch Logs?

AWS CloudWatch Logs is a centralized service for storing, searching, and analyzing logs from AWS services and applications.

❓ How do you send logs to CloudWatch Logs?

You can send logs automatically from services like Lambda and API Gateway, or manually using the CloudWatch Unified Agent for EC2 and ECS.

❓ What is CloudWatch Logs Insights?

CloudWatch Logs Insights is a query engine inside CloudWatch that lets you search, filter, and visualize logs in real time.

Follow me for more such content

• My Site
• My Blogs
• My YouTube Channel
• AWS Community
• Medium
• LinkedIn
• Instagram
• Github
• Dev
• X