AWS Forensics & Incident Response

7 min readJul 3, 2023

Continuing our series of training videos on incident response and forensics in the cloud — the next video covers responding to incidents in AWS at a high level:

What Incidents might you see in AWS?

See “AWS Incident Response in your Pyjamas” (great talk!) by Paco Hope @ AWS

https://owasp.org/www-chapter-london/assets/slides/OWASPLondon-IR-In-Your-Pyjamas-Paco-Hope-20190213-PDF.pdf

What are… Cloud security incident domains?

Service domain — Incidents in the service domain might affect your AWS account, AWS Identity and Access Management (IAM) permissions, resource metadata, billing, or other areas.

A service domain event is one that you respond to exclusively with AWS API mechanisms, or where you have root causes associated with your configuration or resource permissions, and might have related service-oriented logging.

Infrastructure domain — Incidents in the infrastructure domain include data or network-related activity, such as processes and data on your Amazon Elastic Compute Cloud (Amazon EC2) instances…

Application domain — Incidents in the application domain occur in the application code or in software deployed to the services or infrastructure…

From/See “AWS Security Incident Response Guide”

https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-response-guide/introduction.html

What are Common Attacks in the Cloud?

Misconfiguration
Stolen Credentials — Where do you find them?
Phishing — Recent examples
Poisoned Gold Image or Library
S3…

How else might you know you have a problem?

Graphic from “Cloud Security: Defense in Detail if Not in Depth” by SANS

An email from AWS…
Weird IAM
Sudden increase in billing
High CPU Usage…

What is Shared Responsibility? What happens in IR?

Graphic from “Shared Responsibility Model” by AWS

Why is responding to incidents in the cloud hard?

Graphic from “Cloud Security: Defense in Detail if Not in Depth” by SANS

Steps to Responding in AWS

When you are investigating a compromise of a cloud environment, there are a few key steps that you should follow:

Identify the scope of the incident: The first step is to identify the scope of the incident. This means determining which resources were affected and how the data was accessed.
Collect evidence: The next step is to collect evidence. This includes collecting log files, network traffic, metadata, and configuration files.
Analyze the evidence: The next step is to analyze the evidence. This means looking for signs of malicious activity and determining how the data was compromised.
Respond to the incident and contain it: The next step is to respond to the incident. This means taking steps to mitigate the damage and prevent future incidents. For example with a compromise of an EC2 system in AWS, that may include turning off the system or updating the firewall to block all network traffic, as well as isolating any associated IAM roles by adding a DenyAll policy. Once the incident is contained, that will give you more time to investigate safely in detail.
Document the incident: The final step is to document the incident. This includes creating a report that describes the incident, the steps that were taken to respond to the incident, and the lessons that were learned.

What data do you have in AWS

Getting access to the data required to perform an investigation to find the root cause is often harder in the cloud than it is on-prem. That’s as you often find yourself at the mercy of the data the cloud providers have decided to let you access. That said, there are a number of different resources that can be used for cloud forensics, including:

AWS EC2: Data you can get includes snapshots of the volumes and memory dumps of the live systems. You can also get cloudtrail logs associated with the instance.
AWS EKS: Data you can get includes audit logs and control plane logs in S3. You can also get the docker file system, which is normally a versioned filesystem called overlay2. You can also get the docker logs from containers that have been started and stopped.
AWS ECS: You can use ecs execute or kubectl exec to grab files from the filesystem and memory.
AWS Lambda: You can get cloud trail logs and previous versions of lambda.

What logging is in AWS? Where do you look?

https://cloudstudio.com.au/2022/05/14/monitoring-service-aws-azure-gcp-part1/

What is AWS IAM?

AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources.

With IAM, you can centrally manage permissions that control which AWS resources users can access.

You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources.

https://blog.gitguardian.com/aws-iam-security-best-practices/

What AWS IAM Logging is there?

AWS CloudTrail captures all API calls for IAM and AWS STS as events, including calls from the console and API calls.

AWS Identity and Access Management Access Analyzer helps you identify the resources in your organization and accounts, such as Amazon S3 buckets or IAM roles, that are shared with an external entity. This helps you identify unintended access to your resources and data, which is a security risk.

https://docs.aws.amazon.com/IAM/latest/UserGuide/security-logging-and-monitoring.html

Also check-out the policy simulator

How do I block access in AWS IAM?

See “Identity & Access Management” in AWS Well-Architected Labs

https://www.wellarchitectedlabs.com/security/300_labs/300_incident_response_with_aws_console_and_cli/2_iam/

See “Incident_Response_Playbook_AWS_IAM” Jupyter Notebook for how to investigate IAM/CloudTrail

How do you respond to a compromised EC2?

If you’ve identified a potentially compromised EC2 instance -

There are a number of immediate actions you can take:

To limit the possibility of data theft, change the security group to one that doesn’t allow any outbound internet access.
Identify if there was an Instance Profile attached to the EC2. If there was, check CloudTrail logs to see if it may have been abused to access other resources in AWS.
Take a snapshot of the EC2, to enable forensic analysis later on.

What is Virtual Private Cloud (VPC)? How can an attacker move?