EKS Forensics & Incident Response
We’ve continued our series on Cloud Forensics and Incident Response training with this video tutorial on AWS EKS:
How do you respond to a compromised EKS Container or Node?
If you’ve identified a potentially compromised container in EKS, there are two potential ways forward:
- If the container is running on an underlying EC2, then refer to the suggested steps above for immediate actions.
- If the container is running on Fargate, then collect any data required for later analysis before subsequently suspending it.
What EKS GuardDuty Detections are there?
https://aws.amazon.com/blogs/security/how-to-use-new-amazon-guardduty-eks-protection-findings/
Container Investigation Data Sources in AWS?
Amazon S3
Amazon EC2 — Hosting EKS/ECS
Inside Container — EKS/ECS on Fargate/EC2
EKS Audit / Control Plane Logs
- Shows: API Level Calls
- Usefulness: Medium
- Collected by: S3
Docker Container Filesystems
- Normally overlay2 versioned filesystem
- Contains all the files from all the containers
- Usefulness: High
- Collected by: EC2 EBS (API) or Cado Host (SSM/SSH)
Container Filesystems
- Live filesystem as seen by the container, Memory
- Contains all the files from all the containers
- Usefulness: Very High
- Collected by: Cado Host (ECS Exec/kubectl exec))
CloudTrail Logs
- Shows: API Level Calls
- Usefulness: Low
- Collected by: S3
Docker Logs
- Logs what containers were started, stopped
- Usefulness: Medium
- Collected by: EC2 Import or Cado Host
How do you Acquire an Amazon EKS System in Cado?
What is overlay2?
What AWS EKS Logs are Stored in AWS?
It’s important to also analyze AWS logs that are generated for EKS systems.
These contain metadata around starting and stopping containers.
Below you can see a view of AWS logs collected in Cado Response:
What Resources are available?
kube-forensics allows a cluster administrator to dump the current state
of a running pod and all its containers so that security professionals can perform offline forensic analysis.
We previously published a playbook dedicated to investigating compromises in EKS environments. Check out the GitHub repository with sample data taken from a compromised EKS system, and an associated talk on how to analyze it.
What Remediation is available?
