October is Cyber Security Awareness Month, which is a great opportunity for companies and individuals to review and improve their cyber security processes and knowledge. At Nettitude, we will be releasing a new blog post every week of Cyber Security Awareness Month on our latest cyber security research, as well as our insights on the latest industry news and trends. We hope you’ll find them helpful, and as always please contact us with any questions.
+++
As cloud infrastructure has become common, it has also become common for penetration testers to find themselves attacking clients that rely on AWS or Azure environments, for example, for handling, storing, and processing critical data.
There are many new and interesting attack paths an adversary can take once they have obtained some sort of access to the environment.
For AWS, this is a short indicative list:
- Pivoting via Roles and Policies.
- Modifying Lambda functions.
- Modifying CloudWatch logs.
- Exfiltrating database backups.
- Exfiltrating EC2 images and snapshots.
We have written a technical article that details the steps required for exfiltrating EC2 backups (snapshots), and can be found here: https://labs.nettitude.com/blog/how-to-exfiltrate-aws-ec2-data/
Cloud Security Controls
Traditional security controls such as IP Whitelisting, MFA, and account lockout after subsequent login failures are supported to varying degrees, require hacks, and in some cases are outright not advised by the cloud provider.
For example, account lockout after subsequent failed login attempts is not currently supported by AWS. Yes, adversaries can brute force their way into the AWS Management Console, assuming no MFA is implemented. One quick fix for this is to create a Lambda function that works off the back of CloudTrail logs – these logs however are not generated and processed in real time.
Another example is IP Whitelisting, both to the AWS Management Console and to the CLI/API access. Currently there is no easy and ubiquitous way of enforcing this. AWS provides a guide that helps towards this (https://aws.amazon.com/premiumsupport/knowledge-center/iam-restrict-calls-ip-addresses/) but within the same document the following note highlights its immaturity: "Note: It's a best practice not to use the aws:SourceIp condition key."
Due to the issues listed above, it is not uncommon for an attacker to be in a position to bypass the majority of the perimeter controls via the AWS CLI interface. This means that even if a developer needs to jump through multiple security gateways in order to access a private EC2 instance, by using AWS CLI an adversary can obtain direct access to its data.
Detection and Response
The process of creating, sharing, and deleting snapshots leaves a number of log entries, including the following:
CreateSnapshot and DeleteSnapshot events can be very frequent in medium-to-large scale AWS deployments with automated backup processes. However, the ModifySnapshotAttribute is more interesting:
The sourceIPAddress is the IP address from where the attacker issued the AWS CLI command. The userAgent, as you can see, also includes the hostname from where the AWS CLI command was issued. @thesubtlety has written an interesting post on how to overcome this: https://www.thesubtlety.com/post/patching-boto3-useragent/
The most likely IOC here is the createVolumePermission entry, where it also identifies who this snapshot was shared with.
Having said that, AWS Logs operate on an approximate 15 minute delay. That means it can take up to 15 minutes for an AWS CLI command to appear in CloudTrail.
Recommendations
- Deny the ec2:CreateVolumePermission action in all groups, roles, and users. It is unlikely that this will be required in most environments.
- If the ec2:CreateVolumePermission action is required to satisfy business requirements, create a CloudWatch alert that goes off when the volume is shared with a non-whitelisted userId.
- Key rotation should be implemented in all access keys, with more frequent rotation for highly-privileged principals.
- Employ a granular permission policy based on the need to know principle.
- Enforce MFA both to the management console and to the AWS CLI/API.
Conclusion
In our Nettitude Labs article we have shown how an adversary who has compromised a set of AWS access keys can bypass firewalls, private VPCs and bastion hosts, and directly exfiltrate the contents of an EC2 instance without needing logical access to the host itself nor requiring the root’s SSH keys.
There are many variations of this technique, including modifying the permissions of an S3 bucket that the snapshots reside in. It comes to no surprise that traditional security principles such as defence in depth, key rotation, and granular access controls can help minimise the risk of an AWS environment data breach.