How to download folders from S3 bucket
Download the folder from S3
- When using the console, users cannot download files in buckets as folders. This time, I will show you how to download folders using CLI or tools instead of the console.
- There are many tools to access S3, but this time I will explain using official tools and popular tools.
Download using CLI
Download the exported Logs folder from S3
- You can use aws cli to download folders from your S3 bucket.
- For example, if you have ELB or CloudFront access logging enabled, your S3 bucket will contain many exported folders and files. The following command is an example for downloading a folder from an S3 bucket.
aws s3 cp s3://<BUCKET_NAME>/<AAA>/<BBB>/<Folder> ./<Copyed Folder> --recursive
$ aws s3 cp s3://<BUCKET_NAME>/<AAA>/<BBB>/exportedlogs ./apigw_logs --recursive --profile <profile_name>
Extract the compressed file
- Files exported from CloudWatch Logs are compressed as gzip. So you can combine find and xargs in a Linux environment to gunzip them all at once.
find . -name '000000.gz' | xargs -n1 gunzip
$ find . -name '000000.gz' | xargs -n1 gunzip
Download using tool
Use Cyberduck to connect to Amazon S3.
- I’m using Cyberduck for my scp / sftp client. Next, I will show you how to use Cyberduck to connect to S3 bucket and download folders.
- Start Cyberduck. Connect to Amazon S3 with “New Connection". Select “Amazon S3" as the destination and “443" as the port.
- Next, set the IAM access key in “Access key ID" and the IAM secret key in “Password", and press “Connect".
Download the folder from S3
- After you connect to S3, you will see the bucket that exists in your AWS account. Select a bucket.
- Then you will see the folders in your bucket. You can download the folder by right-clicking on the folder.
Reference
- For reference, the following blog describes how to install aws cli and how to get the IAM access key ID and secret key.