4月 7, 2021AWS_en

Download the folder from S3

 

• When using the console, users cannot download files in buckets as folders. This time, I will show you how to download folders using CLI or tools instead of the console.
• There are many tools to access S3, but this time I will explain using official tools and popular tools.

Download using CLI

Download the exported Logs folder from S3

• You can use aws cli to download folders from your S3 bucket.
• For example, if you have ELB or CloudFront access logging enabled, your S3 bucket will contain many exported folders and files. The following command is an example for downloading a folder from an S3 bucket.

aws s3 cp s3://<BUCKET_NAME>/<AAA>/<BBB>/<Folder> ./<Copyed Folder> --recursive

$ aws s3 cp s3://<BUCKET_NAME>/<AAA>/<BBB>/exportedlogs ./apigw_logs --recursive --profile <profile_name>

 

Extract the compressed file

• Files exported from CloudWatch Logs are compressed as gzip. So you can combine find and xargs in a Linux environment to gunzip them all at once.

find . -name '000000.gz' | xargs -n1 gunzip

$ find . -name '000000.gz' | xargs -n1 gunzip

 

Download using tool

Use Cyberduck to connect to Amazon S3.

• I’m using Cyberduck for my scp / sftp client. Next, I will show you how to use Cyberduck to connect to S3 bucket and download folders.
• Start Cyberduck. Connect to Amazon S3 with “New Connection". Select “Amazon S3" as the destination and “443" as the port.
• Next, set the IAM access key in “Access key ID" and the IAM secret key in “Password", and press “Connect".

Download the folder from S3

• After you connect to S3, you will see the bucket that exists in your AWS account. Select a bucket.

• Then you will see the folders in your bucket. You can download the folder by right-clicking on the folder.

Reference

• For reference, the following blog describes how to install aws cli and how to get the IAM access key ID and secret key.