s3cmd Tutorial


IMPORTANT NOTE - Please read before you proceed

  1. I wrote this tutorial for a course, Parallel Programming, which I TA’ed at Johns Hopkins. There is some text in here which is related to a few assignments in the course. If you are not a student in this course, please ignore these points.
  2. If you find the commands below confusing, you can access a GUI to s3 from within AWS Console in the browser. There are other options: Firefox plugin, Chrome plugin, Windows Application, MAC-OS Application.
  3. You cannot have an underscore '_' or uppercase letters in a bucket name. For more details, please refer to the AWS Bucket Restrictions and Limitations.
  4. For power users, who intend to upload/download terabytes of data from AWS S3(as might be the case with some people using NeuroData services), this might not be the ideal application. The AWS team wrote an awesome tool called AWS Command Line Interface which enables you to transfer data using multiple threads. They have a pretty extensive documentation for this here. There is a subtle difference between aws s3 and aws s3api, former deals with the bucket as a whole, the latter deals with individual s3 objects.

Installing s3cmd

Ubuntu, Debian-based Linux
sudo apt-get update
sudo apt-get install s3cmd
CentO, RHEL, Fedora-based LINUX
sudo yum install s3cmd

Configuring s3cmd

s3cmd --configure


Fetch data from a bucket

s3cmd -r get s3://<bucket-name>/ .
s3cmd -r get s3://friends.simple/ .

Create a bucket

s3cmd mb s3://<bucket-name>/
s3cmd mb s3://PP-2015-Tut/

Put Data into a bucket

s3cmd put -r <directory-name> s3://<bucket-name>/
s3cmd put -r test s3://PP-2015-Tut/
s3cmd put -r <directory-name>/ s3://<bucket-name>/
s3cmd put -r test/ s3://PP-2015-Tut/

View Contents in a bucket

s3cmd ls -r s3://<bucket_name>/
s3cmd ls -r s3://PP-2015-Tut/

Set the ACL for a bucket

s3cmd setacl -r s3://<bucket-name> --acl-public
s3cmd setacl -r s3://PP-2015-Tut --acl-public

Check the ACL for a bucket

s3cmd info s3://<bucket-name>/
s3cmd info s3://PP-2015-Tut/
Location: any
ACL:  <your-user-name> : READ
ACL:  <your-user-name> : WRITE
ACL:  *anon* : READ </code>

Delete a file from the bucket

s3cmd del -r --force s3://<bucket_name>/
s3cmd del -r --force s3://PP-2015-Tut/

Remove a bucket

s3cmd rb s3://<bucket_name>
s3cmd rb s3://PP-2015-Tut/

There is much more you can do, just type the magic words to learn more.

man s3cmd
Kunal Lillaney

Kunal Lillaney

rss facebook twitter github youtube mail spotify lastfm instagram linkedin google google-plus pinterest medium vimeo stackoverflow reddit quora quora