Upload a file to S3 with Boto

Amazon S3 is a distributed storage service which I’ve recently been working with. Boto is Python library for working with Amazon Web Services, which S3 is one facet of. This post will demonstrate how to upload a file using boto (a future post will demonstrate who to create the parameters for POST multi-part request that another client can use to upload to S3 without knowing your AWS key id or secret access key).

I will assume you’ve signed up for S3 already, and successfully downloaded and installed boto.

[cce lang="Python"]
import boto

# Fill these in – you get them when you sign up for S3
AWS_ACCESS_KEY_ID = ”
AWS_SECRET_ACCESS_KEY = ”

bucket_name = AWS_ACCESS_KEY_ID.lower() + ‘-mah-bucket’
conn = boto.connect_s3(AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY)
[/cce]

If you’ve previously learnt about S3 you’ll know that bucket names need to be unique, so that’s why we’ve used the AWS Key ID as a prefix (as this is your unique id it’s unlikely someone else will be using it as a prefix). We’ve also converted the bucket to lowercase, as DNS is case-insensitive and it’s nice to use vanity domains of the form http://[bucket_name].s3.amazonaws.com/.

We create the connection object with boto.connect_s3() and this object let’s us interact with S3. We now create ourselves a bucket and upload a file:

[cce lang="Python"]
bucket = conn.create_bucket(bucket_name,
location=s3.connection.Location.DEFAULT)

testfile = “replace this with an actual filename”
print ‘Uploading %s to Amazon S3 bucket %s’ % \
(testfile, bucket_name)

import sys
def percent_cb(complete, total):
sys.stdout.write(‘.’)
sys.stdout.flush()

from boto.s3.key import Key
k = Key(bucket)
k.key = ‘my test file’
k.set_contents_from_filename(testfile,
cb=percent_cb, num_cb=10)
[/cce]

There you have it! The set_contents_from_filename is a particularly nifty method which simplifies all the streaming of data to S3. Amazon’s prototype python S3 library required that the file be loaded into memory. This doesn’t work too well if you are working with large media files.

Oh, and the percent_cb function is a call back that gets called as the upload progresses.