Python to parse fields in Amazon S3 logs

Joel — Tue, 12 Jan 2010 22:23:19 +0000

The log format for Amazon S3 is slightly annoying. Not overwhelmingly so, but the date field has the field separator (a space) in the middle of it and it isn’t encapsulated by quote characters. Here’s some code to split the fields up, assuming you’ve downloaded the log file already (it’s easy enough to list all logs and retrieve them with boto):

import csv
r = csv.reader(open('logfilename'),
delimiter=' ',quotechar='"')
log_entries = []
for i in r:
i[2] = i[2] + " " + i[3] # repair date field
del i[3]
log_entries.append(i)

ferrouswheel » s3

Python to parse fields in Amazon S3 logs