Amazon S3 boto: How to Create a Folder
Amazon S3 boto: How to Create a Folder
Amazon Simple Storage Service (S3) is a flexible, scalable, and highly durable storage solution offered by Amazon Web Services (AWS). But when you’re interacting with this service programmatically, how do you create a folder? The answer lies within the boto3 Python library.
What is boto3?
boto3 is the AWS SDK for Python. It allows Python developers to write software that makes use of services like Amazon S3, Amazon EC2, and others. In this post, we’ll focus on using boto3 to create a folder (or as it’s known in S3 terms, a ‘prefix’) in an S3 bucket.
Setting Up
Before we jump into the code, ensure you have the necessary tools installed and configured:
- Install boto3:
pip install boto3 - Configure your AWS credentials (either through the AWS CLI or by setting the following environment variables:
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY)
Creating a Folder in S3
To create a folder, we’ll be uploading an empty file with a key ending in ‘/’. Remember, Amazon S3 is a flat structure, not a hierarchical file system, which means it doesn’t natively support folders. However, it uses object key names to create the illusion of a folder structure.
Here’s how to create a folder:
import boto3
def create_folder(bucket_name, folder_name):
s3_client = boto3.client('s3')
folder = {
'Bucket': bucket_name,
'Key': f"{folder_name}/"
}
s3_client.put_object(**folder)
print(f"Folder '{folder_name}' created in '{bucket_name}'.")
# replace 'my-bucket' and 'my-folder' with your bucket and desired folder name
create_folder('my-bucket', 'my-folder')
In this script, we’re importing the boto3 library, creating a S3 client, and defining an object (our folder) with the bucket’s name and the folder’s name. The function put_object is used to upload the empty file to S3.
Conclusion
Creating a folder in Amazon S3 using boto3 in Python is a straightforward process once you understand that folders in S3 are essentially a naming convention. Remember to replace ‘my-bucket’ and ‘my-folder’ with your actual bucket and desired folder names.
It’s important to note that while this ‘folder’ structure can be helpful for organizing your S3 objects, each object in Amazon S3 still has a unique key, regardless of the ‘folder’ in which it is placed.
With this guide, you should now be able to easily create folders in Amazon S3 using Python and the boto3 library.
If you found this article helpful, please share it with your network. For more technical guides and explanations, follow our blog. We’re always here to help you navigate your data science and software engineering journey.
If there are other topics you’d like us to cover, please leave a comment below. We’re always looking for new ideas. Happy coding!
References
About Saturn Cloud
Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Join today and get 150 hours of free compute per month.
