How to Generate a Temporary URL to Upload Files to Amazon S3 with Boto Library

How to Generate a Temporary URL to Upload Files to Amazon S3 with Boto Library

Amazon Simple Storage Service (S3) is an object storage service that offers scalability, data availability, security, and performance. One common use-case is the direct upload of files to an S3 bucket without exposing your S3 credentials. In this article, we will walk through how to generate a temporary URL for this purpose using the Boto3 library in Python.

What is Boto Library?

Boto is the Amazon Web Services (AWS) SDK for Python. It allows Python developers to write software that makes use of services like Amazon S3, Amazon EC2, and others. Boto3, the latest version, allows for the creation of pre-signed URLs to upload (or download) files directly to/from S3 securely without requiring direct AWS credentials.

Steps to Generate a Temporary URL in Amazon S3 with Boto3

Step 1: Install Boto3

First, you need to install the Boto3 library. You can do this via pip:

pip install boto3

Step 2: Import Boto3 and Instantiate the S3 Client

Next, import the boto3 module and create a client for S3.

import boto3

s3_client = boto3.client('s3')

Step 3: Generate the Pre-signed URL

The generate_presigned_url function of the s3_client object can be used to create a pre-signed URL. This function takes two mandatory arguments: ClientMethod, which is the name of the client method to call, and Params, which is a dictionary of parameters to pass to the method.

def create_presigned_url(bucket_name, object_name, expiration=3600):
    """Generate a presigned URL to share an S3 object

    :param bucket_name: string
    :param object_name: string
    :param expiration: Time in seconds for the presigned URL to remain valid
    :return: Presigned URL as string. If error, returns None.
    """
    try:
        response = s3_client.generate_presigned_url('put_object',
                                                    Params={'Bucket': bucket_name,
                                                            'Key': object_name},
                                                    ExpiresIn=expiration)
    except ClientError as e:
        logging.error(e)
        return None

    return response

In this function, bucket_name is the name of the S3 bucket, object_name is the name you want the uploaded file to have, and expiration is the time, in seconds, for the presigned URL to remain valid. The function returns the pre-signed URL or None if an error occurs.

Conclusion

The Boto3 library makes it easy for Python developers and data scientists to interact with Amazon S3 and other AWS services. Generating temporary URLs for secure, direct file uploads to S3 buckets is a simple process with the generate_presigned_url function. Although we focused on file uploads in this article, the same function can also be used to generate temporary URLs for downloading files. Now you can manage your data in S3 more securely and efficiently!

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Join today and get 150 hours of free compute per month.