Is it Possible to Retrieve Multiple Objects from Amazon S3 in a Single Request?

Is it Possible to Retrieve Multiple Objects from Amazon S3 in a Single Request?

When dealing with large-scale data storage and retrieval, efficiency is key. One question that often comes up for data scientists and software engineers alike is: “Is it possible to retrieve multiple objects from Amazon S3 in a single request?” The answer is not straightforward, but the short answer is: not directly, but there are workarounds.

AWS SDK and Parallel Requests

Amazon S3 is designed in a way that each object in the bucket is retrieved with a unique HTTP request. While it doesn’t allow fetching multiple objects in a single request, you can achieve concurrent downloads by using parallel requests.

The AWS SDK for Python (Boto3) allows you to make parallelized requests to S3. This means that while you can’t technically get multiple objects with a single request, you can send out multiple requests at once, which can significantly increase the speed of your data retrieval.

import boto3
from concurrent.futures import ThreadPoolExecutor

s3 = boto3.client('s3')
bucket_name = 'my_bucket'
keys = ['key1', 'key2', 'key3']

def download(key):
    s3.download_file(bucket_name, key, f'local/path/{key}')

with ThreadPoolExecutor(max_workers=10) as executor:
    executor.map(download, keys)

This code initiates 10 concurrent downloads from the S3 bucket, thus improving the speed of data retrieval.

Batch Operations

For larger operations, AWS introduced S3 Batch Operations. It is a high-throughput, bulk data-processing feature that automates repetitive tasks such as copying objects or replacing tag sets. However, it’s important to note that Batch Operations are asynchronous, and they don’t provide real-time data retrieval.

S3 Select

Another efficient way to retrieve data is using S3 Select, a feature that retrieves a subset of data from an object by using simple SQL expressions. This way, you can retrieve only the needed data without needing to download the entire object.

import boto3

s3 = boto3.client('s3')
bucket_name = 'my_bucket'
key = 'my_key'
expression = "SELECT * FROM s3object s WHERE s.age > 20"

response = s3.select_object_content(
    Bucket=bucket_name,
    Key=key,
    ExpressionType='SQL',
    Expression=expression,
    InputSerialization={'CSV': {"FileHeaderInfo": "Use"}},
    OutputSerialization={'CSV': {}},
)

for event in response['Payload']:
    if 'Records' in event:
        records = event['Records']['Payload'].decode('utf-8')
        print(records)

Though S3 Select doesn’t allow fetching multiple objects in a single request, it provides a way to avoid unnecessary data transfers by fetching only the required data.

Conclusion

While Amazon S3 doesn’t allow the retrieval of multiple objects in one request, there are several workarounds to increase the efficiency of data retrieval: parallel requests using the AWS SDK, Batch Operations for bulk actions, and S3 Select for selective data retrieval. The best method largely depends on your specific use-case and the nature of your data.

Remember, when working with data at scale, it’s critical to optimize for efficiency, and these strategies can help you improve your data retrieval process from S3.

If you found this article helpful, please share it with your colleagues and friends in the data science and software engineering community. Stay tuned for more “How to” and “What Is” articles demystifying complex technical topics in the world of data science and software engineering.

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Join today and get 150 hours of free compute per month.