Load Data From S3 Buckets
If you use AWS S3 to store your data, connecting to Saturn Cloud takes just a couple of steps.
Before starting this, you should create a Jupyter server resource. See our quickstart if you don’t know how to do this yet.
To connect to public S3 buckets, you can simply connect using anonymous connections in Jupyter, the way you might with your local laptop. In this case, you can skip to the Connect to Data Via
If your S3 data storage is not public and requires AWS credentials, please read on!
Create AWS Credentials
Credentials for S3 access can be acquired inside your AWS account. Visit https://aws.amazon.com/ and sign in to your account.
In the top right corner, click the dropdown under your username and select My Security Credentials.
Under “My Security Credentials” you’ll see section titled “Access keys for CLI, SDK, & API access”. If you don’t yet have an access key listed, create one.
Save the key information that this generates, and keep it in a safe place!
Add AWS Credentials to Saturn Cloud
Sign in to your Saturn Cloud account and select Credentials from the menu on the left.
This is where you will add your S3 API key information. This is a secure storage location, and it will not be available to the public or other users without your consent.
At the top right corner of this page, you will find the New button. Click here, and you will be taken to the Credentials Creation form.
You will be adding three credentials items: your AWS Access Key, AWS Secret Access Key, and you default region. Complete the form one time for each item.
|AWS Access Key ID
|AWS Secret Access Key
|AWS Default Region
Copy the values from your AWS console into the Value section of the credential creation form. The credential names are recommendations; feel free to change them as needed for your workflow. You must, however, use the provided Variable Names for S3 to connect correctly.
With this complete, your S3 credentials will be accessible by Saturn Cloud resources! You will need to restart any Jupyter Server or Dask Clusters for the credentials to populate to those resources.
Connect to Data Via
Set Up the Connection
s3fs will automatically seek your AWS credentials from the environment. Since you have followed our instructions above for adding and saving credentials, this will work for you!
If you don’t have credentials and are accessing a public repository, set
anon=True in the
s3 = s3fs.S3FileSystem(anon=False)
At this point, you can reference the
s3 handle and look at the contents of your S3 bucket as if it were a local file system. For examples, you can visit the s3fs documentation where they show multiple ways to interact with files like this.
Load a Parquet file using pandas
This approach just uses routine pandas syntax.
import pandas as pd
file = "saturn-public-data/nyc-taxi/data/yellow_tripdata_2019-01.parquet
with s3.open(file, mode="rb") as f:
df = pd.read_parquet(f)
For small files, this approach will work fine. For large or multiple files, we recommend using Dask, as described next.
Load a Parquet file using Dask
This syntax is the same as pandas, but produces a distributed data object.
import dask.dataframe as dd
file = "saturn-public-data/nyc-taxi/data/yellow_tripdata_2019-01.parquet"
with s3.open(file, mode="rb") as f:
df = dd.read_parquet(f)
Load a folder of Parquet files using Dask
Dask can read and load a whole folder of files if they are formatted the same, using glob syntax.
files = s3.glob("s3://saturn-public-data/nyc-taxi/data/yellow_tripdata_2019-*.parquet")
taxi = dd.read_parquet(