Dask logo

SCALE PYTHON EFFORTLESSLY

Dask is a free, flexible library for parallel computing in Python. It allows you work on arbitrarily large datasets and dramatically increases the speed of your computations. It was developed with community projects such as NumPy, pandas, scikit-learn, and Jupyter.

WHY USE DASK?

Familiar Interface

Python has a rich ecosystem of data science libraries include numpy for arrays, pandas for DataFrames, xarray for nd-data, and scikit-learn for machine learning. Dask mirrors the api of these libraries to make it easy to switch to a distributed alternative.

Flexible

Sometimes your data doesn’t fit neatly in a dataframe or an array. Or maybe you have already written a whole piece of your pipeline and you just want to make it faster. That’s what dask.delayed is for. Wrap any function with the dask.delayed decorator and it will run in parallel.

Python-Native

Dask is written in Python for easy integration with Python code, easy troubleshooting, and access to the full PyData stack.

Fast

Write your code to run on a laptop and easily scale it up to run on clusters with 1000s of cores. Dask simplifies the big data workflow and its excellent single-machine performance speeds up the prototyping stage, leading to faster model deployment.

Dask is composed of two parts:

Dynamic Task Scheduling

Dynamic Task Scheduling Dask can easily take a set of tasks which normally would have run sequentially and distribute them to a collection of workers. This is similar to Airflow, Luigi, Celery, or Make, but optimized for interactive computational workloads

"Big Data" Collections

Dask has parallel arrays, data frames, and lists that extend common interfaces like NumPy, pandas, or Python iterators to larger-than-memory or distributed environments.

getting started

Get started with Dask in minutes. Below, you can find resources on learning about and using Dask.

View the Dask docs

Learn more about Dask via their full documentation here

Train a Model with XGBoost and Dask

Watch guides and examples from Dask's YouTube channel

FREE DASK CHEATSHEET

This cheatsheet will guide you through the basic utilities of Dask

Learn with dask tutorials

View Dask tutorials on Github, along with code samples for common Dask user tasks

Use dask on GPUS

Dask integrates with RAPIDS and XGBoost for GPU-accelerated data analytics and machine learning. Accelerate your model runtime by 2000x with RAPIDS on Saturn Cloud

Use RAPIDS on a GPU cluster

Learn how to scale to larger data sizes with multiple GPUs. This exercise uses RAPIDS to speed up a machine learning workload on data that would be too large for a single machine.

Train a Model with XGBoost and Dask

XGBoost is a popular algorithm for supervised learning with tabular data. Learn how to use the XGBoost ML library to train a model on a distributed Dask cluster.