Dask has a built-in dashboard that shines a light on how the tool is utilizing threads for parallel computation. It gets updated real-time as executions take place, so you can monitor your tasks as they happen. This article covers how can you access Dask Dashboard and then goes through each graph of status page, discussing how to understand and use it.
Accessing the Dask Dashboard
Local Dask Clusters
If you are running Dask locally on a Jupyter server, rather than on a full Saturn Cloud Dask cluster, you can directly preview it. Once you have created a local
cluster object on a Jupyter server by leaving the arguments to the Dask Client blank, preview the
client object. This will display a link to the Dask Dashboard along with other information on Dask Client. Click on this link and you will be navigated to Dask Dashboard.
from dask.distributed import Client client = Client()
Dask Clusters in Saturn Cloud
For a view of the dashboard you can see while working in JupyterLab, click on the Dask icon on the left-hand side of the JupyterLab screen.
For a detailed view in a separate window, you can open the dashboard directly from the Saturn Cloud UI by clicking the Dashboard link on the Dask card of the resource:
Details of the Dask Dashboard Components
Task Stream : The task stream lets you view tasks across threads. Each row represents a thread and each rectangle represents a task. The colors in these rectangles is for the kind of operation being performed, for example green may stand for ‘sum’ and purple may stand for ‘fitting a model’. The white space between these operations is thread’s idle time. A red rectangle represents communication between threads.
Notice in image above, one of the tasks is colored as green but also has red around it . This means that transferring of data and computation of a task are being preformed in parallel. Lots of red and white in task stream indicates that hardware is not being used effectively since the threads are idling while waiting for data to transfer. In that case you may want to make some changes to your computations or resources.
Progress Bar: The progress bar part of the dashboard indicates how a set of tasks is progressing. Each task is one horizontal bar on this part. Notice that there may be multiple rows of bars. The color coding for each type of operation aligns with the same colors from the task stream . Each progress bar can have 3 components:
- The leftmost part is transparent colored. Transparent colors indicate that completed tasks have been released from memory.
- In middle, the solid coloured parts indicate that these tasks are in-progress hence are in memory.
- The dark gray you see on the right side of bars tells us that these tasks are ready to run.
If you see lot of dark gray in progress bars that may mean that so many tasks are waiting to be run because there are not many threads available to run them. This may be solved by making modifications to your Dask cluster configuration.
Bytes Stored: The bytes stored part of the dashboard tells us the amount of RAM taken by each worker . In this example below, each line corresponds to a worker. We can see that each worker is taking around 6 GB of ram. The total memory consumed by all workers (in this case 18.26 GB) is displayed at top of the chart.
Task processing: This graph represents processing of tasks which were in ‘ready to run’ state (these are the tasks represented by dark gray in the progress bar section). This graph tells how tasks are being processed across each worker. In image on left hand side, you will notice that each worker is running around 400 tasks. This indicates a healthy and equal distribution of tasks amongst workers.
In image on right hand side you will notice that task distribution is unequal. This happens when all workers are not been optimized fully or computations are about to end hence each workers have finished their tasks or are almost finishing.
|All workers in action||Most workers in idle state|
Need help, or have more questions? Contact us at:
- On Intercom, using the icon at the bottom right corner of the screen
We'll be happy to help you and answer your questions!