Dask

Run TimeGPT in a distributed manner using Dask for scalable forecasting.

Run TimeGPT in a distributed manner using Dask.

Dask is an open-source parallel computing library for Python. This guide explains how to use TimeGPT from Nixtla with Dask for distributed forecasting tasks.

Before proceeding, make sure you have an API key from Nixtla.

Highlights

• Simplify distributed computing with Fugue. • Run TimeGPT at scale on a Dask cluster. • Seamlessly convert pandas DataFrames to Dask.

Outline

1. Installation
2. Load Your Data
3. Import Dask
4. Use TimeGPT on Dask

1. Step 1: Installation

   Install Fugue and Dask

   Fugue provides an easy-to-use interface for distributed computing over frameworks like Dask.

   You can install fugue with:

   Install Fugue and Dask

   pip install fugue[dask]
   

   If running on a distributed Dask cluster, ensure the nixtla library is installed on all worker nodes.

2. Step 2: Load Your Data

   You can start by loading data into a pandas DataFrame. In this example, we use hourly electricity prices from multiple markets:

   Load Electricity Data

   import pandas as pd
   
   df = pd.read_csv(
       'https://raw.githubusercontent.com/Nixtla/transfer-learning-time-series/main/datasets/electricity-short.csv',
       parse_dates=['ds'],
   )
   df.head()
   

   Example pandas DataFrame:

   	unique_id	ds	y
   0	BE	2016-10-22 00:00:00	70.00
   1	BE	2016-10-22 01:00:00	37.10
   2	BE	2016-10-22 02:00:00	37.10
   3	BE	2016-10-22 03:00:00	44.75
   4	BE	2016-10-22 04:00:00	37.10

3. Step 3: Import Dask

   Convert the pandas DataFrame into a Dask DataFrame for parallel processing.

   Convert to Dask DataFrame

   import dask.dataframe as dd
   
   dask_df = dd.from_pandas(df, npartitions=2)
   dask_df
   

   When converting to a Dask DataFrame, you can specify the number of partitions based on your data size or system resources.

4. Step 4: Use TimeGPT on Dask

   To use TimeGPT with Dask, provide a Dask DataFrame to Nixtla’s client methods instead of a pandas DataFrame.

   Important Concept: NixtlaClient

   Instantiate the NixtlaClient class to interact with Nixtla’s API.

   Initialize NixtlaClient

   from nixtla import NixtlaClient
   
   nixtla_client = NixtlaClient(
       api_key='my_api_key_provided_by_nixtla'
   )
   

   Using an Azure AI endpoint

   To use Azure AI, set the base_url parameter:

   Azure AI Endpoint Setup

   nixtla_client = NixtlaClient(
       base_url="your Azure AI endpoint",
       api_key="your api_key"
   )
   

   You can use any method from the NixtlaClient, such as forecast or cross_validation.

   Forecast Example

   Forecast with TimeGPT and Dask

   fcst_df = nixtla_client.forecast(dask_df, h=12)
   fcst_df.compute().head()
   

   	unique_id	ds	TimeGPT
   0	BE	2016-12-31 00:00:00	45.190453
   1	BE	2016-12-31 01:00:00	43.244446
   2	BE	2016-12-31 02:00:00	41.958389
   3	BE	2016-12-31 03:00:00	39.796486
   4	BE	2016-12-31 04:00:00	39.204533

   Cross-validation Example

   Cross-validation with TimeGPT and Dask

   cv_df = nixtla_client.cross_validation(
       dask_df,
       h=12,
       n_windows=5,
       step_size=2
   )
   cv_df.compute().head()
   

   	unique_id	ds	cutoff	TimeGPT
   0	BE	2016-12-30 04:00:00	2016-12-30 03:00:00	39.375439
   1	BE	2016-12-30 05:00:00	2016-12-30 03:00:00	40.039215
   2	BE	2016-12-30 06:00:00	2016-12-30 03:00:00	43.455849
   3	BE	2016-12-30 07:00:00	2016-12-30 03:00:00	47.716408
   4	BE	2016-12-30 08:00:00	2016-12-30 03:00:00	50.316650

   Azure AI Models

   When using an Azure AI endpoint, set model to "azureai":

   Azure AI Model Usage

   nixtla_client.forecast(..., model="azureai")
   

   For the public API, two models are available: • timegpt-1 (default) • timegpt-1-long-horizon

   See the Long Horizon Forecasting Tutorial for details on timegpt-1-long-horizon.

   TimeGPT with Dask also supports exogenous variables. Refer to the Exogenous Variables Tutorial for details. Substitute pandas DataFrames with Dask DataFrames as needed.