Local data

Downloading from QuantConnect

Introduction

Running the LEAN engine locally with the CLI requires you to have your own local data. The recommended way to source local data is by purchasing and downloading it from QuantConnect Datasets. This way you can use the same data that is available to your algorithm when you run it in the cloud. You can download this data manually using the CLI or on-demand using LEAN. The available data and pricing information can be found in the user guides.

Using the CLI

Follow these steps to purchase and download data from QuantConnect Datasets using the CLI:

  1. Log in to the CLI if you haven't done so already.
  2. Open a terminal in the directory you ran lean init in.
  3. Run lean data download to start an interactive downloading wizard.
    $ lean data download
    Select the organization to purchase and download data with:
    1) Organization 1
    2) Organization 2
    3) Organization 3
    Enter an option:
  4. Enter the number of the organization you want to be charged for the data you want to buy. If you are only a member of a single organization this step is skipped automatically.
    $ lean data download
    Select the organization to purchase and download data with:
    1) Organization 1
    2) Organization 2
    3) Organization 3
    Enter an option: 1
  5. Go through the interactive wizard to configure the data you want to download.
  6. After configuring the data you want to download, enter "N" when asked whether you want to download more data.
    $ lean data download
    Selected data:
    ...
    Total price: 10 QCC
    Organization balance: 20,096,264 QCC
    Do you want to download more data? [y/N]: N
  7. In the browser window that opens, read the CLI API Access and Data Agreement and click "I Accept The Data Agreement".
  8. Go back to the terminal and confirm the purchase to start downloading.
    $ lean data download
    You will be charged 10 QCC from your organization's QCC balance
    After downloading all files your organization will have 20,096,254 QCC left
    Continue? [y/N]: y
    [1/1] Downloading equity/usa/daily/spy.zip (10 QCC)

Using LEAN

An alternative to manually downloading all required data before running the backtest is to use the ApiDataProvider in LEAN. This data provider automatically downloads the required data files when your backtest requests them. After it downloads a data file, it stores it in your local data directory so that in future runs it'll not have to download it again.

Follow these steps to use the ApiDataProvider to automatically download data when needed:

  1. Log in to the CLI if you haven't done so already.
  2. Open a terminal in the directory you ran lean init in.
  3. Run lean backtest "My Project" --download-data to run a local backtest for the project in ./My Project and update the Lean configuration to use the ApiDataProvider.
    $ lean backtest "My Project" --download-data
    20210322 17:27:46.658 TRACE:: Engine.Main(): LEAN ALGORITHMIC TRADING ENGINE v2.5.0.0 Mode: DEBUG (64bit)
    20210322 17:27:46.664 TRACE:: Engine.Main(): Started 5:27 PM
    Successfully ran 'My Project' in the 'backtesting' environment and stored the output in 'My Project/backtests/2021-03-22_18-51-28'
    Setting the --download-data flag updates your Lean configuration. This means that you only need to use the flag once, all future backtests will automatically use the ApiDataProvider.

Follow these steps to revert the Lean configuration changes so that it uses only local data again:

  1. Open a terminal in the directory you ran lean init in.
  2. Run lean backtest "My Project" --data-provider Local to run a local backtest for the project in ./My Project and update the Lean configuration to only use local data.
    $ lean backtest "My Project" --data-provider Local
    20210322 17:27:46.658 TRACE:: Engine.Main(): LEAN ALGORITHMIC TRADING ENGINE v2.5.0.0 Mode: DEBUG (64bit)
    20210322 17:27:46.664 TRACE:: Engine.Main(): Started 5:27 PM
    Successfully ran 'My Project' in the 'backtesting' environment and stored the output in 'My Project/backtests/2021-03-22_18-51-28'
    The --data-provider option updates your Lean configuration. This means that you only need to use the option once, all future backtests will automatically use the newly configured data provider.

By default the ApiDataProvider does not have a spending limit and will keep downloading files until your QuantConnect organization runs out of QCC. You can use the --data-purchase-limit <value> option to set the QCC spending limit for the backtest.

All the options documented above are also available on the lean research command.

You can also see our Videos. You can also get in touch with us via Chat.

Did you find this page helpful?

Contribute to the documentation: