Lean CLI

Object Store

Introduction

The Object Store is an organization-specific key-value storage location to save and retrieve data. Similar to a dictionary or hash table, a key-value store is a storage system that saves and retrieves objects by using keys. A key is a unique string that is associated with a single record in the key-value store and a value is an object being stored. Some common use cases of the Object Store include the following:

  • Transporting data between the backtesting environment and the research environment.
  • Training machine learning models in the research environment before deploying them to live trading.

The Object Store is shared across the entire organization. Using the same key, you can access data across all projects in an organization.

Local Storage

To add files to your Object Store, add them to the storage directory of your organization workspace or call the Save methods in your algorithm.

To view the contents of the Object Store, run lean object-store ls.

$ lean object-store ls

This command opens File Explorer to show the storage directory. You can delete and rename files in the Object Store directly from the File Explorer. To edit files, open them in a text editor.

Cloud Storage

The CLI enables you to upload files to the Object Store, view a summary of your stored files, and delete files.

Upload Files

To upload files to the Object Store, run lean cloud object-store set <storage-key> <local-file-path>.

$ lean cloud object-store 15737956/signals D:\qc\magic-signals.txt
Setting object 15737956/signals in organization d6d62db48592c72e67b534553413b691

If the file fails to upload, you may have insufficient storage space. If you need more, edit your storage plan.

List Directory Contents

To view all of the files and folders in the root directory of the Object Store, run lean cloud object-store ls.

$ lean cloud object-store ls
Key              Bytes  Folder  Filename
/15710069        None   True    15710069
/15727540        None   True    15727540
/15727540-1      None   True    15727540-1
/15730221        89649  False   15730221
/15730422        89748  False   15730422

To view all of the files and folders that are inside of one of the directories, run lean cloud object-store ls <folder-name>.

$ lean cloud object-store ls 15710069
Key                Bytes  Folder  Filename
15710069/adjusted  60024  False   adjusted
15710069/raw       60143  False   raw

Get File Metadata

To view the metadata of a file in the Object Store, run lean cloud object-store properties <path/to/file>.

$ lean cloud object-store properties 15710069/adjusted
Bytes  Modified             Filename           Preview
60024  2023-08-30 23:08:23  15710069/adjusted  {"12723264

Delete Content

To delete a file or directory in the Object Storage, run lean cloud object-store delete <key>.

$ lean cloud object-store delete 15710069/adjusted

Download Files

Permissioned Institutional clients can build derivative data such as machine learning models and download it from the Object Store. Contact us to unlock this feature for your account.

To download a file or directory from the Object Store, run lean cloud object-store get <key>.

$ lean cloud object-store get 15710069/adjusted
Fetching object store download url
Unzipping object store keys values into: <Current Directory>

Bulk File Upload

To upload all files from a directory to the Object Store, run the following Python script:

from os import listdir, path
from subprocess import run
SRC = ""   # Source directory
DST = ""   # Destination directory (empty is root)
if __name__ == '__main__':

    command = [ "lean", "cloud", "object-store", "set" ]

    for key in listdir(SRC):
        fullname = path.join(SRC, key)
        if not path.isfile(fullname):
            continue
        key = DST + '/' + key
        args = [key, fullname]
        print(' '.join(command + args))
        run(command + args)

Live Trading Considerations

When you deploy a live algorithm, you can access the data within minutes of modifying the Object Store. Ensure your algorithm is able to handle a changing dataset.

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: