Skip to main content

Delta Live Tables CLI (legacy)

important

This documentation has been retired and might not be updated.

This information applies to legacy Databricks CLI versions 0.18 and below. Databricks recommends that you use newer Databricks CLI version 0.205 or above instead. See What is the Databricks CLI?. To find your version of the Databricks CLI, run databricks -v.

To migrate from Databricks CLI version 0.18 or below to Databricks CLI version 0.205 or above, see Databricks CLI migration.

You run Databricks Delta Live Tables CLI subcommands by appending them to databricks pipelines. These subcommands call the Delta Live Tables API.

Bash
databricks pipelines -h
Usage: databricks pipelines [OPTIONS] COMMAND [ARGS]...

  Utility to interact with Databricks Delta Live Tables Pipelines.

Options:
  -v, --version   0.18.0
  --debug         Debug Mode. Shows full stack trace on error.
  --profile TEXT  CLI connection profile to use. The default profile is
                  "DEFAULT".

  -h, --help      Show this message and exit.

Commands:
  create  Creates a pipeline.
  delete  Deletes the pipeline and cancels any active updates.
  deploy  [Deprecated] This command is deprecated, use create and edit
          commands instead.  Creates or edits a pipeline specified by the
          pipeline settings.

  edit    Edits a pipeline.
  get     Gets a pipeline's current settings and status.
  list    Lists all pipelines and their statuses.
  reset   [Deprecated] Use the "start --full-refresh" command instead. Resets
          a pipeline so that data can be reprocessed from the beginning.

  run     [Deprecated] Use the "start" command instead. Starts a pipeline
          update.

  start   Starts a pipeline update.
  stop    Stops the pipeline by cancelling any active update.

Create a pipeline​

To display usage documentation, run databricks pipelines create --help.

Usage​

Bash
databricks pipelines create --settings create-pipeline.json

create-pipeline.json:

JSON
{
  "name": "Wikipedia pipeline (SQL)",
  "storage": "/Users/username/data",
  "clusters": [
    {
      "label": "default",
      "autoscale": {
        "min_workers": 1,
        "max_workers": 5
      }
    }
  ],
  "libraries": [
    {
      "notebook": {
        "path": "/Users/username/Delta Live Tables quickstart (SQL)"
      }
    }
  ],
  "target": "wikipedia_analysis",
  "continuous": false
}
Console
Successfully created pipeline: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003 with ID: 4ee6b5c2-7017-11ec-90d6-0242ac120003.

Edit a pipeline​

To display usage documentation, run databricks pipelines edit --help.

Usage​

Bash
databricks pipelines edit --settings edit-pipeline.json

edit-pipeline.json:

JSON
{
  "id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
  "name": "Wikipedia pipeline (SQL)",
  "storage": "/Users/username/data",
  "clusters": [
    {
      "label": "default",
      "autoscale": {
        "min_workers": 1,
        "max_workers": 3
      }
    }
  ],
  "libraries": [
    {
      "notebook": {
        "path": "/Users/username/Delta Live Tables quickstart (SQL)"
      }
    }
  ],
  "target": "wikipedia_quickstart_data",
  "continuous": false
}
Console
Successfully edited pipeline settings: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003.

Delete a pipeline​

To display usage documentation, run databricks pipelines delete --help.

Usage​

Bash
databricks pipelines delete --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Console
Pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003 deleted

Get information about a pipeline​

To display usage documentation, run databricks pipelines get --help.

Usage​

Bash
databricks pipelines get --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
JSON
{
  "pipeline_id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
  "spec": {
    "id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
    "name": "Wikipedia pipeline (SQL)",
    "storage": "/Users/username/data",
    "clusters": [
      {
        "label": "default",
        "autoscale": {
          "min_workers": 1,
          "max_workers": 5
        }
      }
    ],
    "libraries": [
      {
        "notebook": {
          "path": "/Users/username/Delta Live Tables quickstart (SQL)"
        }
      }
    ],
    "target": "wikipedia_analysis",
    "continuous": false
  },
  "state": "IDLE",
  "name": "Wikipedia pipeline (SQL)",
  "creator_user_name": "username@databricks.com",
  "last_modified": 1641600056265
}

List all pipelines and information on their status​

To display usage documentation, run databricks pipelines list --help.

Usage​

Bash
databricks pipelines list
JSON
[
  {
    "pipeline_id": "8a93e1b4-7024-11ec-90d6-0242ac120003",
    "state": "IDLE",
    "name": "Retail sales pipeline (SQL)",
    "latest_updates": [
      {
        "update_id": "9df03924-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T16:18:57.982Z"
      },
      {
        "update_id": "a57e601c-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-28T18:19:30.371Z"
      }
    ],
    "creator_user_name": "user@databricks.com"
  },
  {
    "pipeline_id": "b46e2670-7024-11ec-90d6-0242ac120003",
    "state": "IDLE",
    "name": "DLT quickstart (Python)",
    "latest_updates": [
      {
        "update_id": "bcd8fa2e-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-12-16T18:19:25.827Z"
      },
      {
        "update_id": "c2c7a2c8-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T22:22:32.586Z"
      },
      {
        "update_id": "c88d2818-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T22:16:18.258Z"
      }
    ],
    "creator_user_name": "user@databricks.com"
  }
]

Start a pipeline update​

To display usage documentation, run databricks pipelines start --help.

Bash
databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Console
Started an update c6462e2e-beb7-4c24-a9d3-962eaa19781b for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.

Start a pipeline update with full refresh​

To display usage documentation, run databricks pipelines start --help.

Usage​

Bash
databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003 --full-refresh
Console
Started an update ed885d3d-4fc0-44fd-b5dd-59471c789327 with full refresh for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.

Stop a pipeline update​

To display usage documentation, run databricks pipelines stop --help.

Usage​

Bash
databricks pipelines stop --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Console
Stopped pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.
Last updated on