Skip to main content

Install Databricks Connect for Scala

note

This article covers Databricks Connect for Databricks Runtime 13.3 LTS and above.

This article describes how to install Databricks Connect for Scala. See What is Databricks Connect?. For the Python version of this article, see Install Databricks Connect for Python.

Requirements​

  • Your target Databricks workspace and cluster must meet the requirements for Compute configuration for Databricks Connect.

  • You have the Java Development Kit (JDK) installed on your development machine. Databricks recommends that the version of your JDK installation matches the JDK version on your Databricks cluster. To find the JDK version of the Databricks Runtime on your cluster, refer to the System environment section of the Databricks Runtime release notes or the version compatibility table.

    note

    Choosing a JDK install that is below or above the JDK version on your cluster might produce unexpected results, or your code might not run at all.

  • You have Scala installed on your development machine. Databricks recommends that the version of your Scala installation matches the Scala version on your Databricks cluster. To find the Scala version of the Databricks Runtime version of your cluster, refer to the System environment section of the Databricks Runtime release notes or the version compatibility table.

  • If you are using user-defined functions (UDFs), the local Scala and Java versions must match the Scala and Java versions of the Databricks Runtime version of the cluster. To find the Scala and Java versions of the Databricks Runtime version of your cluster, refer to the System environment section of the Databricks Runtime release notes or the version compatibility table.

  • A Scala build tool on your development machine, such as sbt.

Version support matrix​

The following table shows compatible Databricks Connect, JDK, and Scala versions. Databricks Connect version numbers correspond to Databricks Runtime version numbers.

Compute type

Databricks Connect version

JDK version

Scala version

Cluster

16.0 and above

16.0 ML and above

JDK 17

2.12.18

Cluster

15.4 LTS

15.4 ML LTS

JDK 8

2.12.18

Cluster

13.3 LTS to 14.3 LTS

13.3 ML LTS to 14.3 ML LTS

JDK 8

2.12.15

Add a reference to the Databricks Connect client​

To set up the Databricks Connect client, first add a reference to the client. In your Scala project’s build file such as build.sbt for sbt, pom.xml for Maven, or build.gradle for Gradle, add the following reference to the Databricks Connect client. Replace 14.0.0 with the version of the Databricks Connect library that matches the Databricks Runtime version on your cluster. You can find the Databricks Connect library version numbers in the Maven central repository.

libraryDependencies += "com.databricks" % "databricks-connect" % "14.0.0"

Configure connection properties​

Next, configure properties to establish a connection between Databricks Connect and your remote Databricks cluster. These properties include settings to authenticate Databricks Connect with your cluster. See Compute configuration for Databricks Connect.

For Databricks Connect for Databricks Runtime 13.3 LTS and above, for Scala, Databricks Connect includes the Databricks SDK for Java. This SDK implements the Databricks client unified authentication standard, a consolidated and consistent architectural and programmatic approach to authentication. This approach makes setting up and automating authentication with Databricks more centralized and predictable. It enables you to configure Databricks authentication once and then use that configuration across multiple Databricks tools and SDKs without further authentication configuration changes.

note

For OAuth user-to-machine (U2M) authentication, you must use the Databricks CLI to authenticate before you run your Scala code. See Tutorial.

Last updated on