In the sections below, we provide example notebooks to demonstrate how to install Caffe on Databricks using Cluster Node Initialization Scripts and how to run example Caffe programs. See Integrating Deep Learning Libraries with Apache Spark for an example of integrating a deep learning library with Spark.
This guide is not a comprehensive guide on Caffe. Please also refer to the Caffe website.
Databricks recommends using Cluster Node Initialization Scripts to install Caffe to make it available on all cluster nodes. The example notebook below installs an init script that builds and installs Caffe from source with GPU support.
To test and migrate single-machine Caffe workflows, you can start with a driver-only cluster on Databricks by setting the number of workers to zero. Though Apache Spark is not functional under this setting, it is a cost-effective way to run single-machine Caffe workflows.