Azure Databricks has good integration options with Azure Data Factory. From calling notebooks, to libraries. But when it comes to getting a value, or a table from Databricks, I found myself going back to spark dataset based lookups which has it’s own limitations on the queries we can write to…

Databricks provides a great environment for ad-hoc and interactive usage of data. However, setting up a local environment for one’s testing can be quite a task. The below post is intended to help setup a working local environment for development and testing of python libraries for databricks remotely using databricks-connect.

Rachit Daga

Tech Enthusiast, Software Engineer @Google, Ex-Microsoft

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store