Testing https://pypi.org/project/MangledDlt/

Background

Not being able to work in my local environment, ie. pycharm and using data from databricks has annoyed me for a long time. I have not been a fan of databricks connect, and the MangledDLT project came to my rescue. To get started, the easiest way is to use Databricks Free Edition, and then generate a PAT (Personal Access Token for yourself).

To get started, you can either provide a config class, or use the environment variables supported by the library. I prefer to use the ENV option personally, I set the required ENV variables in a custom configuration in PyCharm.

Library documentation

See https://pypi.org/project/MangledDlt/

ENV approach

Export the ENVs the way you prefer.

export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
export DATABRICKS_TOKEN="dapi..."
export DATABRICKS_WAREHOUSE_ID="your-warehouse-id"

Or set it up in your IDE / Editor of choice, here is an example from PyhCharm

Configuration approach

See the file OptionalMangledDLTContext.py as a starter. We wrap all operations against mangledlt in a custom context manager, so we can optionally make use of it, or just ignore it (when running the code in databricks) The imports from MangledDLT are built in such a way that they will not be loaded in a databricks context.

To make use of custon config for the context manager above, see the screenshot and code below:

from OptionalMangledDLTContext import OptionalMangledDLTContext
from pyspark.sql import SparkSession


def main():
    config = {
        "host": "yourhost",
        "token": "yourtoken",
        "warehouse_id": "yoursqlwarehouseid",
        "cache_enabled": True,
        "cache_ttl": 600  # 10 minutes
    }

    with OptionalMangledDLTContext(config):
        spark = SparkSession.builder \
            .appName("LocalDev") \
            .getOrCreate()

        # Now you can read from Unity Catalog!
        df = spark.read.table("samples.nyctaxi.trips")
        df.show()


if __name__ == "__main__":
    main()