Data Science Wire

Install private package dependency in declarative pipeline

Reddit r/databricks1mo4 min read

Hi, i am currently using databricks automation bundle to create a python package within the bundle. I have also configured a databricks declarative pipeline that uses this package to create a dummy table. This approach works when working with one dependency which is publicly available: *pyproject.toml* dependencies = ["quinn"] *databricks.yml* resources: pipelines: acd_pipelines_pipeline: name: "${bundle.name}_pipeline" serverless: true continuous: false libraries: - glob: include: ./pipeline/** environment: dependencies: - "${workspace.artifact_path}/.internal/acd_pipelines-0.1.0-py3-none-any

Read the full story at Reddit r/databricks

More in MLOps / LLMOps