Situation
The client managed its Detecting Financial Crime (DFC) data landscape across 13 separate Databricks workspaces and one Fabric workspace, each with its own access policies. With more than 4,000 permissions to maintain and synchronise across platforms, governance had become unmanageable. On average, files received by the Enterprise Data Lake were copied 12 times, an inefficiency that Unity Catalog could significantly reduce.
Approach
Valcon reused existing Databricks workspaces wherever possible to minimise migration effort and collaborated closely with user teams to guide them through the changes. The approach was tailored to the multi-technology environment spanning Databricks, Fabric and DFC-specific frameworks. Valcon guided teams on new Unity Catalog functionality, introduced Terraform for automated workspace deployment, and designed an AI model governance approach for the MLOps teams.
Results
A single, governed approach to access management was introduced for all data users. Automated lineage was implemented across all data assets, including tables, pipelines, dashboards and notebooks. Infrastructure as Code (IaC) was implemented for Databricks workspaces, significantly reducing the time required from the Platform team. Data sharing via Unity Catalog is expected to deliver significant reductions in cloud storage and compute costs.
