Data engineering · 13 min read

Building a Databricks data platform that could survive reality

I built this platform from scratch for a large industrial company: one lakehouse, many source shapes, and just enough discipline to keep batch and streaming pipelines from turning into unrelated systems.

The interesting part of this project was never “using Databricks.” Plenty of teams do that. The interesting part was making one platform absorb wildly different source systems without producing a different architecture for each of them.

Some feeds arrived as SQL snapshots. Others were JSON blobs dropped in storage by APIs and function apps. Provisioning systems wrote XML. Telemetry arrived as text envelopes with base64 payloads. A licensing stream came from Kafka. Some jobs were naturally batch. Some were continuous in spirit but scheduled in practice. All of it had to land in a medallion architecture that operators could understand and that I could change without fear.

What emerged was a pattern I still like: keep bronze aggressively boring, use the right compute mode for the source, push schema and business meaning upward, and treat governance as part of the data model rather than as a separate admin concern.

Bronze had to be boring on purpose

My bronze layer was not where I wanted to be clever. Its job was to preserve source truth with just enough metadata to make replay, deduplication, and debugging possible later.

That meant different landing patterns for different source families: SQL extracts were written as Parquet snapshots, semi-structured feeds used Auto Loader, and a few especially messy sources were stored as full raw payloads in a single text column so silver could parse them deterministically.

source_file_path  STRING
raw_payload       STRING | STRUCT
ingested_at       TIMESTAMP
ingestion_date    DATE
source_status     STRING   -- optional: succeeded / failed

That shape sounds almost trivial, but it prevented a lot of pain. One JSON pipeline in the repo explicitly switched to wholeText reads because line-based ingestion quietly destroyed file boundaries. Another moved from overwrite to append because re-runs were erasing history. Those are the kinds of bugs you only make once if bronze is treated as a preservation layer instead of a convenience layer.

One platform, two compute models

The cleanest architectural decision in this project was admitting that Databricks serverless and classic clusters are different tools, not interchangeable deployment targets.

Serverless was ideal for short medallion transforms and dbt runs: fast startup, little infrastructure friction, and easy scheduling. Classic clusters existed for the awkward edges: JDBC drivers, Kafka connectivity, and heavier workloads that needed more control.

That split showed up everywhere. One Kafka pipeline used a classic task to read Avro messages from Confluent, wrote Delta to a staging path, then handed off to serverless tasks for bronze deduplication and silver upserts. Several SQL-backed jobs did the same thing with JDBC: extract on classic, transform on serverless.

In other words, I stopped asking “can Databricks do this?” and started asking “which Databricks runtime should own which part of this?” That framing produced much more stable jobs.

Streaming worked best when it still looked like a job

A lot of the platform behaves like streaming without requiring me to run never-ending streams. Auto Loader and Kafka checkpoints let me use trigger(availableNow=True) so each scheduled run consumed everything since the previous checkpoint, then stopped.

(
  spark.readStream
    .format("kafka")
    .options(**kafka_options)
    .load()
    .writeStream
    .option("checkpointLocation", checkpoint_path)
    .trigger(availableNow=True)
    .start(staging_path)
    .awaitTermination()
)

I liked this pattern more than a permanently running stream for two reasons. First, operations became predictable: one job run, one set of logs, one obvious failure boundary. Second, it unified the mental model across batch and streaming sources. In both cases the scheduler said “process whatever is new,” and the checkpoint provided the continuity.

The Kafka pipeline added another important layer: bronze deduplicated on topic, partition, and offset before appending, and silver merged on business key. That gave me protection against retries, historical re-reads, and source-side redelivery without pretending the raw stream was already clean.

Schema evolution was not one problem

One lesson I relearned here: “schema evolution” is not a single strategy. XML, permissive JSON, and text-wrapped payloads each fail differently, so they deserve different responses.

For XML provisioning feeds, I used Auto Loader with schemaEvolutionMode = "addNewColumns". Those sources were fairly well behaved, and the right failure mode was to surface new fields without blocking ingestion.

For more fragile JSON feeds, I preferred schemaEvolutionMode = "rescue". That kept unexpected fields in rescued data rather than silently coercing or dropping them. In practice, it bought me time: ingestion could keep moving while I decided whether the new field belonged in silver.

And for a few telemetry-style payloads, I deliberately avoided schema inference in bronze altogether. Reading the entire file as raw text was less sophisticated, but much safer than baking source assumptions into the first layer of the platform.

dbt was the backbone, not the whole skeleton

dbt earned its place in the silver and gold layers because a lot of the hard work eventually became relational: joins, deduplication, dimensional modeling, tests, and documentation. The repo grew into a real dbt project with separate silver and gold domains, source definitions, schema tests, and a mix of full tables, views, and incremental models.

{{ config(
  materialized='incremental',
  incremental_strategy='merge',
  unique_key='source_file_path'
) }}

select *
from {{ source('silver_uploads', 'events') }}
{% if is_incremental() %}
where upload_date >= (select coalesce(max(upload_date), date('2020-01-01')) from {{ this }})
{% endif %}

But I also tried not to force dbt into problems it was not built to solve elegantly. Parsing XML structures, decoding base64 payloads, dealing with source-specific timestamp oddities, or reconstructing deeply nested telemetry shapes stayed in notebooks first. Once the data became tabular and stable, dbt took over.

That line mattered. In one MPC domain, dbt parsed raw IoT Hub envelopes and also maintained append-only configuration tables where defaults were auto-provisioned for newly seen IDs without overwriting values owned by an app. In another area, Delta Live Tables was a better fit because I wanted expectations and reusable live views around telemetry register history.

The weirdest source bug was a SQL type, not a stream

The most memorable ingestion problem in the whole platform came from a SQL snapshot, not from Kafka.

One upstream database wrote Parquet with SQL Server TIME(6) semantics. Spark's normal Parquet path did not like that logical type and, annoyingly, failed lazily enough that a naive try/except around the read was useless. The fix was to inspect a sample file with PyArrow, map unsupported types manually, then re-read the dataset with an explicit Spark schema.

pa_schema = pq.read_schema(io.BytesIO(sample_bytes))
spark_schema = StructType([
  StructField(field.name, map_type(field.type), True)
  for field in pa_schema
])

df = spark.read.schema(spark_schema).parquet(path)

I like that example because it captures what production data engineering feels like. The glamorous architecture decision was medallion + dbt + DLT. The actual Tuesday problem was “why does this one Parquet logical type break a perfectly normal ingest?”

That same pipeline also stripped a password column before silver on purpose. Even in an internal platform, I wanted the rule to be simple: if a field should not survive ingestion, remove it as early and as explicitly as possible.

Governance was part of the design, not the last sprint

Unity Catalog was not just the place where tables happened to live. It was the mechanism that let the platform stay understandable as it expanded across domains.

I kept bronze, silver, and gold schemas distinct; used managed tables where possible; versioned workspace objects through Databricks Asset Bundles; and treated permissions as code. There was even a schema-drift check in CI that snapshots Unity Catalog definitions and fails the pipeline if the checked-in contract no longer matches what exists in the workspace.

That might sound bureaucratic, but it solved a real problem: lakehouse platforms decay quickly if tables can change silently. The snapshot diff forced intentionality. If a schema changed, somebody had to decide whether that was a feature, a migration, or a bug.

The same thinking showed up in workspace governance. Not every user could spin up their own cluster. Shared warehouses and shared classic clusters existed for a reason, and access followed environment-specific groups instead of ad hoc permissions. It kept cost, reproducibility, and supportability tied together.

Quality checks lived in multiple layers

I did not want one grand “data quality framework.” I wanted several small mechanisms that each caught a specific class of mistake.

• Notebook unit tests for transformation helpers, like timestamp parsing and column normalization.
• dbt tests for not-null, uniqueness, accepted values, and documented source contracts.
• DLT expectations where dropping or failing bad records was part of the table contract.
• Quarantine tables for records that should be investigated rather than silently discarded.

I especially like the quarantine pattern. In the MPC models, rows with unparseable timestamps are not simply thrown away. They are written to dedicated quarantine tables with a reason attached. That changes the operational conversation from “the dashboard looks off” to “here are the exact messages that violated the contract.”

What I would change

If I were starting again, I would standardize the ingest contract even harder. The platform already converged on common ideas - source file path, ingested timestamp, append-first bronze, replayability - but the implementation still reflects the history of real projects arriving one by one.

I would probably build a thinner shared ingestion toolkit earlier: common observability, a more explicit contract for succeeded/failed subfolders, and fewer one-off decisions about when to stage through storage versus writing directly to managed tables.

I would also invest sooner in lineage and producer-facing contracts. The platform is already strong at absorbing schema drift. That is useful, but it can make a team too good at tolerating upstream inconsistency. A mature platform should be resilient without becoming a place where source owners never feel the cost of changing things.

The stack

• Storage and tables: Delta Lake on Databricks with bronze, silver, and gold schemas in Unity Catalog.
• Ingestion: Auto Loader, JDBC-based extracts, Azure Functions, and Kafka staging with scheduled checkpointed streams.
• Transforms: PySpark notebooks for source-heavy parsing; dbt for relational silver and gold models; Delta Live Tables for selected telemetry workloads.
• Governance: Unity Catalog permissions, schema snapshots in CI, and workspace configuration managed as code.
• Delivery: Databricks Workflows, Databricks Asset Bundles, and Azure DevOps pipelines. No click-ops required.

The result was not just a set of pipelines. It was a data platform with opinions: preserve raw truth first, make compute choices deliberately, let contracts get stricter as data gets cleaner, and never treat governance as something you add once the useful work is done.