DE-PRO Cheatsheet — DLT, Streaming, Performance & Reliability on Databricks

Use this for last‑mile review. Pair it with the Resources for coverage and IT Mastery to harden production instincts.

CDC: Change data capture, where upstream inserts, updates, and deletes are represented as change events for downstream processing.

Checkpoint: Streaming state and progress location used to recover correctly after restart.

Watermark: Streaming rule that bounds how late data can arrive before the engine stops updating old state.

DLT: Delta Live Tables, Databricks framework for declarative, managed data pipelines.

1) Production pipeline lens (what DE-PRO is really testing)

If two answers “work,” choose the one that is:

• Recoverable: checkpoints/state, idempotent writes, safe retries
• Observable: clear metrics/logs, quality gates, lineage
• Low blast radius: staged changes, small reversible steps

2) Incremental batch + CDC (the safest default patterns)

Upsert with MERGE (CDC)

1MERGE INTO silver t
2USING cdc s
3ON t.id = s.id
4WHEN MATCHED AND s.op = 'D' THEN DELETE
5WHEN MATCHED THEN UPDATE SET *
6WHEN NOT MATCHED THEN INSERT *;

Rules of thumb

• Ensure the ON condition is unique on the source side.
• Make pipelines idempotent: re-running the same input should not double-apply changes.

3) Structured Streaming essentials (checkpointing, watermarks, late data)

The two most-tested concepts

Streaming write (conceptual template)

1(df
2  .withWatermark("event_time", "10 minutes")
3  .writeStream
4  .format("delta")
5  .option("checkpointLocation", "/chk/orders")
6  .outputMode("append")
7  .start("/delta/silver/orders"))

Exam cues

• Late data policy changes outcomes (drop vs update aggregates).
• Triggers control latency/cost; stateful ops need careful tuning.

4) Delta Live Tables (DLT) — what to remember

DLT is about declarative pipelines with built-in operational structure:

• pipeline graph (table dependencies)
• quality expectations
• managed execution/monitoring

    flowchart LR
	  BR["Bronze (ingest)"] --> SI["Silver (clean + dedupe)"]
	  SI --> GO["Gold (metrics)"]

Operator mindset: treat expectations as guardrails; don’t silently pass bad data downstream.

5) Performance pickers (shuffle, skew, file layout)

Shuffle vs skew diagnosis

File layout rules of thumb

• Don’t over-partition (small file explosion).
• Compact when needed; keep partition columns low/medium cardinality.
• Use Z-order/data skipping where supported (concept-level).

6) Reliability + troubleshooting quick pickers

• Streaming duplicates: checkpoint misuse, non-idempotent sink, or incorrect output mode for stateful ops.
• State grows forever: missing/incorrect watermark; unbounded aggregation.
• MERGE “explodes” rows: source not unique on merge keys.
• Reprocessing/backfill: prefer explicit versioning and safe re-runs over ad-hoc manual deletes.