DEA-C01 Catalogs, Crawlers and Metadata Guide

Data platforms fail fast when metadata goes stale or unreadable. DEA-C01 expects you to know the role of the catalog, crawlers, partitions, and schema tracking because many downstream services depend on them.

Data catalog: Central metadata layer that tells query and pipeline services what data exists, where it lives, and how it is structured.

Crawler: AWS Glue feature that scans data sources and infers schemas and partitions for catalog entries.

Metadata freshness: Keeping schema, partition, and connection details current enough that downstream consumers can still query or process the data correctly.

What AWS is really testing here

AWS wants you to separate:

• storing data from describing data
• automatic discovery from controlled metadata management
• technical schema registration from business cataloging and lineage needs
• “the files exist in S3” from “Athena, Glue, or Redshift Spectrum can actually use them”

DEA-C01 often hides metadata failures inside query problems. The data may exist and still be unusable because the catalog, partition state, or schema registration is stale or wrong.

Cataloging chooser

Catalog, crawler, connection, and business context are different layers

Metadata control path

This path matters because the data is not useful to analysts until the metadata layer is usable too.

    flowchart LR
	  A["Data source"] --> B{"How is metadata created?"}
	  B -->|Automatic discovery| C["Glue crawler"]
	  B -->|Controlled definition| D["Manual table definition"]
	  A --> E["Glue connection when source integration is needed"]
	  C --> F["Glue Data Catalog"]
	  D --> F
	  E --> F
	  F --> G["Athena, Glue, Spectrum, downstream analytics"]

Crawlers versus manual definitions

How strong DEA-C01 answers usually reason

1. Ask whether the problem is technical metadata registration, automated discovery, schema control, or business context.
2. Use Glue Data Catalog as the shared technical metadata layer.
3. Use crawlers when discovery should be automated and the inference risk is acceptable.
4. Prefer manual definitions when schema precision matters and crawler mistakes are costly.
5. Separate business cataloging from the technical schema layer.

Partitions and metadata freshness

Decision order that usually wins

When metadata answers blur together, use this order:

1. Decide whether the issue is catalog presence, schema discovery, schema control, partition freshness, or business context.
2. If many analytics services need one technical metadata plane, choose Glue Data Catalog.
3. If changing raw files must be discovered automatically, choose a Glue crawler.
4. If curated definitions must stay precise, choose manual table control over crawler convenience.
5. If the question is really about ownership, glossary, or lineage, move to the business catalog lane instead of stopping at technical registration.

Common tie-breaks

Common traps

Harder scenario question

A team stores new Parquet files in partitioned S3 prefixes every day. Athena queries work only for old partitions, and a crawler sometimes misreads rare optional columns. What is the strongest reading first?

• A. Remove the catalog entirely and query S3 directly
• B. Keep the Glue Data Catalog, refresh partition metadata, and use tighter crawler or manual table control
• C. Replace Athena with Route 53
• D. Disable partitions so the crawler has less to scan

Correct answer: B. DEA-C01 expects you to preserve the catalog layer, keep partitions synchronized, and choose more controlled schema management when crawler inference becomes risky.

Quiz