Timextender
1 min read
-
No single place to manage data quality metadata
Fabric now offers strong building blocks for data quality. You can use Great Expectations in Spark environments, configure Purview Data Quality for no-code rule definition and profiling, and lean on Data Factory activities or materialized lake views with constraints to enforce checks closer to storage.What’s still missing is a single, dedicated data quality module that ties everything together. There is no central place that holds all rule definitions, thresholds, and owners, applies those rules consistently across pipelines, Lakehouses, Warehouses, and semantic models, and surfaces a unified, cross-domain view of data quality.
Because of that, quality logic tends to spread across notebook code in data engineering, T-SQL in warehouses and views, Purview Data Quality rules and scorecards, and custom audit tables or dashboards. Each of these depends on metadata, but they aren’t driven from one shared metadata model, which increases effort and makes consistency harder to achieve.
-
Metadata fragmentation and partial lineage
Out of the box, Fabric’s metadata story is improving quickly, especially with the OneLake Catalog’s Govern tab reaching general availability and deeper integration with Purview. That said, many teams still run into fragmentation in day-to-day work.Different artifacts – Power BI reports, pipelines, Lakehouses, Warehouses, and notebooks – each manage metadata in slightly different ways. Cross-workspace lineage is often incomplete for non–Power BI items, particularly when custom orchestration or external tools are in the mix. Column-level lineage is strong inside Power BI semantic models, but less granular for some upstream layers.
The net effect is that lineage and impact analysis often stop at the boundaries of a workspace or workload. To compensate, teams export metadata with Scanner APIs or build their own registries to close the gaps. This isn’t a flaw in Fabric’s direction so much as a normal stage in the evolution of a fast-moving platform. But if you don’t design a metadata strategy up front, your data quality strategy will end up scattered by default.
-
Manual effort to encode quality rules as code
Across community examples and real projects, the same pattern keeps showing up. Null checks are implemented manually at ingestion into the raw Lakehouse. Pattern validations for emails, phone numbers, and IDs are written in PySpark or T-SQL. Referential integrity checks between fact and dimension tables are coded explicitly, and business rule validations (age limits, date ranges, domain logic) are embedded directly inside notebooks and stored procedures.In practice, you often see a series of PySpark snippets for null detection, duplicate detection, outlier detection, and referential integrity checks, all hand-coded and wired into pipelines.
All of this is metadata, but it lives inside code rather than in a central, declarative model. That makes it harder to reuse rules across domains, to hand work off when key engineers move on, and to prove to auditors how, when, and where data was validated.
-
Configuration and governance overhead for Purview Data Quality
Purview Data Quality is a meaningful step forward: it brings profiling, rule recommendation, scorecards, and integration with the Purview Hub into the picture.At the same time, data teams still report friction in getting it fully operational. There are multiple prerequisites around admin API responses, service principals, managed identities, and workspace permissions. You need to work with specific file formats and Lakehouse targets. Much of the configuration is managed through the UI rather than code, which complicates infrastructure-as-code practices and makes promotion across environments more manual than teams would like.
So while Purview Data Quality strengthens the platform, the configuration and governance overhead means many organizations still lean on custom, code-centric approaches for day-to-day quality checks unless they put an explicit metadata strategy in place.
Fabric and Purview evolved separately and are now being brought closer together. As a result, many organizations still lean heavily on custom, metadata-light code for day-to-day quality checks, while Purview focuses on profiling and high-level scorecards.
