data-quality

What this skill does

# Data Quality

## Purpose

Guide the design and operation of data quality management programs for financial services firms. Covers the six dimensions of data quality (accuracy, completeness, timeliness, consistency, validity, uniqueness) applied to financial data domains, golden source architecture and master data management, data lineage and provenance tracking, validation rule design for security prices, client data, transaction data, and position data, data profiling and anomaly detection, exception management workflows, data quality governance frameworks, and regulatory requirements for data accuracy including BCBS 239, MiFID II, GIPS, and SEC recordkeeping obligations. Enables building or evaluating data quality infrastructure that ensures downstream systems — portfolio management, trading, compliance, reporting, and billing — operate on trustworthy data.

## Layer

13 — Data Integration (Reference Data & Integration)

## Direction

both

## When to Use

- Designing a data quality monitoring framework for a wealth management or asset management platform
- Building validation rules for security pricing, client data, transaction data, or position data pipelines
- Conducting a data quality assessment for regulatory reporting readiness (SEC filings, GIPS, AML/KYC)
- Establishing golden source designations and conflict resolution rules across multiple systems
- Implementing data lineage tracking to satisfy BCBS 239 or MiFID II requirements
- Designing exception management workflows for data quality breaks
- Building data quality scorecards and governance reporting for operations committees
- Investigating root causes of reconciliation breaks, billing errors, or performance calculation discrepancies traced to data quality
- Evaluating data profiling tools or data quality monitoring platforms
- Defining data stewardship roles and accountability structures
- Preparing for regulatory examinations where data accuracy is a focus area
- Trigger phrases: "data quality," "golden source," "data lineage," "data validation," "data profiling," "exception management," "data stewardship," "data governance," "BCBS 239," "data quality scorecard," "validation rules," "data anomaly," "data completeness," "data accuracy"

## Core Concepts

### 1. Data Quality Dimensions for Financial Data

Six dimensions define data quality. Each has domain-specific meaning in financial services.

**Accuracy** — Data values correctly represent the real-world entity or event they describe. A security price is accurate if it reflects the actual market closing price or evaluated value from the designated source. A client address is accurate if it matches the client's current legal address of record. Accuracy failures propagate: an inaccurate price produces inaccurate valuations, performance, billing, and regulatory reports. Accuracy is measured by comparing data against an independent authoritative source — cross-vendor price comparison, custodian-to-PMS reconciliation, client confirmation of personal data. In practice, accuracy is the hardest dimension to measure because it requires an independent reference point for comparison.

**Completeness** — All required data elements are present for every record. A security master record is incomplete if it lacks an ISIN, asset class classification, or pricing source designation. A client onboarding record is incomplete if beneficial ownership for entity accounts is missing. Completeness is measured as the percentage of records with all mandatory fields populated. Financial data completeness requirements are often regulatory: FinCEN requires complete beneficial ownership data, GIPS requires complete portfolio inclusion in composites, SEC Rule 17a-4 requires complete transaction records. Completeness must be defined per record type — a required field for an entity account (beneficial ownership) differs from a required field for an individual account (employment status).

**Timeliness** — Data is available when needed for its intended use. End-of-day pricing must arrive before the nightly valuation batch runs. Trade confirmations must be generated within SEC Rule 10b-10 timeframes. NAV calculations must complete before fund company deadlines. Timeliness is measured as the lag between event occurrence and data availability in consuming systems. Late data is functionally equivalent to missing data if it arrives after the processing window closes. Timeliness requirements vary dramatically by use case: real-time market data must arrive in milliseconds, EOD pricing within hours, and quarterly regulatory filings within weeks.

**Consistency** — The same fact is represented identically across all systems and time periods. A client's legal name must match across the CRM, custodian, PMS, and billing system. A security's sector classification must be the same in the portfolio management system and the compliance monitoring system. Inconsistency typically indicates either a missing golden source designation or a broken synchronization process. Consistency is measured by cross-system comparison for the same entity attribute. Temporal consistency also matters: a security's classification should not change retroactively without documented justification and downstream impact assessment.

**Validity** — Data conforms to defined formats, ranges, and business rules. A CUSIP must be exactly 9 characters with a valid check digit. An account registration type must be one of the firm's defined values. A bond coupon rate cannot be negative (for conventional bonds). A trade settlement date cannot precede the trade date. Validity is enforced through schema constraints, field-level validation, and business rule engines. Invalid data that passes into production indicates insufficient input validation. Validity rules should be versioned and maintained as a formal catalog — when rules change, the change should be documented with effective date and rationale.

**Uniqueness** — Each real-world entity is represented exactly once. A client appearing as two records in the CRM (duplicate due to name variation or data entry error) causes fragmented reporting, missed household billing discounts, and potential compliance failures (wash sale detection across accounts requires a unified client view). A security represented as two master records per custodian causes duplicated positions. Uniqueness is enforced through deduplication at ingestion and periodic duplicate detection scans. Common deduplication techniques include exact-match on identifiers (SSN, CUSIP), fuzzy matching on names and addresses (Jaro-Winkler, Levenshtein distance), and probabilistic matching combining multiple weak identifiers into a confidence score.

| Dimension | Measurement Method | Typical Target | Key Risk if Unmet |
|---|---|---|---|
| Accuracy | Cross-source comparison, reconciliation | >99.5% for pricing, >99% for client data | Incorrect valuations, billing, filings |
| Completeness | Percentage of required fields populated | >98% for critical fields | Regulatory findings, incomplete reporting |
| Timeliness | Lag from event to system availability | Within processing window | Stale valuations, missed deadlines |
| Consistency | Cross-system attribute comparison | >99% agreement | Conflicting reports, audit failures |
| Validity | Format and business rule pass rate | >99.9% | Processing failures, corrupt records |
| Uniqueness | Duplicate detection rate | <0.1% duplicate rate | Fragmented reporting, compliance gaps |

### 2. Golden Source Architecture

A golden source (also called system of record or authoritative source) is the single designated source for a specific data domain. Every data element should have exactly one golden source, and all consuming systems should retrieve that element from the golden source rather than maintaining independent copies.

**Designations by data domain:**

| Data Domain | Typical Golden Source | Rationale |
|---|---|---|
| Security reference data | Security master (fed