Architectural patterns for managing evolving validation rules in distributed identifier systems

Architectural Decision Memo: Centralized Validation for Distributed Identifier Systems

Context and Problem Statement

In distributed systems, the ingestion of numerical identifiers—such as phone numbers or platform-specific account handles—is rarely a static process. As global operations expand, regional numbering plans evolve, and platform-specific registration signals change, the logic required to validate these identifiers becomes increasingly complex.

Many engineering teams initially embed validation logic directly into individual microservices. This approach often involves hard-coded regex patterns or local libraries that check for format compliance. While simple to implement, this pattern creates significant technical debt. When a regional format changes or a new validation requirement emerges, every service consuming that identifier must be updated, tested, and redeployed. This leads to "validation drift," where Service A accepts an identifier that Service B rejects, resulting in inconsistent data states across the persistence layer.

The Architectural Alternatives

Option 1: Embedded Validation (The Status Quo) Logic resides within each microservice.

• Pros: Zero network latency; no external dependencies; simple deployment for isolated services.
• Cons: High maintenance overhead; inconsistent enforcement; difficult to audit; requires synchronized deployments across the stack to update rules.

Option 2: Centralized, Versioned Validation Service (The Proposed Pattern) A dedicated, decoupled service acts as the source of truth for identifier validation. Microservices query this service (or a cached sidecar) to verify identifiers before processing.

• Pros: Atomic updates; consistent enforcement; centralized auditing; separation of concerns.
• Cons: Introduces network latency; creates a potential single point of failure; requires robust caching strategies to maintain performance.

The Decision: Decoupled Schema-Driven Validation

We have decided to move toward a centralized, versioned validation service. This service will expose a REST API that accepts identifiers and returns a structured validity signal. By decoupling the validation logic, we treat "identifier integrity" as a cross-cutting concern rather than a service-specific implementation detail.

This architecture relies on a schema-driven approach where validation rules are stored as versioned configurations. When a rule changes, we update the configuration in the central service, and all downstream consumers immediately benefit from the updated logic without requiring code changes in their respective repositories.

Operational Risks and Trade-offs

The primary trade-off is the introduction of network overhead. To mitigate this, we implement a two-tier caching strategy:

1. Local Cache: Services maintain an in-memory LRU cache for frequently validated identifiers.
2. Distributed Cache: A shared Redis instance stores validation results for a TTL (Time-to-Live) period, reducing the load on the central validation service.

A notable risk is the "stale validation" scenario. If a rule changes, cached results might remain valid for the duration of their TTL, leading to a temporary period of inconsistency. We address this by implementing a cache-invalidation signal or by keeping TTLs short enough that the window of inconsistency is negligible for our specific business requirements.

A Surprising Observation: The "False Positive" Edge Case

During our evaluation, we encountered a surprising edge case involving platform-specific registration signals. We observed that an identifier might be technically "valid" according to international numbering standards (e.g., E.164 format) but "inactive" or "unregistered" on a specific platform like WhatsApp or Telegram.

This revealed that "validity" is not a binary state. We must distinguish between:

• Format Validity: Does the identifier conform to the expected syntax?
• Platform Presence: Is the identifier currently registered with a specific service?

Our centralized service must handle these as distinct signals. A service requesting validation for a bulk list of phone numbers might require a validity_signal (format check) followed by a platform_registration_signal (presence check). Attempting to conflate these into a single "is_valid" boolean led to significant data quality issues in our downstream analytics pipelines.

Implementation Constraints

When integrating with external platform signals, we must respect the operational constraints of the underlying APIs. For instance, when using a Telegram or WhatsApp checker, we must account for per-user concurrency and timeout behaviors. These services are designed for bulk processing, but they are not infinite resources.

Engineers must consult the current API documentation for each specific checker to understand the appropriate concurrency limits. We do not implement arbitrary rate-limiting in our central service; instead, we rely on the documented behavior of the upstream providers to ensure stability. If a specific checker experiences a timeout, our central service is designed to return a "pending" or "retryable" status rather than failing the entire batch, allowing the calling service to implement an exponential backoff strategy.

Evidence for Invalidation

This architectural decision is not permanent. We would consider this pattern invalidated if:

1. Latency Thresholds: The overhead of the network hop to the validation service exceeds our P99 latency budget for critical ingestion paths, even with aggressive caching.
2. Operational Complexity: The maintenance of the central service becomes a bottleneck, where the team managing the service cannot keep pace with the frequency of rule updates.
3. Data Volume: The volume of validation requests scales to a point where the cost of the centralized service outweighs the benefits of consistency, necessitating a move back to a distributed, library-based approach (where rules are distributed as a versioned package/SDK rather than a service).

Conclusion

By moving away from hard-coded validation logic, we trade a small amount of architectural complexity for significantly higher data integrity. The centralized, versioned validation service provides a single point of control for evolving identifier requirements. While this introduces new considerations regarding network latency and cache management, it eliminates the drift that inevitably occurs when validation logic is fragmented across a distributed system. We prioritize consistency and auditability, accepting the trade-off of a network-dependent validation path.