Articles | Open Access | https://doi.org/10.55640/ijdsml-06-02-15

Event-Sourced Governance Of Data Quality Rules For Real-Time, Auditable Healthcare Data Platforms

Ritulkumar Patel , Independent Researcher, Boston, MA, USA

Abstract

Rule-based checks are central to data quality (DQ) assessment on healthcare data platforms, and their parameters change often. In regulated settings each change must be approved, promoted through staged environments, and later explained to auditors, yet DQ tools rarely govern the rule set itself. This study asks whether event sourcing can make such changes both fast and verifiable. We propose an architecture in which every rule-lifecycle action is an immutable, hash-chained event in a log partitioned by rule; separation of duties and risk-tiered approval are enforced before an event is written; idempotent projections drive gated promotion; rule parameters are bound at runtime; and the log head is anchored outside the platform. We evaluated the design in a single-process simulation on synthetic patient data against three baselines: a dbt rebuild pipeline, a mutable parameter table with an audit trigger, and a ledger-style table with a hash-chained audit. The simulation reproduces the relevant Kafka log semantics but is not a production deployment. Approved changes reached production in a median of 0.092 s, against tens of seconds to minutes for the rebuild pipeline; the two table baselines were faster still, so the speed comes from runtime parameter binding rather than from event sourcing. With every record anchored, the hash-chained designs detected all seven tested attack types, whereas the unchained audit table missed every edit of attribution or value. Historical states were reconstructed exactly, an independent verifier found no constraint violations in normal runs and flagged every injected violation, and ablations identified the components that correctness depends on. The main limitations are synthetic data, a simulated log, and modeled rather than measured approval delays

 

Keywords

Data quality, Data governance, Event sourcing

References

M. G. Kahn et al., “A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data,” eGEMs, vol. 4, no. 1, Art. no. 1244, 2016, doi: 10.13063/2327-9214.1244.

N. G. Weiskopf and C. Weng, “Methods and dimensions of electronic health record data quality assessment: Enabling reuse for clinical research,” J. Am. Med. Inform. Assoc., vol. 20, no. 1, pp. 144–151, 2013, doi: 10.1136/amiajnl-2011-000681.

Z. Wang, J. R. Talburt, N. Wu, S. Dagtas, and M. N. Zozus, “A rule-based data quality assessment system for electronic health record data,” Appl. Clin. Inform., vol. 11, no. 4, pp. 622–634, 2020, doi: 10.1055/s-0040-1715567.

Y. Mohamed et al., “Electronic health record data quality variability across a multistate clinical research network,” J. Clin. Transl. Sci., vol. 7, no. 1, Art. no. e130, 2023, doi: 10.1017/cts.2023.548.

C. Blacketer, F. J. Defalco, P. B. Ryan, and P. R. Rijnbeek, “Increasing trust in real-world evidence through evaluation of observational data quality,” J. Am. Med. Inform. Assoc., vol. 28, no. 10, pp. 2251–2257, 2021, doi: 10.1093/jamia/ocab132.

Security Standards: Technical Safeguards—Audit Controls, 45 C.F.R. § 164.312(b). [Online]. Available: https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-C/part-164/subpart-C/section-164.312

Policies and Procedures and Documentation Requirements—Time Limit, 45 C.F.R. § 164.316(b)(2)(i). [Online]. Available: https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-C/part-164/subpart-C/section-164.316

Regulation (EU) 2016/679 of the European Parliament and of the Council (General Data Protection Regulation), Off. J. Eur. Union, vol. L 119, pp. 1–88, May 2016. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2016/679/oj

J. Declerck et al., “Assessing data quality in heterogeneous health care integration: Simulation study of the AIDAVA framework,” JMIR Med. Inform., vol. 13, Art. no. e75275, 2025, doi: 10.2196/75275.

S. Schelter, D. Lange, P. Schmidt, M. Celikel, F. Biessmann, and A. Grafberger, “Automating large-scale data quality verification,” Proc. VLDB Endow., vol. 11, no. 12, pp. 1781–1794, 2018, doi: 10.14778/3229863.3229867.

E. Breck, N. Polyzotis, S. Roy, S. E. Whang, and M. Zinkevich, “Data validation for machine learning,” in Proc. Conf. Mach. Learn. Syst. (SysML), Stanford, CA, USA, 2019. [Online]. Available: https://research.google/pubs/data-validation-for-machine-learning/

S. S. Kumar, R. Chandra, and S. Agarwal, “Rule based complex event processing for an air quality monitoring system in smart city,” 2024, arXiv:2403.14701, doi: 10.48550/arXiv.2403.14701.

A. Polimeno, C. Braghin, M. Anisetti, and C. A. Ardagna, “Maximizing data quality while ensuring data protection in service-based data pipelines,” J. Big Data, vol. 12, Art. no. 62, 2025, doi: 10.1186/s40537-025-01118-5.

Open Policy Agent, “OPA documentation,” openpolicyagent.org. [Online]. Available: https://openpolicyagent.org/docs (accessed Oct. 22, 2026).

M. Zaharia et al., “Accelerating the machine learning lifecycle with MLflow,” IEEE Data Eng. Bull., vol. 41, no. 4, pp. 39–45, 2018. [Online]. Available: http://sites.computer.org/debull/A18dec/p39.pdf

Microsoft, “Manage model lifecycle using the Workspace Model Registry (legacy),” Azure Databricks documentation. [Online]. Available: https://docs.azure.cn/en-us/databricks/mlflow/model-registry (accessed Oct. 22, 2026).

M. Comuzzi, J. Ko, and F. Maggi, “A language to model and simulate data quality issues in process mining,” ACM J. Data Inf. Qual., vol. 17, no. 2, pp. 1–36, 2025, doi: 10.1145/3743144.

F. Zhou, Z. Wang, Q. Liu, J. Li, and P. Liu, “Programming every example: Lifting pre-training data quality like experts at scale,” in Proc. 42nd Int. Conf. Mach. Learn. (ICML), PMLR, vol. 267, 2025, pp. 79281–79327. [Online]. Available: https://proceedings.mlr.press/v267/zhou25aa.html

M. Kleppmann, Designing Data-Intensive Applications. Sebastopol, CA, USA: O’Reilly Media, 2017, ISBN: 978-1-449-37332-0.

M. Overeem, M. Spoor, S. Jansen, and S. Brinkkemper, “An empirical characterization of event sourced systems and their schema evolution—Lessons from industry,” J. Syst. Softw., vol. 178, Art. no. 110970, 2021, doi: 10.1016/j.jss.2021.110970.

J. Kreps, N. Narkhede, and J. Rao, “Kafka: A distributed messaging system for log processing,” in Proc. 6th Int. Workshop Networking Meets Databases (NetDB), Athens, Greece, 2011. [Online]. Available: https://www.microsoft.com/en-us/research/wp-content/uploads/2017/09/Kafka.pdf

B. Schneier and J. Kelsey, “Secure audit logs to support computer forensics,” ACM Trans. Inf. Syst. Secur., vol. 2, no. 2, pp. 159–176, 1999, doi: 10.1145/317087.317089.

S. A. Crosby and D. S. Wallach, “Efficient data structures for tamper-evident logging,” in Proc. 18th USENIX Security Symp., Montreal, QC, Canada, 2009, pp. 317–334. [Online]. Available: https://www.usenix.org/conference/usenixsecurity09/technical-sessions/presentation/efficient-data-structures-tamper-evident

S. Haber and W. S. Stornetta, “How to time-stamp a digital document,” J. Cryptol., vol. 3, no. 2, pp. 99–111, 1991, doi: 10.1007/BF00196791.

B. Laurie, A. Langley, and E. Kasper, “Certificate Transparency,” IETF, RFC 6962, Jun. 2013, doi: 10.17487/RFC6962.

P. Antonopoulos, R. Kaushik, H. Kodavalla, S. Rosales Aceves, R. Wong, J. Anderson, and J. Szymaszek, “SQL Ledger: Cryptographically verifiable data in Azure SQL Database,” in Proc. Int. Conf. Manage. Data (SIGMOD), 2021, doi: 10.1145/3448016.3457558.

X. Yang, Y. Zhang, S. Wang, B. Yu, F. Li, Y. Li, and W. Yan, “LedgerDB: A centralized ledger database for universal audit and verification,” Proc. VLDB Endow., vol. 13, no. 12, pp. 3138–3151, 2020, doi: 10.14778/3415478.3415540.

K. Kulkarni and J.-E. Michels, “Temporal features in SQL:2011,” ACM SIGMOD Rec., vol. 41, no. 3, pp. 34–43, 2012, doi: 10.1145/2380776.2380786.

M. Armbrust et al., “Delta Lake: High-performance ACID table storage over cloud object stores,” Proc. VLDB Endow., vol. 13, no. 12, pp. 3411–3424, 2020, doi: 10.14778/3415478.3415560.

Apache Software Foundation, “Iceberg table spec,” Apache Iceberg. [Online]. Available: https://iceberg.apache.org/spec/ (accessed Oct. 22, 2026).

E. Politou, E. Alepis, and C. Patsakis, “Forgetting personal data and revoking consent under the GDPR: Challenges and proposed solutions,” J. Cybersecurity, vol. 4, no. 1, Art. no. tyy001, 2018, doi: 10.1093/cybsec/tyy001.

B. Fitzgerald, K.-J. Stol, R. O’Sullivan, and D. O’Brien, “Scaling agile methods to regulated environments: An industry case study,” in Proc. 35th Int. Conf. Softw. Eng. (ICSE), San Francisco, CA, USA, 2013, pp. 863–872, doi: 10.1109/ICSE.2013.6606635.

dbt Labs, “dbt Core,” GitHub repository. [Online]. Available: https://github.com/dbt-labs/dbt-core (accessed Oct. 22, 2026).

J. Walonoski et al., “Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record,” J. Am. Med. Inform. Assoc., vol. 25, no. 3, pp. 230–238, 2018, doi: 10.1093/jamia/ocx079.

M. Raasveldt and H. Mühleisen, “DuckDB: An embeddable analytical database,” in Proc. Int. Conf. Manage. Data (SIGMOD), Amsterdam, The Netherlands, 2019, pp. 1981–1984, doi: 10.1145/3299869.3320212.

H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” Ann. Math. Statist., vol. 18, no. 1, pp. 50–60, 1947, doi: 10.1214/aoms/1177730491.

N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions,” Psychol. Bull., vol. 114, no. 3, pp. 494–509, 1993, doi: 10.1037/0033-2909.114.3.494.

B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York, NY, USA: Chapman & Hall, 1993, doi: 10.1201/9780429246593.

Article Statistics

Downloads

Download data is not yet available.

Copyright License

Download Citations

How to Cite

Ritulkumar Patel. (2026). Event-Sourced Governance Of Data Quality Rules For Real-Time, Auditable Healthcare Data Platforms. International Journal of Data Science and Machine Learning, 6(02), 241-260. https://doi.org/10.55640/ijdsml-06-02-15