Enterprise Data Profiling Methodology for Detecting Completeness, Consistency, and Accuracy Anomalies in Account-Level Financial Risk Data
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V2I1P112Keywords:
Data Profiling, Data Quality, Financial Risk, Anomaly Detection, Account-Level DataAbstract
Account-level financial risk data feeds capital calculations, loss provisioning, and regulatory reporting, so hidden defects in this data distort decisions at the highest levels of an institution. Enterprise data profiling detects three defect families early: completeness anomalies, consistency anomalies, and accuracy anomalies. Completeness defects appear as missing or silently substituted values. Consistency defects appear as contradictions between fields, records, and source systems. Accuracy defects appear as recorded values that depart from verified reference values. The core finding is that these families interact, so a program that tests one family in isolation leaves the others unmonitored. Single-column statistics, cross-column dependency discovery, and rule validation together expose defects that no single technique reveals. Bounded, normalized, and interpretable metrics convert profiling output into scores that managers compare across portfolios and reporting dates. Declarative checks and incremental metric computation extend profiling to the volume of records that large institutions process. The key outcome is a layered profiling design that links each defect family to specific profiling tasks, metrics, and controls. The practical significance is direct: risk officers, data stewards, and model owners obtain measurable evidence of data fitness before figures reach capital models and supervisory submissions, and they trace each defect to a responsible source system.
References
[1] Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers? Journal of Management Information Systems, 12(4), 5–33. https://doi.org/10.1080/07421222.1996.11518099
[2] Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002). Data quality assessment. Communications of the ACM, 45(4), 211–218. https://doi.org/10.1145/505248.506010
[3] Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), Article 15. https://doi.org/10.1145/1541880.1541882
[4] Batini, C., Cappiello, C., Francalanci, C., & Maurino, A. (2009). Methodologies for data quality assessment and improvement. ACM Computing Surveys, 41(3), Article 16. https://doi.org/10.1145/1541880.1541883
[5] Ngai, E. W. T., Hu, Y., Wong, Y. H., Chen, Y., & Sun, X. (2011). The application of data mining techniques in financial fraud detection: A classification framework and an academic review of literature. Decision Support Systems, 50(3), 559–569. https://doi.org/10.1016/j.dss.2010.08.006
[6] Naumann, F. (2014). Data profiling revisited. ACM SIGMOD Record, 42(4), 40–49. https://doi.org/10.1145/2590989.2590995
[7] Abedjan, Z., Golab, L., & Naumann, F. (2015). Profiling relational data: A survey. The VLDB Journal, 24(4), 557–581. https://doi.org/10.1007/s00778-015-0389-y
[8] Fan, W. (2015). Data quality: From theory to practice. ACM SIGMOD Record, 44(3), 7–18. https://doi.org/10.1145/2854006.2854008
[9] Heinrich, B., Hristova, D., Klier, M., Schiller, A., & Szubartowicz, M. (2018). Requirements for data quality metrics. ACM Journal of Data and Information Quality, 9(2), Article 12. https://doi.org/10.1145/3148238
[10] Schelter, S., Lange, D., Schmidt, P., Celikel, M., Biessmann, F., & Grafberger, A. (2018). Automating large-scale data quality verification. Proceedings of the VLDB Endowment, 11(12), 1781–1794. https://doi.org/10.14778/3229863.3229867.
