Agentic AI for Autonomous Data Pipelines: A Conceptual Framework for Intelligent ETL/ELT, Orchestration, and Self-Healing
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V7I4P104Keywords:
Agentic AI, Autonomous Data Pipelines, ETL/ELT, Data Orchestration, Self-Healing Systems, Multi-Agent Systems, Schema Drift, Data EngineeringAbstract
As enterprise data pipelines are becoming increasingly complex to operate, architectures that make autonomous decisions, dynamically adapt to changes, and perform self-healing are needed. This paper introduces the Agentic Autonomous Data Pipeline (AADP) framework, a three-layer conceptual model that embeds AI agents into extraction, transformation and load (ETL/ELT) functions, an Intelligent Orchestration Engine (IOE), and a Self-Healing Control Loop (SHCL). Agents specialized according to layer roles are responsible for perception, reasoning, and intelligent action to automate the flow of data without manual intervention. The model proposes an agent role classification, an agent communication protocol, a failure taxonomy framework, and a confidence-bounded autonomy model for agent authority. We compare this architecture to static conventional pipelines and discuss expected gains in Fault tolerance, Schema drift resilience and Mean Time to Recovery (MTTR) as potential outcomes. This paper is offered as a conceptual paper for empiric development and experimentation.
References
[1] T. Akidau et al., "The dataflow model: A practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing," Proc. VLDB Endowment, vol. 8, no. 12, pp. 1792–1803, 2015.
[2] M. Stonebraker and U. Cetintemel, "One size fits all: An idea whose time has come and gone," in Proc. 21st Int. Conf. Data Eng. (ICDE), Tokyo, Japan, 2005, pp. 2–11.
[3] T. Akidau, R. Bradshaw, C. Chambers et al., "MillWheel: Fault-tolerant stream processing at internet scale," Proc. VLDB Endowment, vol. 6, no. 11, pp. 1033–1044, 2013.
[4] J. O. Kephart and D. M. Chess, "The vision of autonomic computing," Computer, vol. 36, no. 1, pp. 41–50, Jan. 2003.
[5] J. Kreps, N. Narkhede, and J. Rao, "Kafka: A distributed messaging system for log processing," in Proc. NetDB Workshop, Athens, Greece, 2011.
[6] M. Zaharia et al., "Apache Spark: A unified engine for big data processing," Commun. ACM, vol. 59, no. 11, pp. 56–65, Nov. 2016.
[7] S. J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed. Hoboken, NJ: Pearson, 2020.
[8] N. R. Jennings, "On agent-based software engineering," Artif. Intell. vol. 117, no. 2, pp. 277–296, 2000.
[9] G. Weiss, Ed., Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence. Cambridge, MA: MIT Press, 1999.
[10] D. Sculley et al., "Hidden technical debt in machine learning systems," in Proc. 28th Conf. Neural Inf. Process. Syst. (NeurIPS), Montreal, Canada, 2015, pp. 2503–2511.
