Harnessing Large Language Models in Enterprise Data Engineering: An On-Call Revolution

Data engineering teams encounter challenges like data quality issues and pipeline failures, especially in enterprise environments. Addressing this, our approach combines the linguistic prowess of models like GPT-4 with data engineering tasks. We autonomously identify and rectify data quality issues, transform anomaly detection paradigms, and automate recovery tasks. Our methodology achieves reduced resolution times, fine-tuned anomaly detectors, and minimized downtime. Empirical evidence showcases enhanced metrics such as reduced MTTR and fewer false positives, advocating a future where AI plays a pivotal role in on-call data engineering. Mitesh Mangaonkar Tech Lead Data Engineer at Airbnb Mitesh is an accomplished Data Engineer and Architect with a remarkable track record in the dynamic realm of information technology and services. He works in the Trust, Risk, and Safety organization at Airbnb. With a versatile skill set encompassing Databases, Big Data technologies, Cloud Computing, Software Development Life Cycle (SDLC), and Hadoop, he has consistently delivered innovative solutions that drive businesses forward. Jing Guo Tech Lead Engineer at Airbnb Engineer who is passion about Data for AI