Apache Airflow for Generative AI: RAG, Fine-Tuning and LLMOps
In a 2024 AI Connect webinar sponsored by Astronomer, speakers from Astronomer and AWS described how Apache Airflow orchestrates RAG, fine-tuning and monitoring for LLM applications. They shared customer examples and two internal Astronomer tools.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Orchestration is the production layer. Speakers said Airflow feeds, retrains and monitors LLM applications.
- Three kinds of LLM users. Consumers prompt existing models, tuners use RAG or fine-tuning, and providers train new models.
- Keep RAG data fresh. Astronomer runs a weekly Airflow ingest so its Ask Astro chatbot reflects new documentation and issues.
- Close the feedback loop. User ratings on log summaries feed a monthly fine-tuning run of a Mistral 7B model.
- Evaluate with humans. Metrics are a first filter; head-to-head human preference decides between models.
In a 2024 AI Connect webinar sponsored by Astronomer, Steven Hillion, SVP of Data and AI at Astronomer, David Xue, a machine learning engineer at Astronomer, and Parnab Basak, a solutions architect at Amazon Web Services, described how teams were putting generative AI applications into production. Their common thread was orchestration: they said Apache Airflow had become the layer that feeds, retrains and monitors large language model (LLM) applications.
The panelists spoke at the time about customer examples, AWS services and two internal Astronomer tools, and answered audience questions on fine-tuning and evaluation.
Why is Apache Airflow used for generative AI pipelines?
Hillion said data pipelines sit behind every executive dashboard, ETL data set and generative AI application, and Airflow manages those pipelines at scale. He said the open-source project, created at Airbnb and now part of Apache, had reached almost 30 million monthly downloads, and that Astronomer contributes about half of its code.
Airflow began as a tool for data engineers but spread to machine learning teams, Hillion said, because it is written in Python and extensible. He said the community had built integrations for LLM services such as OpenAI and Cohere and for vector databases such as Weaviate and Pinecone.
The feature he stressed was "day two" operations: getting prototypes into production, running them reliably, and keeping them observable and compliant, with lineage tracking.
How do companies consume, tune and build LLMs on AWS?
Basak sorted generative AI users into three groups. Consumers use an existing model through prompt engineering and need little machine learning expertise. Tuners adapt a model with their own data, using retrieval augmented generation (RAG) or fine-tuning. Providers train new models, which he said takes a large data-labeling effort and weeks or months of training.
Basak said Airflow fits each stage. Consumers can use it for model evaluation, prompt evaluation and batch inference, he said. For RAG, it moves data from an existing platform into a vector database. For fine-tuning, it handles labeling, training, deployment and monitoring, and he said frequent retraining on real user inputs keeps a model from going stale.
He also outlined the AWS stack at the time, including Amazon Bedrock for serverless access to multiple foundation models, Amazon SageMaker, and Amazon Q.
What do real-world Airflow AI deployments look like?
Hillion described two Astronomer customers. Laurel, a San Francisco company that automates timekeeping and billing for professional services firms, uses Airflow for data preparation, model training, versioning, drift monitoring and now LLM-written billing summaries. "It's not just that MLOps is key to their business, it is their business," Hillion said.
The second, Dosu, runs an automated software engineering teammate inside GitHub repositories that triages issues and suggests code changes. Hillion said Airflow pipelines ingest data from repositories into vector databases and gather feedback on how users receive its answers.
Basak added examples from AWS customers that used Airflow for orchestration, including discharge summaries at a Japanese healthcare provider and lecture summarization at an Indian online course provider.
How did Astronomer build Ask Astro and its log summarizer?
Xue walked through two internal products. The first summarizes long Airflow error logs into one to three sentences. His team fine-tuned the open-source Mistral 7B model on a mix of its own and synthetic data, he said, then added thumbs-up and thumbs-down ratings from users. Those ratings feed a monthly Airflow-scheduled fine-tuning run.
The second, Ask Astro, is a RAG chatbot that answers questions about Airflow and Astronomer products. Xue said many RAG projects ingest data once with manual scripts and then cannot update it. Astronomer instead runs a weekly Airflow ingest from documentation, GitHub issues, forum answers and Stack Overflow, so new features and renamed functions reach the vector database.
How much data does fine-tuning need, and how do you evaluate output?
Asked how many samples are needed, Xue said it depends on the use case. For style or wording changes, "a thousand to 5,000 is a good ballpark to get started with fine tuning," he said, while teaching a model genuinely new knowledge takes far more.
On validation, Xue cautioned that a single ground-truth wording may not be the best answer, so quantitative metrics work as a first filter. In production, he said, teams often compare an old and new model head to head and see "which one does the human prefer." Basak agreed on human-in-the-loop evaluation and noted that AWS offers a model evaluation service.
Hear more from data and AI leaders in 2027
The AI Enterprise Conference 2027 takes place Tuesday, August 31, 2027, at Pier Sixty in New York City, with sessions on putting AI into production at scale. AI Connect also has 24 free live webinars on its 2027 schedule, including AI Demo Days and Agentic AI Showcases. Details are at /webinars/ and /events/ai-enterprise-conference/.
Frequently asked questions
Why use Apache Airflow for generative AI?
Speakers said Airflow already manages data pipelines at scale, and it integrates with LLM services and vector databases, so it can handle ingestion, retraining and monitoring for AI applications.
What is the difference between RAG and fine-tuning?
In the session, RAG supplies a model with domain data on the fly from a vector database, while fine-tuning trains an existing model on your own data.
How many samples are needed to fine-tune an LLM?
David Xue said it depends on the use case, and suggested 1,000 to 5,000 samples as a starting ballpark when tuning for style or wording.
How do you evaluate generative AI output?
Xue recommended quantitative metrics as a first filter and head-to-head comparisons of old and new models using human preference.
This article is based on remarks by Steven Hillion, SVP of Data and AI, Astronomer; David Xue, Machine Learning Engineer, Astronomer; Parnab Basak, Solutions Architect, Amazon Web Services at AI Connect webinar, 2024 on May 29, 2024. More coverage: Webinars.
Filed under: Data & AI Infrastructure, AI Use Cases & Applications, Observability & Data Quality
