AI-Ready Data: Automating Feature Engineering at COLLIDE 2023
At COLLIDE 2023 in Atlanta, dotData's Ian McKenna demonstrated a way to turn disjointed tables into a flat, model-ready feature table. He argued for letting the data surface features statistically, then reviewing them with experts.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- The goal is a flat table. Models need one table, but data usually sits in many disjointed ones.
- Manual feature testing does not scale. Testing hypotheses one at a time with SQL becomes impossible at hundreds or thousands.
- Let the data speak first. McKenna proposed generating features statistically before consulting experts.
- Reuse reduces technical debt. Ad hoc notebook code and discarded features add cost across use cases.
- Guardrails matter. Leakage checks, missing-data thresholds and class-imbalance warnings were part of the approach.
At COLLIDE 2023 in Atlanta, Ian McKenna, Director of Field Engineering at dotData, showed how a team can turn many disjointed raw tables into a model-ready feature table without testing every idea by hand. In a roughly 20-minute session, McKenna demonstrated the company's feature engineering tool in a Jupyter notebook and took questions from the audience on leakage, compute and class imbalance.
His argument, as presented at the time: data teams spend too long guessing which features matter, and the shortcuts they take create technical debt.
What does it mean to make data AI ready?
McKenna said most organizations start with many tables spread across data lakes and warehouses, with aggregations still to be done, missing values, outliers and different sizes and frequencies. The goal is a single flat table, "the input for any typical machine learning model," such as regression or classification.
The talk was about getting from one state to the other. McKenna described the tool, called Feature Factory in his remarks, as a library that adds an API and lightweight exploration apps to a Jupyter notebook.
Why is manual feature engineering slow?
Teams usually narrow the search by working with a subject matter expert, McKenna said. Each hypothesis, such as whether engineers who withdraw large sums churn, can mean writing SQL across several databases. Testing ten, a hundred or a thousand of those by hand, he said, "becomes impossible" without automation.
He also pointed to a second problem. Feature work is ad hoc, written quickly in notebooks with poor naming, and repeated use case after use case. That, he said, creates technical debt or leads teams to throw away features and pipelines that could be reused.
McKenna said the classic fix of storing features in a feature store and then running model-based feature selection is slow, depends on the model and gets worse as the number of features grows.
How does the statistical approach work?
McKenna proposed reversing the order. Instead of iterating with experts first, "throw it all at the problem" and let the data show which features correlate most with the target, saving the expert discussion for later.
He laid out three steps:
- Define the inputs. Specify the target variable and the keys that connect the tables.
- Explore the feature space. Refine it, for example by excluding sensitive columns such as health or demographic data after talking with an expert.
- Select and deploy. Pick features from a ranked leaderboard and build feature pipelines for production.
What did the demo show?
The demo used a loan default dataset with three tables. In code, McKenna defined the tables in a Python dictionary, connected a transactions table and a clients table on client ID, and set the goal as a classification problem predicting default.
The tool then generated features from building blocks such as aggregations, thresholds, filters, look-back windows and interactions between columns. McKenna said it examines up to four columns at a time and can evaluate millions of candidate features. He attributed the scale to a Spark back end and to a proprietary out-of-memory method.
Results appeared on a leaderboard ranked by lift. A team that cares more about explainability than accuracy, he said, can adjust parameters to favor features with little collinearity. A single command returns the feature table and the pipeline, which can generate SQL or be saved and loaded into another notebook for deployment.
What did the audience ask?
Questions covered the practical limits of the approach, and McKenna's answers were specific:
- Leakage: The tool checks for features that would use future data in time series problems, restricting each window to historical data.
- Missing data: Columns missing above a threshold, such as 90%, are dropped, and the threshold can be changed.
- Misleading features: Users can deselect ID columns in the feature space beforehand, or filter them out of the leaderboard afterward.
- Selection method: He said it is regularization based, and independent of the downstream model, such as XGBoost or TensorFlow.
- Categorical data: One-hot and target encoding are supported out of the box.
- Compute: It depends on joins, rows and columns. He estimated a multi-gigabyte dataset joined with five tables at about five hours.
- Limits: He cited a limit of 10 million target rows per experiment and a warning when a class falls below roughly 5%. Oversampling is outside the tool.
Where to follow AI-ready data conversations
Data readiness remains a recurring theme at AI Connect events. COLLIDE 2027, the largest AI conference in the Southern U.S. with more than 1,000 enterprise data and AI leaders, takes place Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. Details are on the COLLIDE page. AI Connect also runs 24 free live webinars on its 2027 schedule, listed on the webinars page.
Frequently asked questions
What does it mean to make data AI ready?
McKenna described it as turning many disjointed raw tables into a flat feature table that can feed a machine learning model.
Why is manual feature engineering a problem?
It relies on testing hypotheses one at a time, often with SQL, and produces ad hoc code that adds technical debt.
How does the tool avoid data leakage?
McKenna said it restricts time windows to historical data so future data is not used in time series problems.
How much compute does it need?
He said it depends on joins, rows and columns, and estimated about five hours for a multi-gigabyte dataset joined with five tables.
This article is based on remarks by Ian McKenna, Director of Field Engineering, dotData at COLLIDE Data + AI Conference 2023 on October 3, 2023. More coverage: COLLIDE AI Conference.
Filed under: Data & AI Infrastructure, Observability & Data Quality
