Human Inputs in AI Security: How Much Can You Trust Your Labels?
In a session at the Data Science Innovators Conference 2023, Samuel Zaruba Smith, a director at Bank of America, said human labels and annotations in supervised security AI are of questionable value and add less as systems improve. He described imbalanced threat data, human-driven failure modes and how to check unsupervised systems.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Labels are not free of risk. Smith said supervised security systems rely on human inputs of questionable value.
- Human lift is limited. He cited experts who put the gain from human inputs at roughly 15 percent at most, and shrinking.
- Failures are mostly human. Mistakes, social engineering and bad labeling cause most security failures, he said.
- Threat data is skewed. Roughly 95 percent benign, 5 percent threat, across several data sets he cited.
- Check before production. Judge the value of human inputs and watch for drift before deploying.
In a session at the Data Science Innovators Conference 2023, Samuel Zaruba Smith, a director at Bank of America whose background spans security, data science, AI and data governance, argued that companies adopting AI for security tasks need to judge how much they can trust the human inputs those systems depend on. His talk, "Applied Security in the Web3 and Data Space," drew on his research and expert opinion about labels, annotations and human-in-the-loop approaches.
Why do human inputs matter for AI security?
Smith said AI use varies in sensitivity. A marketing model that recommends a poorly matched product has a limited effect on the bottom line, he said, but fraud detection, spam filtering and removing hate speech from social platforms carry ethical or security risk.
Those systems are very often supervised, he said, and the vast majority of industry use cases remain supervised to this day, meaning they depend on human inputs "of sometimes questionable value." Much of his work, he said, is judging the value of those inputs and the risks attached to them.
How much do human labels actually add?
Smith said a number of experts believe human inputs to an AI system are worth roughly 15 percent at most in added performance, and that the figure can be much lower in some cases. He pointed to a 2018 Harvard Business Review piece as part of the motivation, which he described as arguing that human inputs are becoming less valuable as AI grows more powerful.
He said the figure changes drastically with the quality of the inputs. Knowing how much to trust human labels, he argued, is as important as knowing how much lift they provide. In his telling, a human may have added a 20 percent accuracy gain 30 years ago, but that gap is shrinking as architectures grow more complex and telemetry makes data richer.
What goes wrong when people are in the loop?
Smith said most security failures are not technical coding errors. Far more often, he said, the cause is human: mistakes, social engineering, potentially bribery, or bad labeling in training data, whether deliberate or not. Privacy, bias, data drift and data poisoning issues can mostly be traced back to humans interacting with AI systems at the training-data level, in a human-in-the-loop step or through prompts, he said.
He also described performance drift: when new data differs from historical data, for example because of a legal change or a shift in the customer population, earlier labels and annotations stop being as useful. Judging the value of human inputs before production, he said, is a highly valuable assessment.
What does the data look like in security problems?
Smith said security data is highly imbalanced. By his account, roughly 95 percent of data is benign and about 5 percent is a real threat, and only about 5 percent of threats are investigated. He gave examples of that pattern:
- A well-known public banking data set in which fewer than 1 percent of points are known fraud or money laundering.
- Cyberattack and e-commerce data sets, commonly under 5 percent.
- Physical security video, where fewer than 5 percent of data points are identified as malicious.
- Electrical power outage data, with around a 10 percent true-positive rate, and cryptocurrency and check-fraud data sets at about 5 percent.
What does an ideal system look like, and how can it be checked?
Smith described an "ideal gold standard" as a privacy-preserving, unsupervised system that needs no human input and finds the small share of cases that matter. His examples included personal digital ID systems with built-in protection for payments, hate speech and image detection on platforms such as video games and social media, and drift and bias prevention. He also pointed to a growing market of AI auditing tools for fairness and bias checking and for estimating the value of human labels.
Asked in the chat how to confirm AI works correctly without human oversight, he said there is substantial literature on optimizing unsupervised systems. Industry-specific studies describe the distributions a data set should show, and teams can use that, in "almost like a Bayesian sense," to set cutoffs and build anomaly detection. Humans may still review whether the system is running correctly, he said, rather than labeling each case as a threat or not.
What is next from AI Connect?
AI Connect continues to run sessions on AI risk and security. The 2027 lineup includes 24 free live webinars, listed on the webinars page, and COLLIDE 2027 takes place Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta; see COLLIDE. Registration is coming soon.
Frequently asked questions
Why are human inputs a risk in AI security systems?
Smith said supervised systems depend on human labels that can contain errors, social engineering or bribery, and many privacy, bias, drift and poisoning issues trace back to them.
How much do human labels improve AI performance?
He said some experts put the gain at roughly 15 percent at most, with less value as AI systems improve.
How imbalanced is security data?
He said roughly 95 percent of data is benign and about 5 percent is a threat, and cited banking data with under 1 percent known fraud or money laundering.
How can you check an AI system with little human oversight?
He said teams can use literature on the distributions expected in their industry to set cutoffs and build unsupervised anomaly detection.
This article is based on remarks by Samuel Zaruba Smith, Director, Bank of America at Data Science Innovators Conference 2023 on April 12, 2023. More coverage: More AI Connect Events.
Filed under: Security & Fraud, AI Governance & Trust
