Training artificial intelligence to enforce even seemingly straightforward rules - like balls and strikes in Major League Baseball (MLB) - is a messy, dynamic process that takes time and careful evaluation of the technology in the wild, according to new Cornell research.
Drawing on their previous work with MLB and its newly implemented Automated Ball-Strike (ABS) system, Cornell researchers in AI and ethics argue that multiple stakeholders must evaluate AI-driven rule enforcement technologies not only for technical accuracy but also in real-life contexts within organizations.
The implications for the research go far beyond home plate. Organizations and governments are increasingly relying on AI-driven technologies to enforce rules in high-stakes areas like criminal justice, policing and labor practices.
"Automating baseball's strike zone seems straightforward: There's a strike zone defined in the official baseball rules. Yet it took MLB seven years and multiple iterations to figure it out," said Andrea Wen-Yi Wang, a doctoral student in the field of information science. "Our question was, 'Why was it so hard?'"
She is the lead author of "Inside Baseball: The Automated Ball-Strike System as an Object Lesson in Technological Rule Enforcement," which she and her co-authors, from the Cornell Ann S. Bowers College of Computing and Information Science, presented at the ACM Conference on Fairness, Accountability, and Transparency (FaCCT) held June 25-28 in Montreal.
In the paper, researchers highlight the inherent gap or "distance" between an existing rule and its technological implementation. In the case of MLB, this distance exists due to the technical constraints of ABS, which uses a combination of Hawk-Eye cameras and computer vision to challenge an umpire's ball or strike call. The distance in the case of MLB is also due to the need to maintain the economy, stability and art of baseball, all of which multiple stakeholders require in order to buy into the use of ABS, the researchers said.
"Umpires do a lot of translations of the strike zone, and those translations are not as clear until you try to use technology to automate them … . Even clear definitions and rules require complex translations because the system has to sit in an existing practice with many stakeholders," said Wang. "Finding a trade-off between these values is a messy process that we typically overlook. It involves a lot of understanding and conversations, and you need to be willing to devote a lot of time before you push out these technological systems."
If MLB took seven years to ready ABS for the big leagues, similar care and caution should be shown when implementing any new technology system, whether it's enforcement technology or generative AI in education, she added.
Through their analysis, researchers challenge the historical approach to evaluating enforcement technologies. They said the technologies must be evaluated based on how they're experienced in practice, within organizations, not solely on how well their outputs match defined targets.
"Early on, MLB learned that nobody liked the automated ball-strike system when it enforced the strike zone per the rule book," Wang said. "Those insights are only revealed through testing and evaluation in practice."
Along with Wang, the paper's coauthors are: Waki Kamino, a doctoral student in the field of information science; David Mimno, professor and chair of information science; Karen Levy, associate professor of information science; and Malte Jung, associate professor of information science and the Nancy H. '62 and Philip M. '62 Young Sesquicentennial Faculty Fellow, all in Cornell Bowers.
This research was partly supported by the National Science Foundation and the Canadian Institute for Advanced Research.
Louis DiPietro is a writer for the Cornell Ann S. Bowers College of Computing and Information Science.
