[Editor's note: This article was written by Emily Nelson and published in IowaNow on Sept. 24, 2020.]
Technology developed at Iowa is used to develop artificial intelligence software that can detect potential safety risks ranging from weapons to slip and fall hazards
Technology born out of military-funded research in the University of Iowa’s Virtual Soldier Research Program (VSR) soon may be used in schools, hospitals, stores, and other places to improve safety.
A new company, MALUM Terminus Technologies Inc.—cofounded by Karim Abdel-Malek, professor of biomedical engineering and director of the University of Iowa Technology Institute; Dan Clay, dean of the UI College of Education; and Richard Ferguson, former CEO of the college testing company ACT—is paying UI for the right to use its technology to develop artificial intelligence (AI) software that analyzes security camera footage to detect potential hazards, such as spilled liquids on the floor or a person brandishing a weapon.
The platform is branded IntelliSee, but the company is developing other products—including using AI predict and prevent sports and work-related injuries.
A UI team led by Abdel-Malek initially created the technology to help predict soldier injuries and enhance soldier performance for the U.S. military. The goal was to minimize injuries experienced by U.S. Marines during training, particularly injuries related to the musculoskeletal system. The intent is to provide customized strength and conditioning programs for individual soldiers.
After a number of school shootings across the country in recent years, Abdel-Malek saw another potential use for the technology.
“I have three kids in school,” Abdel-Malek says. “The idea of recognizing a person with a gun in a school was a natural extension of being a concerned parent, so we began developing a solution for non-military uses. When we asked leaders and security officials at various schools, hospitals, and companies, they also saw the need and asked us to also consider other costly risks like slipping and falling. From this, we’re ultimately building a risk-reducing platform that identifies key hazards so that we can create a safer world.”
The UI team taught the AI software to recognize a shooting posture and different guns by feeding it thousands of hours of incident footage. This same capability is being used to recognize other hazards and will cover several risks. The software is paired with high-end graphical processing units (GPUs) and taps live surveillance feeds across multiple cameras in real time.
When the software detects a risk—like a drawn gun or shooting posture—the system instantly validates the risk and activates a customizable set of alerts that can drive proactive prevention or, if something is already occurring, faster response. Because seconds can be the difference between an incidence and a tragedy, the firm sends the risk, its location, and an image to designated officials—including first responders such as police—or to systems that can lock doors and sound alarms.
“A place like a school or university has several thousand cameras; all those cameras can’t be monitored by someone all the time,” Clay says. “Typically, how it works is if something bad happens, officials go back and watch. This technology watches those video streams live, and when it identifies something potentially bad, it initiates a response.”
The closest comparable technology on the market today uses sound sensors to detect when a gun is fired, Abdel-Malek says.
“It’s good technology, but it’s after the fact,” Abdel-Malek says. “We thought if we could shave seconds off knowing a gun was in the vicinity, we might be able to save lives.
Addressing safety from multiple angles
Abdel-Malek originally reached out to Clay to get his thoughts as an education professional, but what he didn’t realize was that Clay also worked for a short time as a police officer, giving him multiple perspectives. Clay says school shootings are an issue he’s given much thought to, through his professional life and as a father who worries about his children’s safety.
“The beauty of this is that it really does try to solve a significant problem in our society,” Clay says. “There are two important things when it comes to safety in schools: one is keeping kids and teachers safe, and the other is making them feel safe. If you don’t feel safe, you can’t learn. There are not many solutions that do both of these things. I think this technology will make people feel safe and actually will make them safer.”
He says he also appreciates the unobtrusive nature of the technology.
“It’s not a metal detector or armed guard, which don’t make all students feel safer,” Clay says. “In fact, they may make some feel less safe, particularly students of color.”
This technology wouldn’t necessarily replace other security measures, such as metal detectors or school resource officers, but Clay says depending on the particular needs of a school, it could play an important role in a school’s security strategy.
He also notes that it is a neutral solution when it comes to the Second Amendment.
“It doesn’t say who should or shouldn’t have guns, or if guns are good or bad,” Clay says. “It just identifies and tells you when someone has one.”
Preventing trips to the emergency room
Using the technology to identify slip and fall hazards arose out of interest from an Iowa insurance company. According to the U.S. Department of Labor, slips and falls account for more than 1 million emergency room visits each year. The insurance company wondered if instead of a weapon, the software could detect something like spilled milk in a grocery aisle. So, the Virtual Soldier Research team also taught the AI to recognize these types of hazards.
In some ways, this was more difficult.
“We needed it to know that this is a spill, not a mark on the floor,” Abdel-Malek says. “This is a spill, but this is a reflection from the window.”
Abdel-Malek says he expects schools, hospitals, and office buildings to be most interested in the weapon detection feature, while big-box stores and warehouses might find the slip mitigation feature to be the most useful. The firm is building out a platform to help mitigate multiple risks given potential customers have similar needs.
“The environment at Iowa is such that Karim just picked up the phone and called me. At a lot of universities, you wouldn’t just call the dean of another college. But here, that’s just a part of the culture. There’s something special here that makes it more likely that people who bring value to an idea like this are going to find each other.”
—Dan Clay, dean of the University of Iowa College of Education
Potential future uses for technology
The Virtual Soldier Research team is working on additional uses of the artificial intelligence technology that MALUM Terminus Technologies Inc. has licensed and is working toward leveraging.
One would be used to identify potential physical abuse, which Abdel-Malek says he can see being used in care facilities such as nursing homes or in schools to detect fights or bullying. For this, the AI is being taught to recognize certain postures or movements that may indicate assault or abuse and by monitoring the limbs’ velocity and accelerations. An alert could then be sent indicating that the recording from the camera at that time and location should be reviewed.
They also are looking to expand the initial use of the technology, which tracked various aspects of soldiers’ lives to predict and prevent injuries, to be used with athletes.
“For example, in one study we found that the three main components that were most influential in causing an injury were stress, lack of sleep, and load,” Abdel-Malek says. “This research is maturing fast and we can see it in the future being useful in athletics, especially in pro sports.”
Finally, they want to use the AI to track behavior when it comes to motion, such as walking.
“We think how one walks is a signature, like your fingerprint,” Abdel-Malek says. “Could that be used as another biomarker at security checkpoints, like an airport?”
Putting Iowa collaboration to work to keep Iowans safer
Abdel-Malek says his team’s collaboration with people from across campus, such as the UIPD and Clay in the College of Education, has become commonplace and it makes research projects stronger.
“It used to be just people from my team, but now we involve people from everywhere,” Abdel-Malek says. “I find the best ideas come when engineering, computer science, business, medical, education—you name it— come together and talk. I’m so glad we have this incredible team right here on campus.”
Clay says Iowa’s culture of collaboration is unique.
“People collaborate across disciplines on every campus in America, but the community here is different,” Clay says. “We’re a little smaller so we see each other and interact with each other much more frequently. The environment at Iowa is such that Karim just picked up the phone and called me. At a lot of universities, you wouldn’t just call the dean of another college. But here, that’s just a part of the culture. There’s something special here that makes it more likely that people who bring value to an idea like this are going to find each other.”
Abdel-Malek and Clay say they hope that the research done at Iowa and the related products their company is bringing to the market will keep more people out of harm’s way, particularly in schools.
“This is a scalable opportunity to solve what I think is one of our country’s biggest problems in schools right now,” Clay says. “My ultimate goal is to make every school in our state the safest school in the country. I think Iowa can become a national leader for safe schools—both for kids and teachers feeling safe and being safe. It could become the Iowa model for school safety.”
UI research impacts Iowa’s economy
The University of Iowa has long been an epicenter of groundbreaking research. From engineering and astrophysics to the digital humanities and medicine, research done on campus improves and changes lives.
Research activities also impact the state economy by employing people and making purchases for equipment, supplies, and services. They facilitate new knowledge creation in Iowa through inventions, patent applications, and licenses.
UI’s research spending generated $417 million in added income for the Iowa economy in fiscal year 2017-18. Research done at the university’s Iowa Technology Institute alone has led to at least 12 spinoff companies.
“It’s important that the investment our state, our federal government, and our students make in our higher education institution results in improving the lives of people in our country and driving economic development,” says Dan Clay, dean of the UI College of Education and MALUM Terminus Technologies Inc. cofounder.
MALUM Terminus Technologies Inc. (MTTI) currently employs seven people and recently secured a large investment from their capital partners The Asymmetria Group, an Iowa-based investment firm. Through this funding, MTTI is licensing and further developing technology born out of military-funded research in the University of Iowa’s Virtual Soldier Research Program (VSR) and bringing it to market.
The first is their IntelliSee AI (artificial intelligence) risk mitigation platform that autonomously monitors live surveillance feeds to detect a range of potential hazards from the rare drawn weapon to common slip and fall hazards, such as spilled liquids on the floor and more.
“This is research that leads to life-changing behavior,” says Karim Abdel Malek, UI professor of biomedical engineering, director of the Iowa Technology Institute, and MALUM Terminus Technologies Inc. cofounder. “We can make a difference out there. We’re creating jobs. We’re creating Iowa-based technology that has an impact.”