EVANSTON, Ill. — Northwestern University engineers are investigating new ways to tell what the hands are doing — including peering beneath the skin to watch muscles and tendons move. The new methods don't use high-tech cameras, medical scans or advanced imaging techniques. Instead, they use a common smartwatch.
In one project, engineers designed a new smartwatch system to glean activities from the arm and hands. In another, electrodes built into the back of a watch can track more subtle hand movements and gestures, such as a thumbs up or clenched fist.
By tracking these movements, researchers could one day help improve the control of bionic prosthetics or monitor patients in specialized care settings, including elder care. Beyond health care, the systems also could make smartwatches easier to use, much like facial recognition has simplified smartphone access, or extend to monitoring everyday activities, such as helping dental hygienists track how many teeth they clean in a week.
The researchers presented both studies, named TransfHAR and EITWatch , today (Nov. 3) at the Association for Computing Machinery's Symposium on User Interface Software and Technology (UIST). The full symposium will be held Nov. 2–5, 2026, in Detroit.
"Your hands are how you act on the world, but your smartwatch knows almost nothing about what they're doing," said Northwestern's Karan Ahuja, the principal investigator on both papers. "These two projects show how much the wrist alone can reveal — one by sensing the muscles and tendons beneath the skin, the other by learning how the arm moves. We can't recognize every movement yet, but it's a strong starting point."
Ahuja is the Lisa Wissner-Slivka and Benjamin Slivka Assistant Professor in Computer Science at Northwestern's McCormick School of Engineering , where he directs the Sensing, Perception, Interactive Computing and Experiences (SPICE) Lab .
TransfHAR was co-authored by Aidan Bradshaw , a Ph.D. student in the SPICE lab, Riku Arakawa of Carnegie Mellon University, and Xin Liu, a senior research scientist at Google Health Research. EITWatch was co-authored by SPICE lab members Xuanyou Liu and Novel Alam, who contributed equally and led the work.
Keeping watch of the hand
EITWatch takes advantage of a smartwatch's constant contact with the wrist. Using a technique called Electrical Impedance Tomography (EIT), eight small electrodes on the back of the watch send tiny electrical signals through the wrist to map the muscles and tendons beneath the skin.
Each hand gesture — from pointing a finger to giving a thumbs up — produces a distinct pattern as muscles and tendons move. By learning to recognize these patterns, EITWatch can identify specific gestures and translate them into smartwatch commands. A thumbs up, for example, could send a text message, while a pointed finger could activate a step-counter during exercise.
"The part I care about most is accessibility. Touchscreens shut out many people," Alam said. "A second input channel built into the device gives them another way in."
Because the same wrist signals encode what the hand intends to do, the researchers are also exploring whether this kind of sensing could help control prosthetic hands. Looking inside the arm also gives scientists a new way to track the hand, bypassing the need for cameras.
"Right now, the best way to tell what the hands are doing is a camera, and that's a privacy nightmare," Ahuja said. "It also only works when your hand is in view. The wrist is always there."
EITWatch is still a research prototype. In the study, it recognized six common hand gestures with over 92% accuracy and five subtler finger movements with over 91%, but it works best when calibrated for a single user. That means borrowing a family member's smartwatch, for example, might make it more difficult for the device to accurately recognize the new user's wrist movements.
"Going forward, we will focus on two things. For hardware, fitting the electronics inside the watch is the next push," Alam said. "Then there's the people problem. One person wearing the watch does way better. The goal stays simple though. Any wearer puts the watch on and the gestures work. No calibration step."
Watching the whole arm
Although today's smartwatches and smartphones can recognize certain gestures, those movements are typically limited to a predetermined set of motions. TransfHAR takes a more flexible approach. The system first learns from thousands of hours of public, unlabeled motion data from everyday activities such as walking and exercise. It can then learn an entirely new activity from just a few demonstrations. Users can then associate specific movements with customized commands, expanding what their devices can do.
"People with specialized activities, habits or routines aren't well represented in existing datasets. Activity recognition research often focuses on categories that are common across a population, but those don't always capture what matters to an individual," Bradshaw said. "I'm particularly interested in supporting those differences in how people live and move."
TransfHAR uses the smartwatch's built-in motion sensors to capture how the wrist accelerates and rotates and then attributes those motions to different tasks. The watch registers daily repetitive movements, personalizing them to each wearer. Where one user's arm movement may be painting a wall, another user may have a similar movement for brushing a client's hair.
Outside of the average person's daily use, medical providers like nurses in eldercare facilities could use the motions to track how often a patient repeats certain activities. This might include everyday actions for personal hygiene or more specialized movements, such as physical therapy exercises.
"In elder care, a nurse could passively see whether a patient brushed their teeth or shaved, without a camera in the room," Ahuja said. "At the gym, the watch could simply log which exercises you did."
TransfHAR works by using tools already inside smartwatches, such as accelerometers or gyroscopes, which measure how the wrist accelerates and rotates. The researchers used more than 2,700 hours of publicly available movement data to pretrain the system. Then, users can name a new activity, demonstrate it for the watch, and add it to their own set. In a study with 10 participants, the watch recognized seven new activities with about 90% accuracy after a single one-minute recording of each.
"With partners at Google, we're scaling up the data so the system learns more activities and gets more accurate," Ahuja said. "The same approach should carry over to other form factors, like rings and wristbands."
