Current Topics in Privacy Seminar: Yaxing Yao

As AI agents increasingly act on users’ behalf, humans are shifting from directly interacting with systems to overseeing autonomous actions. Human oversight is often proposed as a safeguard for agentic AI: when an agent makes a mistake or encounters risk, a human can simply step in. But how effective is this oversight in practice? Speaker: Yaxing Yao Abstract: In this talk, I will present findings from our studies examining human oversight of LLM-powered GUI and computer-use agents. We first investigate how agents interact with manipulative interfaces, or dark patterns. We find that agents are themselves susceptible to manipulation, often prioritizing task completion over users’ privacy, safety, or interests. Human–agent collaboration also introduces new challenges related to attention, cognitive load, and limited control. We then systematically compare four common oversight strategies, ranging from risk-triggered escalation and plan-level supervision to per-action approval. We find that changing the oversight strategy substantially affects whether problematic actions arise, yet has surprisingly little effect on humans’ ability to stop them once they do. Users frequently receive control at the wrong moment or focus on whether the agent is correctly executing the task rather than whether the action itself is appropriate. Bio: Yaxing Yao is an assistant professor in the Department of Computer Science at Johns Hopkins University. His research lies at the intersection of human-computer interaction and human-centered privacy and security. His recent research explores privacy issues in user interactions with technologies and develops solutions that empower users to understand and control their privacy. He is also interested in privacy education for people from different demographic groups, such as children, families, and older adults. He has published in top human-computer interaction venues (e.g., CHI, CSCW, UIST) and privacy/security venues (e.g., the USENIX Security, S