![[2602.15707] Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.15707] Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU
Summary
This article presents a novel real-time conversational assistant that utilizes audio and IMU data to guide users through procedural tasks, enhancing efficiency and privacy.
Why It Matters
The development of a proactive conversational assistant that relies on lightweight, privacy-preserving modalities is significant as it addresses computational overhead and privacy concerns associated with traditional video-based systems. This innovation can improve user experience in task execution while ensuring data security.
Key Takeaways
- Introduces a real-time conversational assistant using audio and IMU inputs.
- Implements a User Whim Agnostic (UWA) finetuning method to enhance communication efficiency.
- Achieves over 30% improvement in F-score and 16x speedup in processing.
- Demonstrates the potential for edge device implementation without cloud dependency.
- Addresses privacy concerns by eliminating the need for video input.
Computer Science > Multimedia arXiv:2602.15707 (cs) [Submitted on 17 Feb 2026] Title:Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU Authors:Rehana Mahfuz, Yinyi Guo, Erik Visser, Phanidhar Chinchili View a PDF of the paper titled Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU, by Rehana Mahfuz and 3 other authors View PDF HTML (experimental) Abstract:Real-time conversational assistants for procedural tasks often depend on video input, which can be computationally expensive and compromise user privacy. For the first time, we propose a real-time conversational assistant that provides comprehensive guidance for a procedural task using only lightweight privacy-preserving modalities such as audio and IMU inputs from a user's wearable device to understand the context. This assistant proactively communicates step-by-step instructions to a user performing a furniture assembly task, and answers user questions. We construct a dataset containing conversations where the assistant guides the user in performing the task. On observing that an off-the-shelf language model is a very talkative assistant, we design a novel User Whim Agnostic (UWA) LoRA finetuning method which improves the model's ability to suppress less informative dialogues, while maintaining its tendency to communicate important instructions. This leads to >30% improvement in the F-score. Finetuning the model also results in a 16x speedup by e...
