Tactile and Sensory Interactions

Latest papers

• Small yet Assistive: Spatially-Aware Post-Training for Low Vision

  Rishabh Choudhary, Shreyansh Raj, Umesh Goyal, Shubh Kashyap, Shrestha Kumar, Sushovan Jena, Komal Kumar, Hisham Cholakkal, Aditya Nigam · 25 September 2026

  An estimated 1 billion people worldwide live with vision impairment, yet current vision-language models (VLMs) produce descriptions too vague for safe navigation by blind and low-vision (BLV) users. Large VLMs can generate high-quality audio-description-compliant narrations but cannot run on mobile …

• An Affordable AI-Integrated Smart Cane for Multimodal Mobility Assistance of Visually Impaired Users

  Ali Akarma, Adeel Ahmad, Toqeer Ali Syed · 22 September 2026

  Visual impairment affects over 2.2 billion people worldwide, yet conventional white canes cannot detect elevated hazards or provide semantic environmental context. Existing AI-assisted navigation systems typically rely on expensive hardware or cloud connectivity, limiting accessibility in resource-c…

• Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments

  George Xi Wang, Xiangyu Li, Shaoyue Wen, Jiaqian Hu, Junan Xie, Yupeng Wang, Ziyue Shi, Qijun Chen, Maaike Bouwmeester, Yuhua Jin, Jing Qian · 21 September 2026

  Blind and low-vision users often face challenges when locating and physically acquiring objects in unfamiliar indoor environments. Existing vision-language-model-based assistants can provide semantic descriptions but may introduce latency, hallucinations, and guidance that is poorly aligned with emb…

• VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation

  Congsheng Xu, Qiaochu Yang, Fangyuan Shi, Yifan Han, Baijun Chen, Yiming Wang, Haonan Zhao, Zhe Liu, Yao Mu, Daolin Ma, Xiaokang Yang, Hesheng Wang · 18 September 2026

  We propose VT-MUSE, a Multimodal Unified SEquential representation learning framework for visuotactilemanipulation. Existing approaches often encode visual and tactile observations independently before fusion, limiting their ability to capture fine-grained cross-modal dependencies. Moreover, most me…

• Beyond Scene Description: Multi-Agent Orchestration for Non-visual Access to Virtual Worlds

  Toqeer Ali Syed, Ali Akarma, Adeel Ahmad, Danial Hameed · 15 September 2026

  Virtual worlds now host classrooms, meetings, conferences, shops, and social venues, and nearly every interaction they expose assumes a user who can scan a three-dimensional scene, follow avatars, and read floating panels. Blind and visually impaired (BVI) users are left with assistive tools that ea…

• Assisted Spatial Cognition Through Vision-Language Models

  H. Riaz, J. B. Fernandez, I. Mills, D. Hickey, F. Cleary, M. I. Ali · 14 September 2026

  Multimodal AI, powered by Large Language Models (LLMs) and Vision-Language Models (VLMs), is transforming assistive technologies by enabling simultaneous processing of visual and textual data. This advancement holds significant promise for over 43 million visually impaired and neuro-divergent indivi…

• Tact: A Zero-Cost, Browser-Based Pipeline for On-Demand Tactile Braille Storybooks

  Iliano Fasolino · 14 September 2026

  Braille literacy among blind school-age children has fallen sharply, in part because producing illustrated braille pages still requires specialized software and trained labor. We present Tact, a browser-based pipeline that converts a spoken or typed story idea into printable braille with a matching …

• MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places

  Jason Armitage, Ioannis Tsochantaridis, Linda Mazzone, Chuqiao Yan, Srini Narayanan, Sarah Ebling · 31 August 2026

  We introduce MAP, the first benchmark to evaluate multimodal AI systems as assistants for users with accessibility requirements when planning visits to places in the real world. In our evaluation, systems are presented with requests to verify or recommend a point of interest meeting an accessibility…

• BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Models

  Jinghan Zhang, Fengran Mo, Zhiyu Chen, Xiaoyan Han, Kunpeng Liu, Chang-Tien Lu · 28 August 2026

  Although Large language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in the same way. However, it is unclear whether existing AI systems are inclusive enough for blind and deafblind users to access the same functionality …

• ForeSightGuide: An Anticipatory Framework toward Accurate and Low-Redundancy Guidance for the Visually Impaired

  Zhiyuan Wang, Xu Li, Shikang Guo, Wei Meng, Quan Liu, Jie Zuo · 20 August 2026

  Electronic travel aids are pivotal for the independent mobility of the visually impaired. While Vision-Language Models (VLMs) offer rich environmental understanding, they often suffer from excessive false positives in dynamic scenarios, leading to cognitive overload. To address this, we present Fore…

• A Systematic Survey on Image Description Techniques for STEM Domains

  Marco Cardia, Letizia Angileri, Marina Buzzi, Giulio Galesi, Barbara Leporini · 27 July 2026

  The proliferation of visual data in Science, Technology, Engineering, and Mathematics (STEM) fields presents accessibility barrier for individuals with blindness or visual impairments. While recent advances in Artificial Intelligence (AI) offer new opportunities to generate textual descriptions of S…

• Safety-oriented sidewalk and road segmentation for smartphone-based assistive navigation

  Hakan Calim, Anamaria Dumitrescu, Adarsh Bhandary Panambur, Huzaifa Asif, Andreas Maier · 24 July 2026

  Independent sidewalk mobility is essential for blind and visually impaired pedestrians (BVIPs), yet smartphone-based assistive navigation requires perception models that distinguish walkable sidewalks from adjacent unsafe regions. This study presents a safety-oriented semantic segmentation framework…

• VisionAssist: An Open-Source Smartphone Assistant for AI-Based Visual Accessibility

  Ayşe Özlem Çalışkan, Jordi Sanchez-Riera · 21 July 2026

  People with low vision often face challenges in performing everyday tasks that require interpreting visual information. We present \textbf{VisionAssist}, an open-source mobile application designed to improve independence by providing AI-powered visual assistance through a smartphone. The application…

• Design and Validation of a Lightweight 1D CNN for Affective Touch Classification in Soft Plush Companions

  Aleksandrs Vali\v{s}evskis, Aleksandrs Okss, Inese T\=i\c{g}ere, Aleksejs Kata\v{s}evs, Dina Bethere, Anete Hofmane, Airisa \v{S}teinberga, Und\=ine Gavri\c{l}enko, Santa Me\c{l}\c{k}e, Lucie Matou\v{s}kov\'a · 21 July 2026

  Soft, sensorized companions offer a physically safe and emotionally intuitive interface for socially assistive technologies, yet their deformability and multichannel tactile sensing complicate the robust interpretation of human affect. This study presents a complete open-source MATLAB-based framewor…

• VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance

  Yunfeng Liu, Yuandong Yang, Jiarui Han, Zhenpeng Huang, Yuqing Tang, Xiangyu Zeng, Gangshan Wu, Limin Wang · 17 July 2026

  Visually impaired individuals (VIIs) encounter significant daily challenges due to limited access to visual information. Although Multimodal Large Language Models (MLLMs) have achieved impressive results on general vision and language tasks, their practical utility in real-world blind assistance sti…

• Skeleton: Visual Authoring of Non-visual Data Experiences

  Frank Elavsky, Chieri Nnadozie, Lucas Nadolskis, Patrick Carrington, Dominik Moritz · 17 July 2026

  When sighted practitioners author accessible data visualizations, they build navigation structures (the nodes, edges, and input bindings that govern how assistive technologies traverse an interface) entirely in code, with no visual representation. Without a representation to react to, practitioners …

• Entropy in Semantic Memory Navigation in Blind and Sighted Individuals: The Effect of Visual Experience

  Felipe D. Toro-Hernández, Rodrigo Lagos, Sergio E. Chaigneau · 15 July 2026

  Embodied accounts of semantic memory highlight the role of sensorimotor systems in acquiring and storing knowledge. Congenitally blind populations offer a critical test bed for these assumptions, providing an opportunity to assess whether conceptual grounding requires visual experience. In this stud…

• Text-based Tactile Graphics Generation for the Visually Impaired

  Ruihan Gao, Joonghyuk Shin, Ava Pun, Jaesik Park, Wenzhen Yuan, Jun-Yan Zhu · 13 July 2026

  Tactile graphics are a primary medium for blind and low-vision (BLV) individuals to access non-textual information. However, they are difficult to scale or personalize. While recent generative models have revolutionized visual content creation, they are optimized for screen-based visual realism and …

• Video-Based Detection of squint and cataract for accessibility-aware adaptive web interface rendering

  Amar Ranjan Dash, Manas Ranjan Patra · 9 July 2026

  Squint and cataract are major ocular disorders that majorly affect visual perception and interaction capability. This paper proposes a real-time video-based automated detection system for squint and cataract detection based on computer vision and image processing methods. The proposed system uses a …

• OpenGlass: A Sensing-Computing Split Architecture for Local MLLM-Driven Real-Time Visual Assistance

  Mengzhang Li, Yuan Yao · 7 July 2026

  We present OpenGlass, an open-source, privacy-oriented, local-first system for low-latency multimodal visual assistance, with a primary focus on blind and low-vision users. Cloud MLLM assistants offer strong visual understanding, but often require uploading first-person visual data and can suffer mu…

• TacReasoner: A Dynamic Tactile-Language Framework for Interactive Reasoning in Real-World Scenarios

  Kailin Lyu, Di Wu, Long Xiao, Jianning Zeng, Jianwei He, Chang Lin, Lianyu Hu, Lin Shu, Jie Hao, Ce Hao · 7 July 2026

  Among the five primary human senses, tactile is arguably the most fundamental to survival, as it enables the perception of physical contact and interaction in real-world environments. In this paper, we explore two key challenges of integrating tactile sensing into intelligent systems for multimodal …

• VisionAId: An Offline-First Multimodal Android Assistant for People with Visual Impairment, Featuring Personalized Object Retrieval

  Cristian-Gabriel Florea, Stelian Sp\^inu · 3 July 2026

  Over 285 million people worldwide live with a visual impairment, for whom everyday tasks such as avoiding obstacles, locating personal belongings, recognizing familiar faces, or handling cash remain persistent obstacles to personal autonomy. Existing assistive applications are typically limited to r…

• Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

  Yoonhyung Park, Minji Kim, Sungwon Moon, Jiyoung Lee · 2 July 2026

  Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of…

• Making Sense of Touch from the Child's View for Contrastive Learning

  Max Whitton, Zecheng Wang, Puchen Liu, Quang Tuan Truong, Shengao Wang, Manaswi Yadamreddy, Oktay Ozel, Visista Jayanti, Saniya Sekhon, Hanna Samuel Tadesse, Lawrence Miao, Junjie Wang, Jiasen Lu, Chen Yu, Boqing Gong · 1 July 2026

  Is the sense of touch a mechanism for human babies' learning of visual concepts? If so, can we quantify its importance, and to what extent do babies rely on their sense of touch for visual learning? To approach these questions in a principled way, we propose a structured coding system for baby-centr…

• Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation

  Shengqi Xu, Guojin Zhong, Yang Liu, Fanjie Wang, Hu Luo, Hanyu Zhou, Weiyao Zhang, Ziyi Ye, Zuxuan Wu, Yu-Gang Jiang · 30 June 2026

  Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation. These sensors achieve high spatial resolution and multi-dimensional force sensing by utilizing an internal camera to monitor the deformation of their elastic gel surface, thereby indirect…