TactiSense: Haptic Companion for DeafBlind Navigation & Communication

TactiSense is a revolutionary mobile app paired with a custom haptic glove, designed specifically for the DeafBlind community. The app harnesses smartphone sensors—camera, GPS, microphone—to interpret the environment. Speech is converted to text via offline speech-to-text, then mapped to a set of haptic patterns (e.g., finger taps for letters, directional vibrations for navigation). The glove contains ten linear resonant actuators arranged on the fingers and palm, each delivering distinct vibrations. Users learn these patterns through a built-in tutorial and can customize them. The system also provides haptic alerts for obstacles (using real-time object detection) and public transit updates. Offline-first architecture ensures privacy and reliability. The project followed a participatory design process, with DeafBlind individuals co-creating the patterns and form factor. TactiSense is open-source, inviting community contributions. It envisions a world where DeafBlind individuals can independently explore and communicate, breaking through double sensory isolation.

The DeafBlind community faces extreme isolation: existing communication methods (e.g., tactile signing, braille) require proximity or sighted assistance. Navigation aids are bulky (e.g., white canes, guide dogs) or expensive. There is no affordable, portable, all-in-one device that combines real-time communication and navigation using touch alone. TactiSense addresses this by leveraging ubiquitous smartphones and a lightweight glove, turning touch into a rich information channel.

• Real-time haptic translation of speech to tactile patterns (e.g., finger positions for letters/phrases)
• Haptic navigation guidance using vibration intensity and direction (e.g., left finger for turn left)
• Customizable pattern library: users can assign their own meanings to vibration sequences
• Integration with public transit APIs for route info and stop alerts
• Battery life of at least 8 hours on the glove, rechargeable via USB-C
• Companion app for pattern creation, sharing, and firmware updates
• Offline operation: no internet required for core features (speech-to-text, navigation, object detection)

• Not a replacement for human interpreters or tactile signing; it augments but does not replace direct human interaction
• No visual output: the interface is exclusively haptic, with optional audio cues for sighted assistants
• Not a medical device; not certified for critical safety (e.g., avoid it solely relying on for traffic crossing)
• No cloud dependency: all processing happens on-device to ensure privacy and low latency
• Not a standalone solution: many tasks still require a sighted guide or assistive technology

• Smartphone: iOS/Android with BLE, camera, GPS, accelerometer
• Glove hardware: Custom PCB with BLE module (nRF52840), 10 haptic actuators (Linear Resonant Actuators, 12mm), LiPo battery, accelerometer, buttons for mode switching
• Firmware: C++ on the glove, RTOS for real-time haptic commands
• Mobile app: React Native (cross-platform) with native modules for BLE, camera, and sensors
• Speech-to-text: Coqui STT (offline, on-device) with custom models trained on clear speech
• Object detection: YOLOv8 (Tiny) optimized for mobile via TensorFlow Lite
• Navigation: OpenStreetMap + OSRM for routing, offloaded to smartphone GPU
• Haptic engine: Custom library to map patterns to actuator sequences (e.g., 200ms buzz on index finger = 'yes')

The system has two main components: the haptic glove and the smartphone app. The glove communicates via BLE with the app, sending sensor data (accelerometer, button presses) and receiving haptic command arrays. The smartphone app runs a pipeline: (1) Input capture: microphone for speech, camera for objects, GPS for location. (2) Processing: offline speech-to-text (Coqui), object detection (YOLO), navigation route parsing (OSRM). (3) Haptic mapping: a decision engine converts processed data into haptic patterns using a user-defined dictionary (e.g., ‘hello’ = three short buzzes on pinky). The engine considers context: in navigation mode, vibrations indicate direction and distance to next turn. (4) Rendering: pattern is sent as a six-byte command to the glove via BLE, specifying actuator ID, duration, intensity, and interval. The glove firmware interprets these and drives the actuators with PWM. Users can switch modes (communication, navigation, alert) via a button on the glove or the app. The app also includes a pattern editor for customization, which syncs to the glove’s memory. All data stays on-device; only pattern sharing uses local Wi-Fi P2P or Bluetooth.

• Steep learning curve: Users must memorize many haptic patterns. Mitigation: gamified tutorial and gradual introduction.
• Glove battery life: 8 hours may not cover a full day. Mitigation: hot-swappable battery pack.
• Object detection latency: Real-time detection on mobile may cause delays. Mitigation: use dedicated NPU or reduce frame rate.
• Offline speech accuracy: Coqui STT struggles with accents or noisy environments. Mitigation: allow users to train custom voice models.
• Hardware comfort: Glove must fit various hand sizes without causing fatigue. Mitigation: adjustable straps and ergonomic design with community feedback.
• Adoption cost: Glove production may be expensive. Mitigation: open-source hardware, DIY kit option.

• What is the optimal number of haptic actuators for distinguishing patterns without confusion?
• Should directional navigation use intensity (stronger = closer) or distinct finger patterns (e.g., thumb for north)?
• Can we use the smartphone’s built-in haptic engine (e.g., Taptic Engine) as a fallback for users without the glove?
• How to handle multi-word speech translation? Do we use sequential finger tapping or overlapped vibrations?
• Should the glove include a small braille-like display (e.g., refreshable pin array) for high-priority inputs?

The project required deep collaboration across hardware engineering, firmware, mobile development, and accessibility research—a team that never fully assembled. Early interest faded when grant funding fell through, and the complexity of prototyping the glove proved too much for a spare-time side project. It remains a detailed plan and a dream, waiting for the right team to pick it up.

This app is designed in close partnership with DeafBlind individuals from the Helen Keller National Center. All patterns were co-created during workshops. The hardware is open-source, and the app is free to use. We encourage the community to fork and improve it.