New AR System Alters Sight, Sound and Touch

There’s the typical pitter-patter sound that comes from a person drumming their fingers along a tabletop. But what if this normal pitter-patter sound was perceived as a series of hollow echoes? Or rolling thunder? A new augmented reality system called Tactile Echoes provides users with experiences like this, and could be used for a wide range of gaming, entertainment and research purposes.

Notably, the system does not require any equipment between the user’s fingertips and the contact surface, meaning users can enjoy the real sensation of their environment along with the visual, haptic and auditory augmented enhancements.

“Tactile Echoes is possibly the first programmable system for haptic augmented reality that allows its users to freely touch physical objects or surfaces augmented with multimodal digital feedback using their hands,” says Anzu Kawazoe, a PhD candidate at the University of California, Santa Barbara who co-designed Tactile Echoes.

It accomplishes this using a sensor that is placed on the top of the user’s fingernail, which detects the vibrations that are naturally produced within the finger as it touches a surface. The vibrational signals are processed and translated into programmed sounds.

Different tactile feedback and sounds can be played with each interaction, because the vibrational patterns in our fingers change depending on what surface we touch, or the intensity of pressure applied. For example, Tactile Echoes may play a light, fun echo when you tap an object lightly, or play a sudden thud if you jab the object with force.

“We were motivated by the idea of being able to almost magically augment any ordinary object or surface, such as a simple wooden table, with lively haptic and effects that playfully respond to how or where we touch,” explains Kawazoe.

Her team took the system one step further by integrating the wearable device with virtual environments created by a smart projector or a VR or AR headset. In this way, users can “touch” virtual objects in their real environment, and experience enhanced graphic, sound and haptic feedback.

The researchers tested Tactile Echoes through a number of user experiments, described in study published May 26 in IEEE Transactions on Haptics. First, study participants were asked to describe different sounds, and what perceptions and associations are evoked with each one. In a second experiment, participants used Tactile Echoes to complete an interactive, augmented reality video game that was projected onto an ordinary desktop surface. Users reported that the Tactile Echoes feedback greatly enhanced the responsiveness, and their level of engagement and agency in playing the game. [READ MORE]