Multisensory Trajectory Control at One Interaction Point, with Rhythm

We investigate navigation in two dimensions with velocity control, using a single-button interface. Users adjust the controlled-object speed through rhythmic tapping and its direction by pressing and tilting, releasing the button to finalize the rotation. Feedback to control action and object motion is provided by integration of multiple sensory modalities. Tactile pulses are delivered at 30-degree intervals during rotation, emulating the detents of a rotary encoder. Simultaneously, a sonic glissando accompanies rotation, thus rendering upward or downward rotation. Both visual and auditory cues are used to provide absolute positional feedback. Discrete notes are rhythmically played, whose pitch indicates vertical position, while stereo audio panning follows horizontal position. The rhythmic pace aligns with on-screen object speed, as dictated by tapping time intervals. Two studies were designed around a target-following task, under different sensory conditions. Study 1 has shown that target tracking can be effectively performed with multisensory rhythmic interaction. This is true also when the controlled object is intermittently hidden to view, although it was not possible to measure the advantage provided by auditory-tactile feedback. In study 2, no significant performance differences were observed between auditory and tactile conditions in situations of intermittent visual feedback, indicating that, if the two kinds of non-visual feedback are effective, they are essentially equivalent.