Psychophysics of multisensory-motor interaction

The general aim of the “Psychophysics of Multisensory-motor interactions” group is to evaluate human action-perception behaviors in situations where the human agent is interacting with a multimodal interface. Specific objectives are to (i) characterize the role of the haptic modality (tensor, force and touch) in the perception of external objects and body properties, (ii) model cross-modal relations in perception and action, (ii) investigate their consequences for sensory deficit and substitution, (iii) in order to provide guidelines to designers of enactive human-machine interfaces.

In 2005, the group — composed of eleven institutions and research centers decided to collaborate on a limited number of joint and enactive projects started in the first two year of the network. The research concentrated on psychophysical studies that are (i) multimodal in essence, (ii) enactive in the sense that the informational basis for perception and movement control is rooted in the interaction between the observer and the environment, and (iii) mediated by technological interfaces.

The main achievements are now detailed. For the main projects, the theoretical context and the experimental procedure are quickly addressed, together with the main results obtained. Specific recommendations to designers of enactive interfaces are highlighted, as an important outcome of our scientific and technological activities. More detailed aspects about each project can be found in the disseminated actions as well as in the deliverable D.RD2.1.2 delivered to the EU on June 2006 (www.enactivenetwork.org).

Haptic perception of elasticity

INPG - UPPSALA

Elasticity is a very basic experience; it is already perceived at very young age by infants (80-126 years old: [Walker et al., 1980]), and can be evaluated either from visual information (for example: looking at a ball bouncing: [Warren et al., 1987]), or from the pitch of the sound produced by this object (psychoacoustics), or from the manipulation of the object. The aims of the project were to study 1) to what extent observers can pick up in the haptic stimulation only the information about the elasticity of an object when it is squeezed through a tunnel by the observer (see Figure 1), in particular when the relation between the elastic force and the movement produced by the observer is non linear, and 2) the contribution of visual information when it is added to the haptic information.

Observers were presented with objects of different stiffness that they squeezed through a narrow tunnel with the task of judging their elasticity. The CORDIS-ANIMA modelling system developed by INPG, a particle-based physical modelling system using the mass-interaction paradigm, was used to simulate the deformable object. The main result was that observers could efficiently pick up the haptic information when they manipulated the object (see Figure 2). The rather good performance (measured in terms of judging elasticity on a relative scale) did not change significantly when visual information was added. In conclusion, the human haptic system can provide adequate information about object (relative) elasticity without visual assistance.

Figure 1: Snapshots from the visual scene showing the paste object passing through the tunnel (from left to right and top to bottom)	Figure 2: Mean (and SD) perceived elasticity, as a function of stiffness

Recommendation to designers:

• Haptic displays allowing observers to handle virtual objects can be efficiently used to provide information about object properties such as elasticity.
• Small changes in simulation frequency and time lag between inputs and outputs modify the nature of the simulated objects, inducing the enaction of different objects.
• With multisensory interfaces, physical consistency between haptics and vision specifies the agent’s behavior. Literature presents experiments where the lack of consistency produced a visual dominance; on the contrary, in this experiment, the presented consistency led to identical results with or without visual information.

Key reference:
Couroussé, D., Jansson, G., Florens, J.-L., & Luciani, A. (2006). Visual and haptic perception of object elasticity in a squeezing virtual event. In Proceedings of the EuroHaptics 2006 Conference (pp. 283–288). Paris (France) : July 3-6 2006.

Tactile – Force-feedback synergy

INPG - COSTECH

This research project deals with the exploration of a possible synergy between tactile and force-feedback information. The question of the precise role of the tactile dimension in haptic perception is particularly essential for the design of haptic interfaces. Using the Tactile - Force Feedback device (T-FFD) we have started to explore two major questions having implications both in grip control (prehension) and in exploration (perception of spatial irregularities):

• What role plays the deformability of the exploratory body in prehension/exploration.
• Can tactile information overcome the “one-point” characteristic of FFD combining spatially distributed tactile information with one-point force interaction and approximate the spatial distribution of forces in real situations?

The experiments conducted in 2006 concerned the exploration of spatial irregularities (see Figure 1). They were conducted using the TactPHANToM device (PHANToM FFD equipped with the standard stylus, coupled with tactile Braille cells (array of 4*4 piezzoelectric pins) mounted in a box attached to the stylus). The scene was composed of a virtual bridge and of an avatar (exploratory body) controlled by FFD; the avatar’s collisions with the bridge gave rise to tactile and/or force information. Blindfolded participants were required to follow the bridge and to maintain contact with it in 3 different experimental conditions: [P] force feedback but no tactile information; [T] tactile feedback but no force feedback; [PT] both force and tactile feedback.
The main results show that participants were able to use tactile information to efficiently guide their displacements. Tactile information improved the performance, i.e., decreased the number and duration of losses during contour following. It offered the ability to anticipate the location of the edges (exteroceptive function); it also played a proprioceptive role as it offered to perceive the direction of self-displacement relative to the edges of the bridge.

Figure 3. The tactPHANToM (left), the task of following the virtual bridge (middle), and the main results (right).

Recommendation to designers:
Synchronisation and coherence between tactile and force feedback is of a particular importance, allowing a coherent information both on the environment and on the performed movement.

Key reference:
Declerck, G., & Lenay, C. The role of tactile augmentation of a PHANToM FFD studied on a task of goal oriented displacement. 2nd Enactive Workshop – Montréal (Canada) : 25-27 May 2006.

Postural exploration of the visual space

UM1 – HFRL

Human postural coordination in stance exhibits spontaneous coordination modes: in-phase motion between ankles and hips for low frequencies / small amplitudes of body movements (ankle-hip relative phase, ørel » 20°), and anti-phase motion for high frequencies / large amplitudes of body movements (ørel » 180°). These two postural coordinative states — together with other self-organized properties such as bifurcation, hysteresis, critical fluctuations, and critical slowing down — emerged out of a virtually infinite combination of the many degrees of freedom involved in the accomplishment of supra-postural tasks (Bardy et al., 2006). In the ENACTIVE framework, a technological biofeedback device, Enactive Posture (Figure 1), was developed. This device allows the exploration, the learning, and the rehabilitation of all possible antero-posterior ankle-hip patterns during stance. With Enactive Posture, the participant is required to produce a particular relative phase pattern (in the figure a relative phase of 90° between ankles and hips) in the Lissajous plane (ankle angle vs. hip angle plot). Data from the two goniometers fixed on body joints (ankle and hip) can be used to generate real time visual feedback in the same ankle–hip configuration plane. Participants are instructed to perform “body-drawing”, i.e., to reproduce by moving their body the requested shape. Performance can be quantified by root mean square value between requested and performed relative phase.

Figure 4. Enactive Posture, a customized VR system devoted to the (re-) learning of human postures.

In 2005, this device was used to discover the dynamics of the entire postural repertoire of healthy humans, evidencing the location of postural attractors and repellors (Figure 2). In 2006, Enactive Posture was introduced at the Grau du Roi post-stroke rehabilitation center in the Montpellier region (France). Stroke patients (N = 12) exhibited a different postural dynamics with the disappearance of the 0° in phase pattern together with a loss of stability for the anti-phase pattern in comparaison with the healthy population.

Recommendation to designers:
Enactive Posture provides (i) multimodal (proprioceptive and exteroceptive) and (ii) movement-based information about postural coordination, (iii) is an interface between the (real) performer and the (virtual) environment (virtual shapes to draw/reproduce), (iv) has many applications including Aging (daily postural training programs), Rehabilitation (of spontaneous postures following strokes), Sport and Ergonomics (learning new postures with biofeedback), Video games (additional degrees of freedom), and Art (‘Body drawing’ - body / pencil).

Figure 5. Produced (Y) as a function of required (X) postural relative phase (°) showing attractors and repellors (red: Y=X) in healthy subjects

Key reference:
Faugloire, E., Bardy, B. G., Merhi, O., & Stoffregen, T.A (2005). Exploring coordination dynamics of the postural system with real-time visual feedback. Neuroscience Letters, 374, 136-141.

Perceiving whether an object is within reach using active exploratory movements

UM1 – HFRL – DEI – MPI

People often exhibit great skill in detecting their own action capabilities and especially, in our case of interest, in detecting whether an object is within or without reach (Carello et al., 1989), even when they are only 5-month old (Yonas & Hartman, 1993). What is the basis of this knowledge? In considering this problem, we might consider not only optics (or acoustics) but additional perceptual modalities. This option arises when we take into account the fact that the observer’s head is never completely stationary. We argue that there is perceptual information that does not lie in invariant patterns within ambient energies (e.g., optics, acoustics, inertia, chemical, etc.) but that extends across these ambient energies, due to multimodal stimulation.
We formalized an invariant relation specifying egocentric distance across the optical, inertial and acoustical consequences of the perceiver’s motion. We used that equation to drive the display of a target so as to set it virtually at different distances. Participants in our experiments had to judge verbally (“yes/no”) the “reachability” of a static virtual target (see Figure 1). We manipulated the intermodal relation between energies (e.g., by adding a gain in the relation) and we used two different types of display (projection screen vs. HMD). Our results showed that that this intermodal invariant was perceived and used by humans. Introducing a gain in that intermodal relation shifted the perceived distance exactly as we predicted and quantified it. Our intermodal invariant specifying distance does not even exist without movement; it has to be actively generated and cannot be passively experienced.

Recommendation to designers:

• Perception and action are tightly intricate
  • Perceptual information that an interface should provide strongly depends on the need of the task.
  • The interface should also permit adequate (task related) exploration to allow the user to generate the perceptual information necessary to perform the task.
• It is fundamental to maintain some invariant relations (between structures of simulated energies) that exist in the real world and that specify task relevant information. It implies to allow real time interactions and a special emphasis on synchronization and relative scaling of each device constituting the interface.

Key reference:
Stoffregen, T. A., Bardy, B. G. & Mantel, B. (2006). Affordances in the design of enactive systems, Virtual Reality, 10, 4-10.