Special users and uses

Enactive Interfaces open new possibilities in human-computer interaction, both for what regards the subjects of the interaction and the activities they can perform. The main objective of this task is to set-up recommendations for the design and development of Enactive Interfaces and Applications that allow special users to access to a wide range of the Information Society Technologies: eliminating the “digital gap” that is being created and trying to improve the quality of life of these users.

The number of special users is large and their typology varied. We have focused mainly on: Vision impaired people, Hearing impaired people and Motor impaired people. A brief description of the main experiments undertaken at 2006 and their main results is provided below. You can find a more detailed description of most of these experiments in the report “Results of the first experimental assessment and recommendations for the design and development of Enactive applications for special users” available together with some videos of the experiments at the ENACTIVE website.

Visually Impaired people (PERCRO, ULUND, UPPSALA and LABEIN)

Pan and zoom tools in audio haptic environments

Objective: evaluate different strategies to pan and zoom in a virtual scene both in visual and non-visual navigation.

Experiment description: twelve sighted users evaluated different pan & zoom tools with a complex 3D scene (a traffic model, see figure 1), using the PHANToM device. To compare sighted interaction with interaction using only touch and hearing, the users first performed the test with vision and after that without visual feedback. Two different implementations for the zoom were tested: Stepwise zoom by pressing a key on the keyboard and Click & drag zoom using the PHANToM. For the pan or scroll functions, three designs were tested: Moving the world using the arrow keys, Moving the world by pushing the sides of the limiting box with the PHANToM stylus and Moving the world by clicking & dragging using the stylus.

Figure 1: Views of the traffic model.

Results: all users complete the tasks both with and without visual feedback, confirming that it is quite possible to understand and interact with large and complex haptic environment even in the non-visual case. The results show that all the pan designs are working quite well, but the drag function with the stylus seems to be the most popular option for non-visual interaction. For the zoom, it was clear that the design where the user is in contact with the same point of the model before and after the zoom works well. However, the design of the drag zoom function was not a success. Finally, the test highlights the fact that user preferences depend a lot on whether or not they have access to visual feedback. This is a reminder for anyone developing non-visual applications that the interaction needs to be specially designed for the non-visual case

Audio-haptic tools for Navigating a 3D scene

Objective: examine the influence of different navigational tools on spatial perception

Experiment description: a task of locating three targets and then reproducing their positions was chosen to test effects on spatial memory. The targets to be located were small boxes to make it virtually impossible to find objects by chance. Eleven sighted persons and one blind user tested, with the PHANToM premium, three navigational tools:
3D audio using the “ears in hand” metaphor (the listener position is attached to the tip of the stylus), Linear fixture (movement of the user’s finger constrained to a line towards the object) and a constant radial force that attracts the tip of the stylus to the object.

Results: a weak constant attractive force was shown to be useful. This force should be weak enough to allow the user to resist it, while at the same time strong enough to attract the user to the target once the grip is released. The use of a fixture to restrict the user to a path leading to the object was also useful. More users preferred the attractive force, but one user liked the fact that with the fixture he had to perform the movement himself. 3D sound feedback with the ears of the listener attached to the PHANToM position (“ears in hand”) is a type of feedback which may help users to gain an understanding of a spatial environment and which may also increase the sense of immersion within the environment

Haptic-audio drawing program

Objective: evaluate a virtual haptic-audio drawing program prototype

Experiment description: it consists of a virtual paper sheet, which a user can draw a relief on. When the PHANToM pen is in touch with the virtual paper the user can draw on it while pressing the PHANToM switch. The haptic image is produced as positive or negative relief depending on which alternative is selected. To enhance the user’s perception of localization a sound field was added. When the cursor moves in the virtual room, the pitch of a position tone is changed, brighter upwards, and mellower downwards. The mode information is conveyed by the volume and timbre of the tone. In free space, a pure sine wave is used. When the user is in contact with the virtual drawing paper the volume is louder. And when the user is drawing the tone is changed to a saw-tooth wave. It was tested in a formal pilot test with 11 blindfolded sighted adults.

Results:

• Drawing lines with a haptic drawing tool is not too easy but not too difficult either; it is hard to use especially for users who also don’t handle an ordinary pencil very easily.
• Both positive and negative relief is possible to feel and to work with. Negative relief is preferred when working with simple line shapes. There are indications that negative relief shortens examination times.
• Sound feedback did not seem to have a positive effect on task completion times.
• Both vertical and horizontal virtual paper work in a short, but what about ergonomics?

Teaching maths platform

Objective: analyse the use of enactive interfaces in the educational field and compare the results obtained with different haptic devices (2D/3D force feedback, 1/2 contact points, big/small workspace,..) and different interaction strategies to try to define “criteria” for the use of the most suitable tools and strategies for each task and user preferences.

Figure 2. a) PHANToM desktop, b) PHANToM Omni, c) GRAB, d) Haptic desktop

Experiment description: the maths platform can be used with different haptic devices (see figure 2): PHANToM desktop, PHANToM Omni, GRAB device (a two-finger 3D force feedback device developed by PERCRO) and Haptic Desktop device (a 2D force feedback device developed by PERCRO). These devices have different features that can affect to the user performance. The platform also allows:

• Different strategies to facilitate the navigation: deactivating (activating) shapes of the scene to simplify it, creating interactively reference points where the user can come back at any time through attraction forces, zoom and panning,….
• Different interaction strategies: free exploration of 3D shapes, constrained/guided exploration of curves and boundaries of 2D shapes, external/internal exploration of 2D shapes (Positive/negative relief),…
• Different haptic effects: virtual bumping to indicate special elements, virtual walls,..
• Different methods to execute an action (“user interface”): verbal commands, keyboard, haptic effects (tapping, stopping,…), switches of the device

This platform also provides some scenarios in order to: learn curves, learn shapes (2D and 3D) and learn mathematical concepts (symmetries, intersections,….). Most of these scenarios are provided as educational games that can also help to the spatial awareness development. Six users participated in some preliminary tests using the Omni device.

Results:

• Navigation strategies: attraction forces are very useful to achieve a specific target, interactive creation of reference points is a good technique to facilitate the navigation, virtual bumpings are effective to indicate special points (i.e. intersection points). The optimum value of some parameters (magnitude of force,..) depends on each device.
• Interaction strategies: the usefulness of each interaction mode depends on the user task. Constrained movement (useful to follow curves and contours of shapes because impede the loss of contact), guided constrained movement (can facilitate getting an “overview” of the shape, mainly at initial phase of the exploration), external exploration- positive relief- (facilitate recognising distances between shapes of a scene) and internal exploration - negative relief (facilitate recognising the dimensions of a shape and the interaction with a shape with respect to the positive relief).
• User Interface: the use of the keyboard should be similar to other common tools

Learning a route in a city-map

Objective: study the effect of a haptic reference point for learning a route on a virtual map

Experiment description: the reference point was provided with one of the two arms of the GRAB device (a two-finger haptic device) while the route was walked with the other arm. Three experimental conditions were used: the reference point was located at the start of the route, at the end or not used. The experiment consisted for each participant of five rounds. Each round was divided into a learning phase and a test phase. It started with the learning phase where the movement of the exploring finger was constrained (constraint forces) to follow the route. Then, in the test phase the participants' task was to freely move their exploring finger from the start to the goal following the streets that constituted the route according to their memory. At the end, the participant was asked to draw a map of the route on paper. The experiment included 30 sighted participants with blindfold.

Results: there were no significant differences between the three experimental conditions concerning accuracy and time. This is surprising as it was expected that using reference points in this way would improve the performance. However, the results of the drawings were quiet high in all three conditions, which suggests that the participants were able to create a rather accurate mental representation of the route.

Most participants could quickly use the system to navigate a virtual map and thought that this is a good way to learn routes. Two participants even felt that learning routes using haptics is as efficient as using vision; they both stated that they remember routes by turns and not by using an internal visualization of a map. However it is necessary to have more control of the audio help provided automatically to avoid confusion/irritation to the users.

Analysis of Sensory Memory Displaying Textures with Multimodal Devices

Objective: assess the influence of sensory stimuli on higher levels of memory.

Experiment description: a memory game, that may be accessed by any combination of sensory modality, was implemented to explore how well modalities integrate each other and which improvements can add to single modality fruition. The game is based on the concept of the "card memory game". In the simplest case, 40 cards are displaced on the virtual desktop and the user may operate with mouse or haptic to turn the card and search for couples of common values. In a more complex stage, the card can be explored by associating audio and haptic textures to the values of the card, and letting the user to inspect their "virtual value" by audio-haptic exploration instead of a visual one. A version of the game is published on http://www.enactivenetowork.org/MMM/WHC2007 (ENACTIVE MultiModal Magazine). Three users performed an extensive test session.

Results: haptic exploration in ENACTIVE modality improved memory performance with respect passive audio/video exploration, by a factor ranging between 20-25%. Moreover the performance improvement was unrelated to the time of completion.

Hearing Impaired people (UNIGE)

Speech-reading tests on a MPEG-4 virtual face

Objective: investigate if a 3D virtual face animated with MPEG-4 can be used by the hearing impaired community for speech-reading, see figure 3.

Figure 3: 3D virtual face animation for speech-reading

Results: Seven adults with different hearing losses participated in the tests

• Lip reading efficiency on a real and virtual character: experiments show that a virtual human can be used to help hearing impaired or deaf people as a substitute for an audio message although the understandability (i.e. percentage of correct answers) has been evaluated to 77% for the real character and 50% for the virtual character.
• Lip reading efficiency on a virtual character seen in ¾ and front view: the difference of the point of view does not influence the understandability.
• Speech-Reading efficiency with 2 different facial animation creation methods: it seems that an automatic way of facial animation creation allows a more efficient speech-reading (58%) than an animation created with an optical tracking device, such as the vicon (50%). This point is interesting since it shows that no expensive and complex devices are required to make an understandable animation and it also opens many doors for real-time applications and interactivity with virtual human since the animation can be easily build in real-time using a simple phoneme-viseme database.

Motor Impaired people (DIST and HFRL)

Auditory interfaces to enhance subjects performance in balance tasks

Objective : investigate the possibility to improve the performance of subjects involved in a balance task in presence of auditory feedback.

Experiment description:the system consists of a balance mechanical-electronic system and an auditory feedback generator system. The balance system (see figure 4) is made up of two parallel plywood platforms interconnected by a pair of wheels allowing the superior platform a bolt rotation (13 degrees for each side) in the medium-lateral plane only. During the tests, twenty-three healthy subjects were asked to get on the swaying springboard for 210 seconds and to maintain the superior platform in the reference plane (zero inclination condition) for the maximum time they could, doing balance through their weight. This task was performed in a silent room watching a white wall to limit information coming from visual channel and in different auditory conditions.

Figure 4: Balance mechanical-electronic system

Results:

• Auditory feedback seems to modify the motor behaviour of subjects involved in balance tasks, in terms of enhancement of responsiveness;
• Auditory feedback conveys to subjects information about the oscillations of the springboard helping subjects to improve their balance performances. However, the auditory feedback changing rhythm and lateralization does not appear to provide more information than the other ones. In fact, the lateralization auditory feedback results to be the most effective to map the oscillations of the springboard.
• Subjects had a good perception of the auditory feedback and their attributes (rhythm and lateralization), but they had not full consciousness of the correlation degree between oscillations and sound attributes changes.

Stabilization of posture relative to audio referents

Objective: evaluate the possible enhancement of motor performance accuracy and on the improvement of information communication concerning control and perception of dynamic orientation in stance due to an auditory feedback.

Experiment description: using the same balance platform described above, subjects were asked to reach and to maintain, for the maximum time they could, a set of seven different target positions (0 degrees, 3 right/left degrees, 6 right/left degrees, 9 right/left degrees) using an auditory feedback mapping in real time the spatial position of springboard. For each target position a “tolerance region” was defined with amplitude of 1.2 degrees. In this region the auditory stimulus becomes silent. Subjects (ten healthy subjects) were requested to stand on the springboard so that the auditory stimulus was silent.

: the data analysis for trials seems to confirm the hypothesis that subjects exploit successfully an auditory information flow to improve their perception accuracy in a motor task in addition to information furnished by the other sensory channels.