Virtual Eyes Can Rearrange Your Body:
Adaptation to Visual Displacement In See-Through Head-Mounted Displays

ABSTRACT

Among the most critical issues in the design of immersive virtual environments are those that deal with problem of technologically induced intersensory conflict and one of the results, sensorimotor adaptation. An experiment was conducted to support the design of a prototype see-through head-mounted display (HMD). When wearing video see- through HMDs in augmented reality systems, subjects see the world around them through a pair of head-mounted video cameras. The study looked at the effects of sensory rearrangement caused by a HMD design that displaced the user's "virtual" eye position forward (165 mm) and above (62 mm) toward the spatial position of the cameras. The position of the cameras creates images of the world that are slightly downward and inward from normal. Measures of hand-eye coordination and speed on a manual pegboard task revealed substantial perceptual costs of the eye displacement initially, but also evidence of adaptation. Upon first wearing the video see-through HMD, subjects' pointing errors increased significantly along the spatial dimensions displaced (the y dimension, above-below the target, and z dimension, in front-behind the target). Speed of performance on the pegboard task decreased by 43% compared to baseline performance. Pointing accuracy improved by about a 1/3 as subjects adapted to the sensory rearrangement, but it did not reach baseline performance. When subjects removed the see-through HMD there was evidence that their hand-eye coordination had been altered. Negative aftereffects were observed in the form of greater errors in pointing accuracy compared to baseline. Although these aftereffects are temporary, the results may have serious practical implications for the use of video see-through HMDs by user populations (e.g., surgeons) who depend on very accurate hand-eye coordination.

1. INTRODUCTION

The National Research Council's (NRC) report on virtual reality expressed concern regarding current knowledge about the effects of displays on the visual system:

An important set of issues concerning perceptual effects in the visual channel that are only now beginning to be addressed concern ... augmented reality displays... Many of the perceptual issues that have important implications for the design of technology for the visual channel, and about which current knowledge is inadequate, concern how humans respond to various types of sensorimotor alterations associated with the visual display. (Durlach & Mavor, 1995, p. 48).

Furthermore, the NRC report recommend research on how synthetic environments "might alter the mental model ... of [the user's] own body" (Durlach & Mavor, 1995, 45). This study directly tackles both of these issues: (a) changes in visual perception following use of augmented reality displays, and (b) changes in users' mental models of their bodies. We report on the sensorimotor effects of one key design feature of existing video see-through HMDs -- visual displacement of the user's eyes to a virtual position -- the entrance pupil of the HMD's cameras.

Engineering design restrictions in this generation of video see-through HMDs require displacing human vision. The cameras that record the physical world and the video displays that present the physical world to the viewer cannot occupy the same position in space. The entrance pupil of the cameras must be in a different position in space than the entrance pupil of the viewer's real eyes because video displays are in front of the user's eyes. In the prototype tested in this experiment, the user's virtual eyes, the video cameras, are located 62 mm above and 165 mm forward from the viewer's natural eye location (see Figure 1). This video displacement of the eye location is a form of sensory rearrangement that forces the user to adapt to the system. Therefore, this study applies perceptual adaptation research directly to the engineering and design of HMDs to evaluate the human cost of the sensory rearrangement.

Figure 1: This picture shows the exact location of the miniature video cameras on the see-through HMD. They displace the user's vision 62 mm above and 165 mm forward from the viewer's natural eye location.

1.1. Design challenges of see-through head-mounted displays.

See-through HMDs are called for in applications where virtual objects are superimposed on the physical environment to enhance a user's experience. Nowhere is the promise of see-through head-mounted displays (HMDs) and augmented reality more compelling than in medical imaging applications. "Medical Imaging, since its birth, has provided a valuable and yet non-surgical possibility to see what was unseen before: the internal world of the human body" (Yao, 1994, p14). Virtual environment designers try to use see-through HMDs to take doctors and radiologists one step further towards a kind of "X-ray vision, " seeing the internal anatomy of the human body superimposed on the body of the patient.

Augmented reality displays may offer a number of advantages for medical imaging. Doctors will not have to divert their vision to a side monitor or viewing screen to see inside a patient's body. The virtual image of the internal organs and the real body of the patient will be merged. Doctors have always used natural observation of the body for diagnosis. With successful augmented reality systems the natural ability to observe the symptomology of a body will be extended and augmented by fusing normal observation with the visualizing power of the X-ray, the magnetic resonance machine, and the ultrasound machine. At least, this is the promise of the research and development program.

But a number of design challenges must be overcome before the promise of see-through HMDs becomes reality. One of the challenges of such devices is providing depth information that accurately merges the virtual scene with the real scene as well as keeping the real and virtual scene registered in space as the viewer or the patient move. As we will see in this study, another is building a system that minimizes sensory rearrangement and the need for user adaptation.

Two approaches to HMD hardware design are now common. Real and virtual views of the world can be merged either: (1) via a semi-transparent mirror as with optical see-through HMDs (Buchroeder, Seeley, & Vukobratatovich, 1981; Berman & Melzer, 1989; Droessler & Rotier, 1990; Rolland, 1994; Rolland, Ariely, & Gibson, 1995), or (2) via video cameras mounted on the helmet as with video see-through HMDs (Bajura, Fuchs, & Ohbuchi, 1992; Edwards, Rolland, & Keller, 1993; State et al., 1994). A discussion of design issues and the relative merits of each approach can be found in Rolland, Holloway, and Fuchs, 1994.

2. SENSORY REARRANGEMENT, INTERSENSORY CONFLICT AND ADAPTATION TO VIRTUAL ENVIRONMENTS

Humans adapt in many ways to new environments. For 100 years experiments have been conducted on ways in which humans adapt to altered perceptual environments (e.g., Stratton, 1896, 1897; Ewert, 1930; Snyder & Pronko, 1952; Held & Bossom, 1961; see reviews such as Kaufman, 1974; Rock, 1966; Welch, 1978). This research is important to virtual environment design because virtual environments are altered sensory environments. But studies in this tradition have rarely -- if ever -- been intended to guide the design of new visual devices or communication technologies. Rather, these studies have explored theoretical questions in perceptual development or attempted to answer variations on the nature-nurture question: how much of perception is innate, how much is learned, etc. Experiments explored how much subjects adapted to perceptual rearrangement and what aspects of perception were modifiable by perceptual learning and which were not. If adult perception could radically adapt to new perceptual environments, then, it was reasoned, this would provide evidence in favor of a strong role for experience in the component of perception under study.

Because optics were well understood and visual perception was of interest, many experiments explored this question by using lenses and other optical devices to alter the spatial properties of the retinal image of the world (the proximal stimulus). Although the devices were not of central interest in these studies, various "head mounted" systems were used over the years to "distort" the retinal image: Galilean telescopes (e.g., Stratton, 1896, 1897), view restricting tubes (e.g., Dolezal. 1982), mirrors (Kohler, 1964), contact lenses (Taylor, 1962 ), and the popular prismatic lenses (e.g., Held & Bossom, 1961). Because we are interested in optimizing the design of a see-through head-mounted display, this research is quite applicable because, unwittingly, it shows the negative effects of vision distorting optical design on the perceptual system. The research also provides theoretical and methodological guidance regarding problems induced by sensory rearrangement (e.g., Held & Gottlieb, 1958; Held & Bossom, 1961).

Immersive virtual environments (VE) and telepresence systems are likely to induce some form of sensory rearrangement for the foreseeable future. Video see-through head-mounted displays are a good example of a virtual reality (VR) component that, by nature, entails sensory rearrangement. Sensory rearrangement is a change in the normal relationship between body movements and the resulting inflow of sensory stimuli to the central nervous system. It can also result from discoordination of one sensory inflow pattern with that of another sense, for example a mismatch between vision and touch. This is known as intersensory conflict (Reason, 1978; Oman, 1991). In VEs, sensory rearrangement can result from a discoordination of displays to the various senses. According to Welch (1995), "it is not so much the absence of certain stimuli that causes serious perceptual and behavioral difficulties with telesystems, but the presence of intersensory discrepancies, such as mismatches between sensory modalities and delays of sensory feedback" (p. 1).

Intersensory conflict puts a stress on the user's body, especially when the conflict involves the vestibular system (Reason, 1975). The stress can have cognitive, behavioral, and physiological manifestations. For example, performance is slowed down immediately after entering a HMD-based virtual environment. Movements are short and tentative, the user may be slightly uncoordinated, hand reaching behavior is uncertain and inaccurate. Sensory rearrangement can also contribute to simulation sickness (Biocca, 1993). During extended use, users may experience sweating, eyestrain, stomach awareness, and vomiting (Kennedy et al., 1992). To minimize the noxious effects caused by intersensory conflict, susceptible users may limit their movements. Such altered behavior is a concern in most training environments because inappropriate behaviors learned in response to the simulator can negatively transfer to the real environment.

In this study we wanted to explore how a user's motor system would adapt to VE induced sensory conflict between the visual and kinesthetic-proprioceptive systems. We focused on the relationship between the eyes and the hands because intersensory conflict between vision and sensed hand position (proprioception) is critical to performance in VEs. A central component of medical, military, and other training systems is learning subtle, coordinated hand-eye movements.

2.1. Research Questions

Although the research questions involve perceptual adaptation, a point needs to be made regarding the distinction between the research goals of basic studies in perceptual psychology and studies in human-computer interaction. Our experience with this study suggests that a misunderstanding of this distinction may be a possible source of confusion with some readers. The goal of this study is not to uncover some new form of perceptual adaptation or extend the theory of perceptual adaptation.

This study is based on a different research logic, the logic of the design sciences (Simon, 1969). Most design research on virtual environments attempts to create technological artifacts that augment human ability (Biocca, 1996), not ones that manipulate human abilities solely for the purpose of experimentation and observation. Our goal is not to simply observe distortions in human perception caused by technological artifacts, but to eliminate unwanted outcomes or, ideally, to augment human abilities. For example, the glasses with prismatic lenses constructed and used in the long series of studies on visuo-motor adaptation (e.g., see review by Welch, 1978) did nothing of practical value outside the logic of the experiment. The fact that these glasses distorted and interfered with human vision was a desirable property. So even though perceptual distortions have been observed in the past, this by itself does not provide any specific, design relevant knowledge (see Carroll, 1991 on the specificity problem) on how to engineer head-mounted displays or how human adaptation will interact with the unique features of different HMD designs. In this design-oriented study we attempt to answer basic design questions related to human-computer interaction in virtual environments.

This study sought to answer the following questions:

The presence of negative aftereffects has tremendous practical significance for the use of VEs, especially in medical applications. Consider, for example, the use of see-through HMDs by surgeons. Some form of safety protocol would be necessary if use of a video see-through HMD were to temporarily alter the hand-eye coordination of a surgeon! But the issue of negative aftereffects extends to many other VR applications as well. What detrimental negative aftereffects might influence user performance in applications requiring high levels of hand-eye coordination: e.g., engine repair, athletics, weapons aiming.

3. METHOD

The present experiment used a 3 X 2 mixed, experimental design with three within-subjects and two between-subjects factors. The main within-subjects factor was type of HMD, the three levels of which were:

1. no HMD,
2. see-through HMD, and
3. a control-model of the HMD (see description in apparatus section below).

3.1. Subjects

Fourteen subjects participated in the study, 12 males and 2 females. All subjects were right-handed and had an interpupillary distance (IPD) of 64 mm (+/- 1mm). The latter requirement was set to match the parameters of the equipment as described in the next section. Seven subjects had no previous experience, one had very little experience, four had some experience, and two had a lot of experience with HMDs. All subjects had 20/20 or corrected-to-normal vision.

3.2. Apparatus & Measures

Video see-through head-mounted display.The study focused on the adaptation effects of University of North Carolina's prototype video see-through HMD (See Figure 1). The main components of the system are a flight helmet from Virtual Research, opaque HMD using LEEP optics (Howlett, 1983), and two miniature custom-made fisheye lens video cameras (Edwards et al., 1993). The fisheye lenses were custom designed and built to match the FOV of the LEEP optics when integrated in the flight helmet, and to precisely compensate for the optical distortion of the optical viewer. The cameras are laterally separated by 64 mm to match the separation of the LEEP optics of the LCD displays inside the HMD.

A key design feature to note in this study is the location of the cameras and their effect on vision. Viewers see the real world through the cameras that are located 62 mm above and 165 mm forward of the viewer's natural eye point (see Figure 2). When wearing the HMD this altered eye position has the effect of displacing the images (environment) downward and inward. Therefore, objects are visually closer and lower down than normal, but, of course, they are not closer or lower when one reaches out to touch them. Most users are unaware of this slight change when putting on the HMD.

On the right is UNC's see-through HMD used in the experiment. Note the miniature video cameras located on the top of the helmet. On the left is a control HMD also used in the study. The control HMD was designed to match the weight, center of mass, and field of view of the test HMD, but allowed normal direct viewing of the natural environments.

Control head-mounted display. The control HMD (also shown in Figure 2) was designed to control for the potential effects of the weight, center of mass, and FOV of the test HMD on task performance. But the control HMD allowed viewers to directly see the natural environment. The control HMD matched the weight (7 lb.), center of mass, and FOV (73.7 x 60.8 dg.) of the see-through HMD. The actual field of view for each subject varied depending on the size and anatomy of the subject's head since subjects' eyes varied in their distance to the window. Our estimates were that the FOV of the control HMD would be within 10% on the X dimension and approximately 11% on the Y dimension. Further, we estimated that the FOV would be smaller and the bias, therefore, would probably be against any performance advantage for the control-HMD.

Open-Loop X-Y Pointing Accuracy Measure. Studies of adaptation require dependent measures of coordination between perceived visual spatial position and perceived haptic/proprioceptive location. The two must be measured independently. The X-Y pointing accuracy measure used in this study was an improved version of a reliable and valid measure of adaptation with a long history (Held & Gottlieb, 1958). Viewing through a pair of holes, subjects saw one of four, randomly lit, red LED lights inside the dark interior of a light sealed box (Figure 3). Subjects were instructed to touch the light with their right index finger. A calibrated touch screen captured the exact location touched and provided a measure of X-Y pointing error. A mirror set at 45 degrees gave the subjects the illusion of seeing a light straight in front of them while preventing them from seeing their hand. This feature kept subjects from using sight of their finger to "home in" on the target or from obtaining feedback as to their accuracy. The lamp for feedback shown in Figure 3a and 3b was not used in this open-loop measure, as suggested by the term "open-loop" itself, but was used at a later stage of the experiment as described in Section 3.

Figure 3: Diagram of the X-Y pointing accuracy measure that allowed users to point straight ahead at an object without seeing their hand. This light-sealed box had an opening at the bottom (See 3a). Subjects looked through view ports to see one of four LED's reflected off a 45 degree two-way mirror (See 3b). To the subjects, the LED's appeared to shine from the back of the box. Subjects touched the virtual LED's without seeing their hand or receiving feedback.1 Their pointing accuracy was recorded on a touch screen as X-Y coordinates.

Figure 4. Representation of the measure of pointing accuracy along the Z (depth) axis. Subjects pointed at the location of the white peg underneath a shelf. Subjects received no tactile or visual feedback of their pointing accuracy. A mirror at 45o allowed data recording using a video camera.

Open -Loop Z Pointing Accuracy Measure. This apparatus was used as a measure of pointing accuracy along the Z axis. Subjects were seated in front of a dark shelf from which a white rod protruded (See Figure 4). The subject's task was to touch the point on the bottom of the shelf where the rod would protrude if pushed through the shelf. The shelf prevented subjects from seeing their hand and gauging their accuracy (no feedback). Pointing accuracy was recorded by a video camera aimed at a polar grid pasted on the underside of the shelf.

This pegboard task is a standard measure of manual dexterity and hand-eye coordination. In this experiment it also gave subjects immediate sensory feedback of the discrepancy between their visual sense of spatial location and their kinesthetic-proprioceptive sense of location, causing a recalibration of the latter.

Pegboard Task. This is a standardized test of manual dexterity (Lafayette Pegboard, model 32027). See Figure 5. The bowl of pegs was placed in front of the subjects and the board was 1 ft. away from the bowl. This task requires quick and precise ballistic movements of the hand. This is the kind of task that requires calibration (or recalibration) of hand-eye coordination. In cases such as this study where the eye location has been rearranged, this task should require recalibration of the visuo-motor system.

3.3. Procedure

Following instructions, various physiological and behavioral trait measurements (interpupilary distance, depth perception, previous exposure to HMDs) were taken . These are not reported here. The experimental procedure is outlined in Table 1.

Step 2. Initial measure of effects of HMD. Following the preexposure period, subjects stood up and experimenters assisted them in putting on the HMD. Depending on the order to which subjects had been assigned, subjects either put on the see-through HMD or the control HMD following the baseline tasks. Subjects walked over to the measurement devices. They were pretested on the pointing accuracy measures (pretest X,Y,Z measures: 5 trials each).

Step 3. Exposure period. Subjects returned to the table and performed 10 timed trials of the pegboard task following the same procedure used during the preexposure period. It is important to note that this pegboard task served a number of purposes in this step of the experiment: (1) It was the first and only time that subjects had the opportunity to observe the effect of the HMD on their hand-eye coordination. The subjects repeated the hand-eye coordination task of quickly putting small pegs in the holes 250 times (10 trials X 25 pegs per trial). This allowed for visuo-motor adaptation to the HMD. (2) the 10 timed trials allowed the experimenters to observe the gradual process of adaptation and how practice and experience in the new environment diminished the initial effects of the HMD on performance.

After 10 trials of the pegboard task, subjects returned to the measurement devices and were measured once again for their pointing accuracy while still wearing the HMD (posttest X,Y,Z measures: 5 trials each).

Step 4. Post-exposure period. Subjects removed the HMD and performed the X, Y, Z pointing accuracy measures (5 trials each). This measured was used to detect the presence of differences between their pointing accuracy before-and-after using the HMDs.

Following a 5 minute rest seated at a chair, subjects repeated the same sequence of tasks and measures wearing the other HMD. If they wore the see-through HMD helmet in the first part of the experiment, they now wore the control HMD and vice-versa.

After the experiment, subjects were treated to a sequence of closed loop (feedback) trials of the X-Y pointing measure to recalibrate their visuo-motor coordination to normal. They then filled out a short simulation sickness questionnaire (not reported here).

4. RESULTS

4.1. Effect of See-Through HMD Usage on Manual Task Performance

The times to complete the manual pegboard task are reported in Figure 6. A 3 X 2 X 10 (type of HMD X HMD order X repetition) mixed, repeated measures analysis of variance was conducted on completion times for the pegboard task. HMD type significantly affected subjects' time to perform the manual task [F (2, 22) = 102.45, p < .001]. When using the see-through HMD (Mean = 76 sec.), subjects took an average 43% longer than their baseline performance with no HMD (Mean = 53 sec.) or the control HMD (Mean = 51 sec.). Subjects' performance times improved over the 10 trials. There was no effect for the order in which the subjects used the HMDs [F (1, 11) = 1.21, p = .21].

4.2. Effect of See-Through HMD Usage on Hand-Eye Coordination

In some dimensions there was a significant effect when subjects used the control HMD either before or after the see-through HMD. In those cases, the control HMD data are shown for both orders of HMD use.

A 2 X 2 X 3 X 5 (type of HMD X HMD order X measurement stage X repetition) mixed, repeated measures analysis of covariance was conducted. The measurement stages were a) before conducting the pegboard task, b) after the task, and c) after removing the HMD. The covariate was baseline-pointing error (no HMD). The between subjects factor was order of HMD use. The dependent variable was pointing accuracy along each of the three spatial dimensions. Separate analyses were conducted for errors along the X, Y, and Z dimensions.

Pointing Errors Along the X Dimension (left-right of target) See Figure 7a. Although errors appear slightly higher when subjects used the control HMD, type of HMD had no effect on subjects' ability to point accurately on a target along the X dimension [F (1, 11) = .98, p = .35]. Effects for order of HMD usage [ F (1, 11) = 1.83, p = .20] and measurement stage [ F (2, 22) = 1.01, p = .38] are not significant.

Figure 7. Effect of HMD type on pointing errors in each spatial dimension (X,Y,Z). (Subjects are wearing the HMD when measured.)

Figure 8. Aftereffect of HMD Type on Pointing Errors by Spatial Dimension. (Subjects are not wearing a HMD when measured).

Pointing Errors Along the Y Dimension (up-down of target). There was a significant main effect for type of HMD on pointing accuracy along the Y dimension [F (1, 11) = 9.77, p < .01] See Figure 7b. When subjects were wearing the see-through HMD that displaced their vision upwards, they tended to point downward of the actual target position. The maximum amount of error was 27mm, or only 41% of the amount of eye displacement along the Y axis (65mm). There also was a main effect of measurement stage (i.e., pre-posttest) [F (2, 22) = 8.21, p < .002] as well as an interaction of type of HMD by measurement stage [F (2, 22) = 30.85, p < .0001]. Subjects' errors tended to decrease following their completion of the manual task while wearing a HMD, but this adaptation effect appears restricted to usage of the see-through HMD.

Pointing Errors Along the Z Dimension (front-back of target).See Figure 7c. There was a significant main effect of type of HMD on pointing accuracy along the Z dimension [F (1, 7) = 63.29, p < .0001]. When subjects wore the see-through HMD that displaced their vision forward, they tended to point short of the target. The maximum amount of error was 65 mm, or only 39% of the amount of eye displacement along the z axis (165mm). There was a main effect of measurement stage [F (2, 14) = 174.76, p < .0001]. Subjects' errors were less pronounced after they conducted a manual task using the HMD. Although there was no main effect for the order of HMD usage [F (1, 7) = 1.35, p < .28], there was an interaction of measurement stage and order [F (2, 14) = 4.92, p < .03] as well as an interaction of type of HMD by measurement stage [F (2, 22) = 28.25, p < .001]. There appears to be no effect when the control HMD preceded use of a see-through HMD. When the control HMD was used after, there appears to be an effect on pointing error. This may be due to residual aftereffects from the see-through HMD.

4.3. HMD Use And Negative Aftereffects On Hand-Eye Coordination

For pointing on the Z dimension there was an interaction of type of HMD by measurement stage [F (2, 22) = 28.25, p < .001] as well as interaction of measurement stage and order [F (2, 14) = 4.92, p < .03]. This interaction is apparent in Figure 8. The different pointing error values for the control HMD when used before or after use of the see-through HMD along Z dimension indicate the presence of the significant interaction of order with measurement stage. The aftereffects following the order the see-through HMD is used first appeared to persist and were still present when the control HMD follows use of the see-through HMD.

5. DISCUSSION

This study asked three main research questions regarding the consequences of this adaptation. The discussion is organized around answers to these questions.

5.1. Research Question 1: How much will the user's initial motor performance deteriorate because the present design of video see-through HMDs displaces the eyes forward and upward?

5.1.1. Effect of the use of the video see-through HMD on manual task performance

The see-through HMD appeared to have a significant effect on the visuo-motor performance. Performance on a manual task requiring hand-eye coordination took 43% longer with the see-through HMD (see Figure 6). This decline in human performance appeared to be caused by intersensory conflict between the visual system and the kinesthetic system. After the subjects put on the see-through HMD, their hand motions were uncertain and tentative. When subjects quickly reached for the peg holes with the pegs, subjects significantly overshot the peg holes in the initial trials. There was a reduction of the effect of the see-through HMD. Errors stabilized near the end of the 10 trials as can be seen in Figure 6.

5.1.2. The Effect Of Video See-Through HMDs On Hand-Eye Coordination.

As expected, the pointing errors were greatest along the spatial dimensions displaced by the see-through HMD. The errors were systematic. Because their virtual eye position was moved up, images appeared lower than normal. But their kinesthetic system continued to point as if vision had not changed. Subjects' artificial "taller" point-of-view made them point lower than the target. See data on the effect of the see-through HMD for the Y axis (above-below target) in Figure 7. Likewise with virtual-eye position also pushed forward, objects appeared closer. Subjects erred by underreaching for objects before adaptation. See data on the effect of the see-through HMD for the Z axis (forward-back of target) in Figure 7. Because their vision was displaced along 2.5 times more along the Z dimension than along the Y dimension, errors were much greater in the Z dimension.

5.2. Research Question 2: Will users adapt to see-through HMDs and, if so, how quickly?

The amount of error dropped further (by about 1/3) as subjects further adapted to the sensory rearrangement while performing a task that required quick and precise hand motion. See the lower error rates after the manual task in Figure 7. Error rates would probably have dropped further if subjects had worn the see-through HMD for a longer period of time. The designers of the HMD hoped that users would adapt quickly. There is evidence of quick, if not complete adaptation, by users. The rapid, careful hand movement of the pegboard task led to quick adaptation in little more than 15 minutes. The pegboard task required careful movements similar to those to those found in surgery, but allowed for far more precise control and measurement. The quick adaptation with this task leads of the expectation of similar adaptation in other hand-eye coordination tasks such as those found in surgery.

The study did not have subjects wear the HMD for an extended period until they had fully adapted and reached baseline (no HMD) performance levels. Previous research on adaptation suggests that with continued practice the subjects might have performed at speeds close to their pre-exposure, baseline speeds. But after 10 trials and approximately 15 minutes of practice inserting a total of 250 small steel pegs, the lines in Figure 6 are somewhat parallel. This suggests that achievement of baseline performance might take quite a few more trials of extended practice. But even if subjects adapted to the visual displacement, they would also have to adapt to the poorer resolution and the more limited field of view of the video see-through HMD as compared to unmediated vision. The overall effect on vision may prevent performance from reaching levels exhibited without the HMD.

5.3. Research Question 3: Will adaptation to see-through HMDs lead to negative aftereffects, and what is the exact extent of those aftereffects?

Unfortunately, this desirable adaptation to the virtual environment may be linked to a undesired outcome for the user in the physical environment. Figure 8 shows the presence of significant negative aftereffects when the subjects removed the see-through HMD. After experiencing the altered spatial dimensions of the virtual environment, the users automatically recalibrated their visuo-motor system. The visuo-motor system was still calibrated for the virtual environment once the see-through HMD was removed.

Subjects reentered the physical environment and found that adaptation to the virtual environment altered their performance. The HMD removed, subjects exhibited a negative aftereffect, overshooting the target in the pointing task in a direction opposite the errors made when they entered the virtual environment. The effects with the HMD are consistent with the effects one would predict from perceptual adaptation studies using other forms of visual displacement (see review by Welch, 1978).

5.4. Some Limitations Of The Study

The mixed design allowed us to observe the effect of HMD on performance either in a between subjects analysis or a within subjects analysis. We anticipated the aftereffect might carryover across the within subject conditions. By counterbalancing the order in which the HMDs were used, we were able to demonstrate that this aftereffect only carried over and persisted when the see-through HMD was used, and not when the control HMD occurred first. But this may introduce additional and undesirable complexity into the analysis. While this design is more efficient in terms of time, reviewers have suggested that researchers designing similar studies might preferably eliminate the carry over aftereffect across within-subject conditions by separating the use of the HMDs by one full day.

6. IMPLICATIONS FOR THE DESIGN OF SEE-THROUGH HMDS

6. 1. What are some of the design options for minimizing the effects found with this generation of see-through HMDs ?

Another solution is to completely redesign the system (video-cameras/viewer) so that the visual displacement is reduced to zero. But this solution will trade its achievement of zero visual displacement against FOV, simplicity of design, and perhaps some comfort to the user. If a large FOV is mandatory, a redesign of the viewer needs to be undertaken. For example, designers might consider a tiling technique using several small displays to increase the FOV without loosing resolution and compactness (Kaiser, 1992; Rolland et al., 1994). The more compact the system, the less visual displacement it will have. In this scenario the video camera input will either cover a smaller FOV than that provided by the viewer or be segmented to be tiled on different displays.

Absolute zero eye displacement may not be necessary. Maximum pointing errors were only about 40% of the virtual eye displacement. This suggests that pointing errors might become negligible before the virtual eye displacement reaches zero. The effectiveness of future designs can be measured to see if they lessen the pointing error rates found in this study. The measures used in this study and other standardized, reproducible measures could provide benchmarks for assessing the relative human performance value of various competing designs.

6..2. What are the implications for use of this generation of see-through HMDs by specialized user populations, such as surgeons, who depend heavily on the hand-eye coordination?

6.3. How long might the negative aftereffects persist?

7. CONCLUSION

ACKNOWLEDGMENTS

REFERENCES

1 The lamp at the bottom of the figures 3a and 3b was not used during the experiment, it was only used at the end. A closed loop pointing procedure was used to "recalibrate" the subjects' hand-eye coordination back to the physical world