Immersive Virtual Reality to assist retraining of acquired cognitive deficits: 
First results with a dedicated system

Luigi Pugnetti, Laura Mendozzi, $ Achille Motta, Annamaria Cattaneo, @ Elena Barbieri, * Aaron Brancotti, # Luigi Cristiano and Carlo L. Cazzullo. 

Scientific Institute "S. Maria Nascente" and Multiple Sclerosis University Center 
- Fondazione Don Gnocchi, via Capecelatro 66, 20148 Milano; 
$ Psychiatric Clinic "Città di Monza", Monza; 
@ Dept. of Psychology, University of Padua; 
* University of Milan, Computer Sciences Dept;
# Cyberfunk Italy SRL, Signoressa (TV)

Abstract

It is proposed that affordable immersive virtual reality (IVR) technology has the potential of improving current procedures for the clinical evaluation and retraining of acquired cognitive impairments of the adult through the simulation of quasi-real 3D interactive situations. To pursue this goal, we have promoted the ARCANA project, a nationally-founded collaborative initiative aimed at the development of IVR systems dedicated to research and clinical applications in the field of cognitive science. The hardware, the virtual environment and the cognitive demands of the ARCANA systems have been designed to meet the requirements for utilisation with impaired patients. In addition, the virtual environments are parameterized, and hence easily configured by the therapists. Cognitive performance can be analyzed in terms of both competence and processing aspects. The systems store behavioural data in many formats and enable the monitoring of biomedical data from the behaving subjects. 
The clinical validation of ARCANA 1 - devised to assist the assessment and retraining of control components of cognitive activity - is in progress in our Institute on brain damaged patients, while the physical and psychological effects of long- term IVR exposure on normal individuals are concurrently assessed.

Key words: immersive virtual reality, cognitive retraining, executive functions, affordable technology, biomedical monitoring. 

1  Introduction 
As the theoretical and empirical foundations of cognitive rehabilitation are continuously evolving, it has been recognized that its chances of improving the quality of life of brain damaged patients depend also on a correct application of methodologies and a viable search for better technological aids  [1]. While others have discussed in greater detail the potential of virtual reality (VR) technology to meet the requirements for construction of cognitive rehabilitation tasks [2], we have specifically focused on the rationale for developing  immersive VR systems to assist the diagnosis and the retraining of deficits of higher-order cognitive processes. 
Because concerns over limitations of present-time affordable technology have been raised, we propose that the concept that should lead the development of clinically useful immersive VR systems is analogous to that suggested by Wierda et al. [3] for the development of so-called "cognitive trainers"; i.e. that emphasis should be put on cognitive and perceptual issues rather than on perfect realism. 

2 The project. 
The A.R.C.A.N.A. (Advanced Research for the Computer-based Assessment of Neuropsychological Ailments) project started in October 1993 with the support of Cyberfunk Italy and, more recently, by the Italian Health Ministry with a research grant to our institute. Three sub projects dealing with different aspects of IVR modelling of cognitive paradigms and their simulation are being developed by our team. 
The rationale that led the development of ARCANA 1 system is fully explained in a recent paper by our group [4]. Briefly, we were interested in developing an affordable clinical tool for the enhancement of our diagnostic and retraining potential of the so-called "strategy application disorders" (SAD) [5], which result from direct or indirect damage to the frontal lobes. When severe, SAD is associated with gross behavioural abnormalities (dyscontrol syndromes) that impair one's ability to carry out an unsupervised life. When milder, SAD can substantially disrupt one's social and productive life and prevent a good readjustment of otherwise unimpaired subjects. From a neuropsychological standpoint, the cognitive functions that are required for an adequate social adjustment and a purposeful behaviour are called "executive functions"; these are adaptively controlled by a Supervisory Attentional System (SAS) - a term to indicate a complex functional network that connects the frontal lobes to almost all cognitively relevant parts of the brain [6].
In general, the retraining of mildly-to-moderately impaired subjects after traumatic brain injury, cerebrovascular accidents or other central nervous system diseases has challenged clinicians and therapists partly because of the lack of efficient instruments to assess and retrain the patients under everyday life, open-ended - but nonetheless safe and controlled - situations. In short, we have proposed that IVR may be useful for: 

1)	simulating and making interactive both well-known  and de-novo devised paradigms; 
2)	obtaining behavioural and physiological data which -besides retaining their clinical meaning -  are less stereotyped than those seen under standard laboratory conditions; 
3)	approaching from a novel stance the still unresolved problems of transfer of diagnostic information to real-life situations. 

We have also suggested that the retraining of deficits such as those of strategy application can be assisted more effectively by the simulation of a whole range of quasi-real situations, thus making the generalisation of improvements less problematic.
In essence, the patient is encouraged to show how good he is in tackling the simulated situation we are presenting him. If he is good enough, we may easily re configure the simulation and present him with a different version of the same general situation or with a completely new one and analyze again his behaviour. This process can be repeated iteratively until learning of critical skills - and eventually transfer to other situations - can be demonstrated.
 
ARCANA 1 was devised as an entry-level Program to IVR, to probe both patients' and therapists' attitude toward this new technology. Although it is specifically derived from a well-known neuropsychological paradigms that assess cognitive flexibility, it will nonetheless allow more empirical and behaviourally-oriented research and applications. 
Another system, ARCANA 3 - a driving simulator - is currently being developed with the aim of addressing more specifically deficits of attentional control while the patient is attending to a situation in which he has to adapt momentary intentions to complex and unpredictable external events, while also showing basic precept-motor cognitive abilities.  

3 The Virtual Environment of ARCANA 1. 
After wearing a Head-Mounted Display (HMD), the user finds himself in a virtual building which, in fact, is a series of rooms of variable polygonal shape with entrance and exit doors. The rooms may be connected with corridors of variable length. Rooms and corridors may be empty or may contain a number of fixed or animated objects (e.g. furniture, paintings) to make them resemble familiar environments. 
The interaction is kept as simple as possible. The user can freely navigate the environment by simply holding a pointing device (a jokerstick) in the direction of the intended movement. The doors open when touched on the knob with a virtual key. 

4 The Task. 
First, the patient should be made fully acquainted with the equipment and then assigned a task which may be as simple as describing what he sees through the HMD or as complex as to exit the building in the shortest possible time. This goal is accomplished by developing a strategy to move from a room to the next, since a room may be left only by selecting the correct door. To assist the choice, explicit or implicit visual cues are made available to the user. For example, valid cues may be the colour or the shape of a door but not a combination of them. Feedbacks may also be explicit or implicit: e.g. one cannot be certain that the selection is correct until the exit door is reached. The opening of the exit door is an implicit confirmation of correctness. Once a strategy has been consolidated, the criterion can be changed to probe mental flexibility. Interfering stimuli and tasks can be introduced to match more closely the multiple-subgoal conditions of most everyday life situations. The cognitive model, the demands and the setting of ARCANA 1 have been recently discussed in greater detail in a related paper [4]. 

5 The Setting. 
Adequate settings can be any quiet clinical facility with sufficient room for both equipment and the user to walk around a 3 x 3 meters surface. Adequate safety measures and a constant supervision by trained personnel is also required. 
Since objective measurements are obtained from the ARCANA 1 system, it is mandatory that any user is made perfectly comfortable and acquainted with the equipment before exposing him to an IVR experience. Although brain damaged patients may be considered at grater risk for negative reactions to "wired" equipment and unusual experiences, our still growing experience with a variety of them suggests that their level of acceptance is extremely good provided they are carefully informed beforehand and care is used to avoid disorientation.

6 Performance Assessment. 
Cognitive performance on ARCANA 1 can be analyzed by counting the errors made on the primary task, as is done in the standard neuropsychological tests (competence analysis) and/or by tacking into account the strategy employed and the cognitive steps involved (process analysis) as is more appropriate in a rehabilitation setting. Both types of analysis are made easier by automatic scoring routines, report files, play-back options, graphic and numerical outputs of ARCANA 1. 

7 The Virtual Environment and the Task of ARCANA 3.
This application has been devised to represent as many as possible prototypical traffic situations with which a patient who is going to resume driving after a major head trauma or other pathological events must confront with. As already stated, perceptual and cognitive issues have been privileged at the cost of realistic kinestetic feedbacks which qualify the process of controlling the car. The rationale and the cognitive model underlying this application are being discussed in a forthcoming paper. Though driving is considered to be essentially a visual task, it is also a perfect paradigm to assess and retrain a number of other cognitive skills which are not directly involved with navigation by sight (e.g. tracking and scanning, orienting, and visuomotor control) such as long-term planning, momentary decision-making, focusing, sustaining and dividing attention, memory and learning.     
We are currently using a prototype of ARCANA 3 as a tool for psycho physiological monitoring of central and peripheral body signals in healthy adults and to assess the incidental encoding of episodic visuospatial material also in patients.
A third system will propose cognitive situations typical of everyday life social interaction places such as shopping centers, railroad stations and recreational areas. This system will also include real interaction possibilities between users. 

8 Hardware and Software 
The ARCANA systems are based on affordable hardware such as high performance 486 workstations. ARCANA 1 System specifications are reported in Table 1. The whole system (VR image processor, magnetic tracker system, HMD and other peripherals) has been calibrated in order to prevent image rendering rate not to drop below 12/sec. at times of highest image complexity (i.e. n. of polygons to be generated and complexity of software controls on the VE). 
The proprietary software have been designed to allow parametrical setting of almost everything is generated in the VE. By means of a friendly software interface any operator can configure the VE in a number of ways to obtain - for example - a grading of cognitive demands within a given model and to extend the variety of paradigms embedded within the primary task (e.g. to manipulate the attentive demands by inserting distractors at the needed places, to stress memory by increasing the time the cue has to be retained, etc.). 
The systems store codes of every user's behaviour (head's and arm's movements) and its  time of occurrence. These data are then analyzed to give summary reports and graphical displays which add significant information for both statistical and clinical purposes. For example, the visual scanning strategy of a subject can be easily displayed and related to specific cognitive steps (Figure 4). Audio-visual monitoring is also a feature of ARCANA systems. Subjective views generated by the VR server are synchronized with outside views of the behaving subject taken by a VHS camera. Combined images are stored and can be played back to assist off-line analysis. 
Another important feature of ARCANA systems is biomedical data acquisition, which can be accomplished by connecting them digitally with other equipment, such as a battery-operated digital polygraphic recorder. The recording of biological signals during IVR can serve many purposes. The interested reader is referred to a recent publication for more details on this issue [4]. Briefly, we are currently using this facility to recover auditory and visual event-related potentials (ERPs), on-line monitoring of heart-rate, muscle tone (EMG) and eye movements (EM) from behaving subjects during their IVR sessions and, in the next future to implement feedback paradigms. 

9 Implementation Steps
The second release of ARCANA 1 IVR system is about to start clinical testing to determine: 
- its safety under acute and rather prolonged exposures (i.e. prevalence of reported side-effects during and after IVR immersions of up to 60 min.);    
- the adequacy of the  3D scenario in terms of visual, auditory, interactive elements and feedback responses; 
- the adequacy of the task proposed in terms of its cognitive demands, meaning, and acceptability; 
- system performance quality; 
- on-line monitoring of subjects' relevant physiological parameters and behavioural data gathering. 
Iterative tests on normal subjects are run in parallel with tests on brain-damaged patients to ensure that the system has reached the needed functionality also on cognitively impaired subjects. Collaborative studies have also been planned to assess the reliability of results. Finally, clinical studies will be carried out to determine the value of ARCANA systems as clinical tools with respect to currently used PC-based systems and their adequate positioning within a comprehensive cognitive retraining protocol of selected samples of brain-damaged patients with cognitive sequelae.

Acknowledgements: this work has been partly supported by a grant of the "Gruppo Volontari per la Ricerca sulla Sclerosi Multipla". Our thanks also to Mr. Tullio Bortoletto and Ing. Renzo Castellini of Cyberfunk Italy for their generous contribution and continuing technical support and expertise.   

Table 1

Immersive Virtual Reality System for Project  ARCANA 1
Basic Components and System Configuration
Hardware 

1 - VR Server 
	Compaq Prosigna, Pentium, 60 Mhz, 16 Mb RAM, 
	HD 200 Mb, CD Reader, Monitor 
2 - Graphic and acceleration boards
	Custom made 
1 - Head Mounted Display
High resolution type, Low voltage CRT screens, light weight, fully adjustable for every head size, Interpupillary distance adjustment, separate focus adjustment
1 - Tracking System for HMD
	Polhemus Fastrak - 120 Measurements/sec - 4 msec Latency, 
	10 feet Range
1 - Tracking System for " Joker Stick"
	Polhemus Fastrak 120 Measurements/sec - 4 msec Latency, 
	10 feet Range
1 - Data Acquisition Server
	Computer 486 DX 33, 8 Mb RAM, complete with:
		- 2 Video Capture boards for simultaneous images acquisition 	from 1 Video camera and from Monitor of VR Server
		- Monitor SVGA
1 - Audio Generator and triggering System
	- Professional  Sound Synthesiser controlled via MIDI
	- Trigger Interface to  Biomedical Devices
1 - Biomedical Data Acquisition System
	As per End User request
Software

	Proprietary Software Program, custom designed for assessment and retraining of patients with cognitive deficits.
 
Technical specifications were kindly supplied by Cyberfunk Italy Srl, Medical Division, of Signoressa (Treviso). 

References

[1] Y. Ben-Yishay and L. Diller, Cognitive remediation in traumatic brain injury: update and issues. Archives of Physical and Medical Rehabilitation 74 (1993) 204-213. 

[2] A.A. Rizzo, Virtual Reality applications for the cognitive rehabilitation of persons with traumatic head injuries. Proceedings of the IInd International Conference on Virtual Reality and Persons with Disabilities, H.J. Murphy (ed.), Center on Disabilities, CSUN, Northridge CA (1994). 

[3] M. Wierda et al., Immersive virtual environments as trainer: system design from a cognitive stance. Proceedings of 76th AGARD Symposium on Virtual Interfaces: Research and Applications, NATO, (1993, in press). 

[4] L. Pugnetti et al., The evaluation and retraining of adults' cognitive impairments: which role for virtual reality technology? Computers in Biology and Medicine, suppl. (1994) (in press).  

[5] T. Shallice and P.W. Burgess, Deficits in strategy application following frontal lobe damage in man. Brain 114 (1991) 727-741. 

[6] D.A. Norman and T. Shallice, Attention to action: willed and automatic control behavior. In:  R.J. Davidson, G.E. Schwartz and D. Shapiro (eds.), Consciousness and Self-regulation, Plenum Press, London, 1986, pp. 1-18.