VR and Occupational Therapy

An investigation of the potential of virtual reality as a learning platform for students on the autism spectrum to practice activities of daily living and develop independent living skills.

The clip above shows real-time footage of a middle school student on the autism spectrum using the prototype for the cashier interaction, which targets the skills of waiting in line, interacting socially with the cashier, and paying. What you see is the result of his actual head movements and what you hear are his unprompted reactions when the cashier speaks to him.

PROJECT DESCRIPTION

In collaboration with a team of occupational therapists, I developed a series of interactive virtual reality (VR) experiences to provide students with Autism Spectrum Disorders (ASD) the opportunity to take virtual trips to community places (e.g. a grocery store, restaurant, subway station) to develop independent living skills. The VR experiences use 360 degree video to simulate a physical presence in real world settings, allowing students to practice activities of daily living and real-life social interactions within a safe, low-stakes environment.

The initial prototype is a virtual trip to a grocery store, which targets skills that are commonly included on students’ Individualized Education Plans (IEPs), such as finding items on a grocery list, using signs around the store to locate items, understanding aisle categorization, asking an employee for help, interacting with the cashier, and paying.

The goal was to design and develop a prototype for a low cost VR experience, which simply requires access to a mobile phone and Google Cardboard. The platform could potentially be usable during 1:1 occupational therapy sessions as a supplement to traditional Community-Based Instruction. The VR experiences would allow students with autism the opportunity to repeatedly practice targeted skills in a realistic environment without real-world consequences, prior to going out into the community.

BACKGROUND

My inspiration for my thesis project came from a conversation with a colleague, Kaitlin, who is an occupational therapist (OT) who works with students with Autism Spectrum Disorders (ASD) on a daily basis. Over a cup of coffee one afternoon, Kaitlin was telling me about her morning, which she had spent at the subway station with a group of her middle school students, teaching them how to buy and swipe a metro card, read a subway map, and safely ride public transportation to and from the school. This is a normal morning for Kaitlin. Trips like these occur on a weekly basis and vary in location, ranging from the subway station, to the cafe or restaurant down down the street, to the local grocery store.

What Kaitlin is doing, like many other occupational therapists who work with students with ASD, is using an evidence-based instructional strategy called Community-Based Instruction to help students develop functional independence as they approach the transition into adulthood. 

Evidence shows that the best way to help students develop skills to become more independent is to take them out of the classroom and into the real world, to practice in the natural settings where a skill is ultimately expected to be demonstrated. 

DEFINING THE NEED

While Community-Based Instruction (CBI) is the ideal method of instruction to help students with ASD develop functional independence, it can present many challenges. It is often difficult for OTs to take students out for CBI as often as needed to thoroughly master the skills. Listed below are the challenges identified through numerous surveys and interviews conducted with occupational therapists. 

1. Access

Many schools do not have access to places nearby in the community where OTs can take students to practice. At some of the schools I visited, I saw mock grocery stores or cafes (such as the one pictured below) set up in classrooms, designed to allow students to practice life skills and social interactions through role-playing. 

2. Time

CBI can be difficult to incorporate into students' daily schedules.

3. Safety

Safety is also be a concern during Community-Based Instruction trips, specifically when teaching students navigation/street safety skills, how to ask strangers for help, and in crowded places. 

4. Logistics

Organizing CBI trips can be quite a process, requiring permission slips, reimbursement forms, and partnerships with local businesses in the community. Below is a sample checklist from a school, which OTs are required to complete prior to a CBI trip with their students. 

THESIS QUESTIONS

What if we could we could create an immersive virtual reality environment to simulate a physical presence in real-world, community settings, and allow students to repeatedly practice targeted skills within a safe, low-stakes environment? 

What if students could have repeated exposure to community settings before going out into the community, without real-world consequences?  

PARTICIPATORY DESIGN 

After proposing the idea of virtual Community-Based Instruction to several occupational therapists, I submitted an Institutional Review Board Proposal (IRB), which was approved for a formative usability evaluation at Manhattan Children's Center (MCC), a school which offers instruction to students with Autism Spectrum Disorders. The approval of the IRB meant I would have access and be able to collaborate with a team of five occupational therapists throughout the design and development process. The goal of the formative usability evaluation was to involve therapists in the early stages of the design and development of the 360 VR experience through iterative testing, before the prototype was tested with students. After each usability testing session, I made improvements and changes to the prototype based on identified problems and recommendations of therapists.

INSTRUCTIONAL DESIGN

Prior to beginning design and development, I consulted with the OTs and had numerous conversations with subject matter experts about which community setting would be best to build as an initial prototype (e.g. a cafe, subway station, grocery store, etc.). Unanimously, everyone I spoke with advised me to build the grocery store first, as shopping is a large part of most independence and CBI curriculums, and since a shopping trip requires students to use such a wide variety of skills. 

Dr. Kristie Koenig, Associate Professor of Occupational Therapy at NYU and Principal Researcher of the ASD Nest Program, suggested using the grocery store scenario as an opportunity to teach Executive Functioning skills, an area of difficulty for many individuals with ASD. She provided me with examples of how each skill could be taught in the context of a shopping trip. Below is a diagram of Executive Functioning skills, all of which could be applied to the grocery store instructional activities. Beth Rosenberg, founder of Tech Kids Unlimited, also provided guidance throughout the instructional design phase, helping to identify skills to integrate into the instructional activities. 

USER RESEARCH

During the instructional design phase, the team of occupational therapists from MCC invited me to observe a Community-Based Instruction trip in action, which provided valuable insight about my target users, how a typical CBI trip is structured, and how tasks are scaffolded to teach targeted skills. What I learned during my observation is that the trips are incredibly structured, and by no means a field trip, as evidenced by the data collection form shown below. The therapists each had a clipboard and a velcro wallet for every one of their 15 students. As we walked around, they worked with individual students on specific tasks, based on goals in their Individualized Education Plan (IEP), keeping track of how much prompting each student needed. 

The CBI trip also provided me with an opportunity to observe and talk with students on the autism spectrum about what they preferred or disliked, and what was difficult for them about a shopping trip. 

INTERACTION DESIGN

After the initial collaborative design sessions at Manhattan Children's Center, during which instructional design and user research were the focus, I began to consider the interactions. I mapped out the experience of the virtual trip to the grocery store, from the time a user virtually "walks" in to the payment process at the cashier.

Below is a small section of a flow diagram with decision points and possible outcomes, which corresponds with the map above and illustrates the nonlinear structure of the 360° VR experience.  Each diamond represents an interaction point, where the user will be presented with a task within the VR environment, such as finding an item on the grocery list, asking a store employee for help, or choosing the correct amount of money to give the cashier. The user receives positive feedback or redirection from the narrator based on the selections he/she makes within the virtual environment using gaze input and the button on the Google Cardboard. Correct choices trigger a new 360 video scene, so that the student advances to the next aisle or task. 

The next step was to write a script for the narrator voiceover, as well as for the actors in the 360 video. Below are a couple small snippets from the original version of the script, which also corresponds with the map and flow diagram above. The script evolved after multiple iterations based on the feedback from the OTs regarding the language used and the difficulty level of the tasks.

For the original version of the script above, I received feedback from the OTs that I was using overly complex language, and needed to simplify the feedback and use more concrete language. This was an interesting discovery, as my initial inclination was to provide an extensive explanation or a lot of verbal prompting for the student users. However, I learned from the OTs that a strategy they use with many of their students on the autism spectrum is called "Least to Most" prompting, which means the teacher or therapist starts with the least intrusive prompt, and continues to deliver prompting as needed until a student performs the desired task. 

PRODUCTION

After many weeks spent working on design, it was time to begin filming the 360 video. Getting permission to film in a grocery store proved to be an unexpected challenge. After getting turned down by every store I contacted in the NYC, it was time to reach out to my network. I received permission to film at a grocery chain called Fresh Thyme in Chicago. The Fresh Thyme team was incredibly accommodating, allowing me to set up a little studio and even providing me with employees who were willing to be actors for the cashier interaction. 

I used a 360° camera called the Ricoh Theta S to record the footage, and also recorded audio at different locations throughout the store. Filming every 15 feet throughout the store without a team and trying to stay out of the way of customers was a challenge, to say the least, but after a week of shooting and recording, I returned to NYC with the material I needed to begin editing and then building the interactive VR experience in Unity.

CONTINUED ITERATION

I returned to Manhattan Children's Center for  several more rounds of prototype testing with the occupational therapists prior to user testing with students. Over the course of a few weeks, we reviewed progress and aligned features and user stories accordingly. For example, it was really important for the therapists to be able to see what the student was seeing, so that they could keep track whether a student is meeting his or her goals or needs additional prompting, something that they were able to realize only once they had interacted with the prototype. This feature was added (using a software called Reflector) and tested the next week.

STUDENT USER TESTING

The occupational therapists were incredibly important throughout the design process. However, I needed to get my prototype in front of students to answer my original question of feasibility. I was able to complete multiple rounds of user testing with students on the autism spectrum, thanks to a NYCDOE school and an organization called Tech Kids Unlimited, a nonprofit organization to teach kids with special needs advanced technology skills. So far, I have tested the prototype with 14 students on the autism spectrum, some during 1:1 occupational therapy sessions.  

The clip below shows real-time footage of a student using the prototype for the cashier interaction, which targets the skills of waiting in line, interacting socially with the cashier, and paying. What you see is the result of his actual head movements and what you hear are his unprompted reactions when the cashier speaks to him in the VR environment. 

The second clip shows a student using the version of the prototype which uses a static 360 image, as opposed to 360 video used above in the cashier interaction. The student is tasked with finding one of the items on his shopping list (wheat bread), which he successfully completes. When he selects the correct item, he receives positive feedback, and the item is crossed off his list and added to the cart. If he had selected the wrong type of  bread (white bread, for example), he would have received redirection to check his list and try again.

During the collaborative design sessions at MCC and during user testing with students, I often used A/B testing as a technique to compare two different versions of the same design in order to determine which one performed better against our predetermined goals and learning objectives. For example, in the two versions of the design below, I was testing the visual cues in the scene, and evaluating how much prompting the student would need to indicate which items in the scene are clickable. Through testing, we determined that Version A provided too much prompting, as it did not require the students to look around and try to find the correct items from their grocery list (one of the learning objectives), since the visual cues were so obvious. Version B, in which the Reticle enlarges slightly when the user's gaze is on a selectable item, provided less prompting and actually required the student to look around and search for the item he needed to find on the grocery list, while still giving him a clue that the item could be clicked on. 

User Experience METRICS 

During the user testing sessions, the variables I intended to test were the students' tolerance of wearing the Google Cardboard viewer, the level of presence experienced by students while interacting within the virtual environment, the quality of the user experience, and the occupational therapists' ability to facilitate and integrate the VR experience into instruction.

100% of students who tested the 360° VR experience expressed positive feedback, and 93% of students were able to complete the tasks within the virtual environment. 

The students smiled, laughed, and asked a lot of questions during the user testing sessions. One student did not want to take the Cardboard viewer off, as he wanted to continue looking around. Another requested that I build a "subway edition" of the VR experience so that he could practice buying a metro card and riding the train. I received an email from an OT with the following quote from a student a week after I worked with her students:

Students required various levels of prompting to complete the two tasks that were tested: finding an item on the shopping list and choosing the correct amount of money to give the cashier. We found that there was a bit of a learning curve, as with most new experiences. For the most part, once students completed the first task (usually requiring prompting) and figured out how to use their gaze and the button on the Google Cardboard to select something, they required little to no prompting on the second task. 

For the cashier interaction, many of the students actually responded out loud to the cashier when she spoke to them. This indicated to us that the students were actually experiencing a level of presence within the environment. One student who was extremely shy actually took the Cardboard viewer off to look away as soon as the cashier in the video asked how he was doing (we were able to see his view on our computer screen), and the OT prompted him to respond. He put the Cardboard back on, smiled shyly and nodded when the cashier asked him if he found everything okay. 

INSIGHTS 

User testing provided many valuable insights to guide the design of future iterations of the prototype. Perhaps the biggest realization was the need to differentiate the VR experience, so that the occupational therapists could have the ability to customize a student's experience depending on his or her individual needs. What I quickly learned after working with the OTs and their students is that there is no “One Size Fits All” solution when it comes to designing for students on the autism spectrum. Since there is a wide variation in symptoms among individuals with autism, the needs from one student to another can vary significantly. Below is a quote from a collaborative design session with an occupational therapist, during which she explained the differing sensory needs of the students she works with:

Though the majority of the students we tested the VR experience with did not have any difficulty completing the tasks within the virtual environment, one student was overwhelmed when we started by showing him the 360 video scene in which there was movement and sound (the cashier interaction). He was only able to wear the Google Cardboard for about 30 seconds before he took it off and said, “That was cool, but I think I’m done now.” We then asked him if he wanted to try a different kind of activity, and gave him a task to find an item on a list, which used a static 360 static image instead of video. He was much more comfortable, and was able to successfully complete the task once the movement from the scene was removed. This realization led us to consider the idea of scaffolding, or tailoring the VR experience to the needs of the student, with the ultimate goal being for students to work up to the full sensory experience. 

The next iteration of the prototype will give OTs the ability to scaffold the 360 VR experience, so that different sensory components can be added in according to IEP goals and as students begin to master skills. 

CONCLUSIONS 

So far, I have received an incredible amount of support from the educational community, specifically occupational therapists who work with students with ASD on a daily basis, encouraging me to continue pushing this project forward. All of the OTs I have shown the prototype to perceived the app to have potential as a learning platform for their students on the autism spectrum, and those who tested it during a 1:1 session were successfully able to facilitate and integrate the VR experience into their instruction. I am currently in discussions with multiple schools interested in piloting the app as a supplemental activity for curriculum developed to help students with ASD develop functional independence. 

After continuing iteration based on user testing results, my next step will be to measure whether students are able to transfer, or generalize, the skills learned in the virtual environment to the real world settings. My ultimate goal for this project is to build a series of instructional scenarios, or a curriculum module which would allow therapists/teachers to choose from a variety of community settings (e.g. a restaurant, grocery store, street crossing, subway station, city bus, cafe, etc.) and different Executive Functioning skills.