VR Evaluation

• Introduction
• Getting started with VR Evaluation
• Creating Quiz
• Creating Survey
• Pre/Post Question Events
• Creating User Defined Evaluation
• Distributing VR experiences
• Viewing Results

Introduction

Evaluation allows measuring the performance of the user by using

• Quizzes
• Surveys
• User Defined Evaluation

To have access to the evaluation tool you need to be part of an organization, creating an organization and managing users in an organization is done through SimLab Admin, which is included in SimLab Composer.

VR experiences can be distributed to end users using SimLab VR Viewer, or LMS (Learning Management System). Any LMS supporting xAPI (Tin Can) or Scorm can be used to distribute VR experiences to users 

In case of no LMS system is available SimLab VR Viewer can be used. It includes all that is needed to distribute VR experiences and VR-based Training. 

After users run the experience, results can be accessed using SimLab Admin. 

• Getting started with VR Evaluation
• Creating Quiz
• Creating Survey
• Creating User-Defined Evaluation
• Distributing VR experiences
• Viewing results

Getting started with VR Evaluation

To create a VR experience including evaluation you need to be part of an organization, which you can be invited to by any admin of the organization (each organization can have one or more admins), or you can create an organization using SimLab Admin

SimLab Admin, allows the admin to define and manage Admins / Teachers 

For creating lessons and distributing them to students, the user needs to be an admin or teacher in the organization
Admin has the extra ability to add and manage other users. 

The following tutorial shows the process of creating an organization using SimLab Admin, and adding new admins and teachers,

In addition to creating organizations and defining users, SimLab Admin allows the creation of Groups and Courses.

Group is a collection of courses that makes it easy to find them. Educational institutes can for example create groups Like Summer 2023

A Course is a collection of lessons that serves a purpose, it can be something like Math 101, or Safety introduction.

When a lesson is created it should be added to a Course, which is a part of a Group.

When creating an organization a default Group and Course are added, which can be renamed. Other Groups and Courses can be added also.

Creating Quiz

Each Quiz created using SimLab Composer may include one or two types of questions

1- Multiple Choice Question
2- Interactive Question

In a Multiple Choice Question, the student needs to select the correct answer from any number of choices (from 2 for true/false questions to any number of choices defined by the instructor) 

In an Interactive Question, the student should select the correct 3D model representing the correct answer 

For the Quiz the teacher can determine the time allowed for the student. If the student did not submit answers before this time, answers will be automatically submitted by the end time. The teacher has the option of keeping time open. 

VR experience can have one or multiple quizzes and/or surveys. The teacher can determine when a quiz or survey is started, the VR experience can show material then run a quiz, or it can start with a quiz or survey 

The following tutorial shows how to create a quiz using SimLab Composer 

Creating Survey

Each Survey created using Composer may include a mix of three types of questions

1- Multiple Choice Question
2- Interactive Question
3- Star Rating

In a Multiple Choice Question, the student needs to select one of the available choices

In an Interactive Question, the student should select the 3D model from the scene

In a Star Rating, the user needs to rate the experience (4, 5, and 10 stars) based on the designer's choice

Unlike Quiz, Survey does not have a correct answer.

But it shares with Quiz other features 

Pre/Post Question Events

Training Builder is needed to determine when to start a Quiz or Survey in the scene. Starting a Quiz and Survey can take place after the student learned new material, or even at the start of the scene if this is simply a Quiz VR experience.

This is not the only place you need to make Quiz or Survey utilize the Training Builder. For a special question, you may need to arrange the scene in a specific way or make sure the student looks in a specific direction, for those situations you can use Pre/Post Question Events.

This is done by clicking on the Pro/Post Question Even button on Quiz/Survey dialog 

If this is the first time the Pro/Post Questing Event button is clicked a new event is added to the Training Builder as shown in the following image 

This can be even used to execute all preparations needed for the question.
If it was clicked before it will open Training Builder and focus on the Pre-Question Event so the user can check it, or update it.

Post Question Events work in the same way, the only difference is that they are called when the user ends the question, so they are usually used to restore the state of the scene if needed.

Creating User Defined Evaluation

In addition to using Quizzes to evaluate participants in VR, you can create your own customized evaluation criteria through the Report Response.

The report is a Response in the Training Builder that can be connected to any event, and upon the occurrence of the connected event, it will submit a User-defined status.

^{In this example, the Report Response will be triggered once the user enters the object named "Table", and it will report the completion of "Continent", and submit the value of the variable "Capital"  as the facing continent, and the value of the Variable "Population" as the duration.}

---

Report Response Types

There are 4 types of reports that can be submitted using the Response Node :

1. Completed

The Completed Response will report the status "Completed" with an object name that would describe what has been completed, you can also attach a variable to the score field to report a numerical value, as well as a variable to the Duration field to report a time interval.

2. Passed

The Passed Response will report the status "Passed" with an object name that would describe what the user has passed, as well as the score when you attach a variable to the score field to report a numerical value.

3. Failed

The Failed Response will report the status "Failed" with an object name that would describe what the user has failed in, as well as the score when you attach a variable to the score field to report a numerical value.

3. Scored

The Failed Response will report the status "Scored" with an object name that would describe what the user has Scored in, as well as the numerical value of the score through attaching a variable to the score field.

---

Report Response Versatility

The Report Response, unlike Quizzes, is very versatile and can be used to report the status of any event throughout the VR experience. You can, for example,  have multiple Reports for each phase of the experience, or different reports for the same event but are triggered under different conditions.

^{In this example, 2 Report Responses are connected to the same event, but each will be submitted depending on the value of the variable "Total_score" which will determine whether the user has passed or failed.}

To learn more about utilizing the Report Response in VR, watch the following tutorial video.

Distributing VR experiences

Distribution methods

VR experiences created using Composer can be shared using SimLab VR Viewer, or LMS (Learning Management System) 
If your organization is already using an LMS, then you may want to utilize it to distribute VR experiences, if you do not know what an LMS is, or your organization does not have one then you can complete the task using SimLab VR Viewer

Distribution using SimLab VR Viewer

SimLab VR Viewer includes all the tools needed to upload VR experiences and share them with user(s) or groups. The process is described in the following tutorial.

Distribution using LMS 

Composer supports exporting VR experiences as xAPI (Tin Can) or Scorm packages, both xAPI and Scorm are standards utilized in most LMS systems, if your LMS system supports xAPI or Scorm simply you need to export the VR experience to the correct output supported by your LMS  

VR experience to xAPI

The following tutorial teaches the user how to export the VR experience as an xAPI package and how to use it in TalentLMS

VR experience to SCORM

The following tutorial teaches the user how to export the VR experience as a SCORM package and how to use it in Moodle 

Most modern LMS system supports at least one of the two xAPI and Scorm.
The following tutorial shows how to run VR experiences in BlackBoard (BB) LMS

Viewing Results

After students run VR experiences including evaluation, the teacher can use SimLab Admin to view results.  To do this, the teacher starts SimLab Admin, then logs in to his account.

From SimLab Admin the teacher selects the Group and Course under which the Lesson she/he wants to view its results, exists. 

By double-clicking, the teacher can view results for Quizzes, Surveys, and User-defined measurements

The teacher can view Firs trial, Last trial, or Highest trial (assuming multiple trials are allowed), the teacher can set a passing mark which is by default 50% of the full score, the teacher can export results to a CSV file 

The following tutorial shows how to use SimLab Admin to view the results