4.0 Calibrate

The final stage of sim development involves calibrating the sim to real users. Here, the sim is put in front of as many people representative of final users as possible, to look at how well the sim meets program goals and learning goals.

This stage can be bruising for the developers and programmers, so needs to be driven by the more objective project manager or sponsor liaison. Almost everyone involved with the sim need to be available and accessed for short bursts of rapid changes.

And while listed here as a distinct stage, in fact the real process of getting user feedback should be constant and ongoing from the first day of sim design. The more integrated the process of calibrating is, the less dramatic and expensive changes will have to be here.

Having said that, almost invariably, sim elements need to be better explained. Often, new (and even simpler) levels need to be added to the front end of the experience. New tips and other pedagogy have to explain why things are happening the way they are.

Meanwhile the gameplay has to be refined to make things easier or harder. This often just involves the tightening up or loosening up of some key variables, such as timers and enemy AI's.

About 25% of the project budget is spent here.

Some initial steps include:

• Step 4.01: Fix, Add, and Subtract Characters, Scenes, and Pedagogy
• Step 4.02: Calibrate SCORM or other tracking compliance
• Step 4:04: Integrate with Learning Management System (LMS)

Step 4.03: Package the Support Material

Sims often have support material.

In some cases, such as self-paced sims either in the form of viral casual games or even take-for-credit educational simulations, this material may be in a traditional web site.

• Bibliography for the simulation: the answer to the question: how did you come up with the content for the simulation?”

• Further reading: a list of books and articles where an interested person can go to get more information on the content overview in the sim.

• Case or other packaging.

• Next steps: if someone is interested in the topic, what can they do?

In sims that are designed for class deployment, material for both instructor and student can be critical. Here are some examples:

• Slides and talking points: fully usable and editable decks of slides, with minimum formatting, they can be inserted into any class. This can include walk-throughs on how to install or access, use the interface, use any prepackaged self-evaluation tools, and submit results where necessary.

• Details on the underlying system: simulations are often made up of rules and equations. Which rules are included, and any relative weighting, should be included.

• Technical support frequently asked questions: a list of the top 20 technical problems, with answers, culled from past deployments.

• Examples of best, typical, and worst plays: either detailed screenshots and text, or video clips, showing a range of plays, with both annotations and analysis of actions.

• Tips for debriefing: a guide to how to debrief the simulation, if necessary. Typically, soft skill simulations require two to three times the amount of space in the manual dedicated to helping instructors with debriefing then more technical skills.

• Cheat codes: a list of keystrokes or other hidden ways for someone to "cheat" in the simulation, such as jump ahead, have unlimited resources, see equations or variables that are otherwise hidden, play any video clip, or see alternative paths.

• Modding tips: a list of places and examples where the simulation is easy to change.

• Examples of syllabi: three or four examples of places in typical course outlines where other professors have used the sim. This material can typically be edited as well.

(List from my book Learning Online with Games, Simulations, and Virtual Worlds.)

In some sim deployments, communities such as chat rooms are co-deployed. These should be set up during this step as well.

These support materials tend to fit in with and take into consideration that sims take users through cycles of frustration and then resolution. Here is a chart that maps predictable waves, with an optional framework at the bottom when used in a classroom context.

Step 4.05 Pilot for Usability

A Sim from the world-famous Acton Business School

The sim is put in front of people representative of the final audience with the instructions for them to engage. The goal is to test the usability of the sim - can students find their way around? Can users do what they want to do? The sim designers should not even to be in the same room, using video tapes instead.

While it is too late at this point to change much about the sim, there is opportunity to alter or add text, such as tips and instructions. This step is much easier if all throughout the process test audiences and other focus groups have reviewed key decisions, such as graphic representations and story.

Step 4.06: Rerecord talent where necessary

Bring in as much of the original voice and acting talent as necessary to rerecord parts of the sim that have been changed.

This is the most miserable of all steps. It is expensive, hard to get right (sound, for example, is a challenge to match up, even for the same actor, between one recording session and another), hard to schedule (the original talent may not be conveniently available, or may be sick, or...), and often feels unnecessary when there are just a few lines to be changed.

Knowing how miserable this step is leads one to do a lot of things better the first time, such as: making sure the dialog is right, recording some alternatives, and using vague phrases instead of specific.

Step 4.07: Pilot the Sim to Test for the Achieving of Learning Objectives

Find a sample group representative of the final audience, put them through the sim in as accurate to the final deployment as possible, and then test them afterwards for new conviction and competence.

Of every step and every audience, this is the group to rigorously test. Use intrusive and possibly expensive methodology, including 360's and interviews. If the learning goals are met, move on the the next step (make any small changes if necessary). If learning goals are not met, make necessary changes and redo this step.

This is also the stage to test, if instructors are used, material and processes to support them.

Step 4.08: Create and Present Marketing Material

Create and make available the final marketing material to promote the sim. Here are some avenues and approaches.

• Gather photographs and video clips of the sim. Get endorsements from leaders. Record people using the sim from the pilot groups.

• Show the sim to members of the appropriate press, be it internal or external. Have a demonstration level to present to that target audience.

• Put the self-contained "entice mode" online. People can watch it and get excited.

• Create a trailer, much like a movie trailer.

This final marketing material can be presented on or linked to from the web site to garner buzz.

Avoid a common mistake by: Promoting the sim heavily. Never assume that just because it is great, people will find it and use it.

Step 4.09: Put on Final Server

The sim has to be installed on the final deployment server.

The database integrator must ensure connectivity of the sim to the LMS. Further, any media files, such as video or sound, must be configured and optimized. The content should be pre-cached around the world if necessary.

This step requires a working relationship with the IT department that cannot be assumed in most organizations. If the sim is not considered mission critical with high level support, this step can take a lot longer than it should.

Step 4.10: Test Sim with a Subset of Final Population

Conduct a relatively quick dry run of the final roll-out. This should be done twice.

First, make the sim available to a representative subsection of the final target population for enough time to enable them to use it in an accurate context. For example, make the sim available to a test class or community and let them work through the whole thing. Look for any final technical problems and any little content fixes.

Then, make the sim available using the final configuration for the full deployment, but at an obscure time and for only about twenty minutes. Have colleagues around the world test it, for example, if that is the ultimate range. Look for compatibility with different computer configurations.

Stress test the final server configuration of a sim using at least 20 concurrent users, plus people accessing it from all over the world (if that is the target audience). Especially look at the performance of rich media.

Step 4.11: Deploy Sim for Final Population

The World Anti-Doping Association's Play True Sim

The single biggest milestone in a sim development project is making the sim available for the final population. This can be to a target community in an enterprise or the entire world.

No matter how much planning and testing was done, there are always surprises, including server stress issues, database issues, language misteps, and browser incompatibility issues.

It is best to have a full, ready-response team for the first 24 hours. This team needs to be able to troubleshoot problems in real time. This is also the time to test tracking abilities.

Avoid a common mistake by not planning vacations for the week following the release of the sim, expecting work to be over.

This step also kicks off any marketing or other communication programs, usually with quotes and other user thoughts taken from the pilots. Where sims are required, this also has to be communicated.

Step 4.12: Gather Metrics around "Simulation Use and Effectiveness" in Full Population

Once the sim has been deployed, either through a Learning Management System or web portal, it has to be tracked. Usually, this means different categories.

• Access and use
• Successful completion
• Effectiveness

Access and use are easiest, of course to measure. Any system will be looking at usage, number access points, number of unique visitors, how far they get, when they return, and so on. In some cases, the number of exposures to key messages may be measured. The rate of growth in the amount of people accessing a free sim, such as a serious game, can by itself measure how successful/viral it was. The growth in volume of a paid sim obviously is critical for the business model, and tracked through either an LMS or CRM. This information can be used to tweak any social media strategy, if one is used.

Completion is also fairly easy to track and report. Typically, a LMS can determine if an employee made it through a required sim. (Lack of completion for someone who was required to take the sim can kick off a reminder activity, again hopefully executed by the LMS.)

But measuring effectiveness is a significant new activity, often minimally funded. Where 360's are involved, the sponsoring group may have to nag managers, for example. Pre and post assessments may be used. Meanwhile case studies have to be gathered and edited. Minimally impactful tools can be used to approximate effectiveness, such as this Bristol method. To circumvent this, learning organizations may determine the level of effectiveness during the pilot of limited implementation, and then just extrapolate that based on access or completion numbers.

Results from One Sim Deployment (Source: Case Study in The Complete Guide to Simulations and Serious Games), Completed Five Months Later

Step 4.13: Create the Final Report

The last step in a sim project is to write up the final report. This report can be between 5 and 20 pages, and can include:

1. Introduction
   
   
   
   • Importance of subject matter
   • Goal of the project
   • Brief summary of sim development and deployment
   • Summary of results
   
   
2. Background
   
   
   
   • Old process
     
   • Justification for sim approach (including third party citations)
   • Learning goals
   • Program goals
   • List item 2
     
   
   
3. Development Process and Milestones
   
   
   
   • Content Model (including footnotes where necessary)
     
   • Interface, Systems, and Goals Decisions
   • Level and Story Design
   • Surrounding Content
     
   
   
4. Results and Assessment Methodology
   
   
   
   • Decisions around assessment
     
   • Implementation
   • Hard data (often as much around pilot as final implementation)
   • Analysis of hard data


5. User Comments and Anecdotees
   
   
   
   • Short responses
     
   • Longer anecdotes
     
   • Photographs of users
   • Press or other coverage
   
   
6. Best Practices, Lessons Learned, and Suggestions for Future Versions and Sims
   
   
   
   • Things that went well
     
   • Things that did not go well
   • Suggestions for improvements
   • Suggestions for new sims
   
   
7. Conclusion
   
   
   
   • Summary of approach
     
   • Final thoughts
   
   
8. Appendix
   
   
   
   • References
   • Research
   • Screen shots
     
   • Survey questions
   • Relevant raw data
   
   

This document can range from hard-nosed analysis to fluff marketing piece. The reality is that it should be somewhere in between.

Step 4.14: Patch and update the Sim

At various points after the launch of the sim, fix and add functionality.

This task can be significant, as updates to hardware and software, including graphics cards and even Adobe Flash, can render working sims unworkable. Even new security features of operating systems can prevent the operation of key components.

A 3D Graphic Problem that Came Up After Sim Release, Based on a New Graphic Card

Similarly, small pieces of data, including text and visuals, may change in an organization (such as a new product number), rendering the sim impaired for content reasons. Again, organizations must maintain the ability to change little pieces of data for the life of the sim.

If working with a vendors, the contract should specify maintenance durations.

Avoid a common mistake by: Having funding or access to key talent to maintain a sim for at least six months after the release.