2.0 Create and Design

Here, the sim is designed. The goal is to produce a great design document and/or sim bible, between 30 and 50 pages long.

The designer immerses him or herself in the content, looking at established best practices, lots of tiny relationships, and then relevant existing sim genres. The learning objectives and requirements are formalized, often using a client liaison and program sponsor. The look and feel are nailed down, hopefully with the work of a good graphic designer. Any technical decisions, including media, authoring environments, and end-user requirements, are nailed down

Step 2.01 to 2.03: Identify Learning Goals and Program Goals

Identify and gain consensus on goals and metrics. I prefer to think in terms of a balanced scorecard. Here are the most important categories:

Learning Goals

Simulations have to rigorously and predictably develop learning objectives in students. But the learning objectives take on a different form. I think in terms of:

• Competence: Students can use appropriate actions in a situation.
• Conviction: Students have a passionate belief in a typically invisible or counter-intuitive system. This belief is enduring and flexible. If a student learns conviction, they will improvise. They cannot be easily dissuaded. They have commitment to the ideals, not just compliance. Teaching conviction is so much more powerful than other learning objectives, but it is very hard to measure objectively, and it sure falls through the cracks of traditional Kirkpatrick methodology.

Program Goals

It is in the program goals, however, that most otherwise great sims get caught up. Here they are:

• Engagement: Sims have to be engaging to students. To many people, that is their "reason for being." I would not argue they have to be fun, but they do have to be fun enough. More than fun, they have to be immersive. Students can't be looking at the clock.
• Convenient: The work around accessing a sim has to be less than 5% of the total participation time for a student. They increasingly have to be accessed completely online. And they have to be short - about an hour for corporate employees, although easily ten or fifteen hours for academic environments. And they have to be well chunked.
• Acceptable cost per student: The cost per student has to be reasonable for the learning objectives, typically around $100 per corporate student and $45 per academic student.
• Acceptable time to creation and delivery: The time from approval to deliverable has to be around 4 months for a corporate environment, although up to one year for other (wiser) environments.
• Comfort level of implementers and sponsors: Finally, and most amorphously, the sponsors and the implementers have to be comfortable with the material. They have to support it, understand it, and push for it. They have to believe in the material so completely that they will not accept failure.

Identify metrics for all of the above.

Step 2.04: Create a High Level Budget and Time Frame

Prepare a high level budget and time frame for development.

How Long Does it Take and How Much Does it Cost to Create a Custom Serious Game for an Organization? What does a simulation cost?

The most effective sims, single-player and Adobe Flash based, will take about six months to create from scratch, and take about 45 minutes of student time, and cost about US$100K.

However, there are significant modifiers that are cumulative to both decrease and increase the default time and cost. For example:

Multiplayer Instead of Single Player:	Multiplayer in Addition to Single Player:
-25%	+60%
Very Light Weight Mechanics:	3D Client Installed “Game-Like”:
- 70%	+ 100%
Reuse Established Approach:	Total Creation of New Genre:
Complete adherence to an existing genre: -60% Engine already exists: additional -50%	Completely new genre: +30% Flexible and reusable architecture: additional + 20%
Number of Critical Decision Makers is less than 4	Number of Critical Decision Makers is greater than 10
-25%	+100%

Steps 2.05 to 2.07: Optionally, Bring in an Outside Vendor

If an organization is to bring in an outside, end-to-end vendor (including Clark Aldrich Designs, or a partner such as WebCourseWorks), then the three process steps are:

• Step 2.05: Put out an RFP.
• Step 2.06: Be pitched a big idea.
• Step 2.07: Pick an end-to-end vendor.

Step 2.08: Set up Sharing Infrastructure and Meeting Schedule with Shareholders

A critical but often under-appreciated step is setting up an infrastructure and meeting schedule with shareholders for the life of the project.

• Infrastructure can include a shared repository for content, including assets and a calendar. This can include Google Docs.

• Meeting schedules include conference call numbers, web tools, and meeting times. There should be a meeting with the client of the sim at least every two weeks, and ideally every week. These meetings can take between 30 minutes and one hour, and will involve reviews of progress (on both sides, as often the client will have to gain their own consensus on material as well) and some joint decision making. Not having this meeting will extend the production time of a sim by at least 20%.

Both of the above may be tiered. For example, there will be more frequent meetings and a dedicated repository for internal teams.

Step 2.09: Review Existing Training Materials

In this step, collect all of the top-down patterns that have already been created, including established analysis, best practices, and rules. In this step (and maybe only in this step), traditional educational content and linear material, if they exist, such as courses and curricula, books, reports, famous or inspirational quotes, and rules and policies are very helpful. They also serve to set a scale for what the sim will and won't cover.

Step 2.11: Interview Subject Matter Experts

Interview the relevant subject matter experts.

Interviewing subject matter experts for educational simulations can be tricky. Here are fifteen generic questions:

1. What situation that you experienced epitomized the subject matter? (This could be a real time meeting, or an event that took place over weeks, months, or years.) Were there multiple situations?
2. What were your available options? At each moment, what could you have done in that situation, and what might a naive or inexperienced person done? What did you end up doing?
3. Why would the naive approach fail? What would it not have taken into account?
4. What were clues that you saw that informed your knowledge of the situation? What did you see immediately, and what information for which you had to look? How did you look?
5. What did you want success to be? What did the conclusion end up being?
6. What were you looking for to suggest that things were going well? What were you looking for to suggest that things were not going well?
7. What were the "maintenance" or routine activities that you had to do (even including body language)? What would happen if you did not do them?
8. What was the moment were you knew you were successful? (or not.)
9. What was each person's best case and worse case outcome? What were their strategies and actions?
10. What would have been three to five legitimate alternative approaches to the problem or situation?
11. What were the three to five high level metrics that you were monitoring? Time? Commitment? Alignment?
12. What trade-offs were you willing to make? What trade-offs did you make?
13. Can you graph the high level metrics over the course of the experience?
14. What were the inflection points for each?
15. How do the actions impact the high level metrics? What else impacts the high level metrics (be as specific as possible)?

Step 2.12: Build out the Simulation Model

The next step, after all of the traditional rules and analysis are collected, is to uncover the hundreds of tiny relationships that represent the simulation elements. Simulation elements model some part of reality, and the role of someone in it. More specifically, simulation elements are content that describe:

• What Actions are available;
• How the actions impact relevant Systems; or
• How those systems produce feedback and results.

Further, simulation elements are typically:

• Interactive/dynamic;
• In some way incomplete/abstracted; and always are
• Driven by learning goals or research questions.

Here's an example of the connection between the three layers of simulation elements:

If a riptide is pulling a swimmer out into the ocean, the action available to the target person is swimming and the desired result is getting to shore. But because of the system of ocean currents, the best action for our target to get the result is not swimming towards the shore (against the riptide), but at a right angle to the riptide first to get out of the current, and then swim to shore.

I visualize the three layers as such:

Actions

One set of tiny relationships is around actions. Yell. Beg. Put tongue A in groove B. Invest money. Run. These are all examples of actions.

For actions, here are the biggest questions: what are the seriously considered options available to an expert? Then, what do naïve people do? Can they be defined very specifically, down to exact quotes or levels of magnitude?

For example, imagine we were creating a sim around end user computer security. some of the actions available are: a user (when getting an email) can follow an embedded link to a web site (and then perhaps enter personal data), open the attachment, forward an email to a friend, log in or not, even install the suggested program. Or they can try to figure out if the email is legitimate or not. Or they can delete the email, or perhaps report it to their manager or IT department.

Results

The second category of a sim on which we need to focus is results. For results: we ask, what does success and failure look like? Is it all or nothing, such as the accomplishment of a mission? Or are there three or four things that a person is trying to balance and grow? Or is success in the sim (as well as from the sim) the ability to consistently apply an increasingly complex set of competencies?

Identifying the results of failure is more interesting, more important, and more counter-intuitive for most instructional designers than identifying success. We have to figure out, what are the various types of failure one can experience, and what are the situations that lead to them? What are the immediate wrong things to do, and what are long-term failures?

Systems

The final set of relationships to identify is that around systems. Systems, practically defined, are what get in the way between actions and desired results. If the collection of all sets of tiny relationships are an iceberg, the systems are the part of the iceberg that is underwater - often a huge hidden mass.

Here are two quick examples: When playing Chess, a person may want to capture the other player’s king, but the systems of rules and positions and the activities of an opponent on the board need to be navigated and overcome (which is of course what makes it fun and interesting). Meanwhile, in leadership, we may want to build a great team, but the rules of accomplishment, personal egos and motivation, and reward need to be navigated (which can be fun, but more likely frustrating).

For systems: some questions might be: are there processes or mazes that have to be followed? Are there opponents that are striving to keep the person from being successful? Are there hidden processes that others are following (in our computer security example, bad guys may be taking scraps of personal data and crafting highly targeted profiles for scams)? Are there cycles or balancing loops or feedback loops? Are there delays? Are there some mathematical relationships?

See more on systems here.

Many of these relationships are so simple that it feels absurd to even capture them in a document. But there power comes in their rigor, volume, and integration.

This step must come after the framing of the scope of the sim.

It is a worst practice to, in the research process, start here. This worst practice often occurs when either a researcher or subject matter expert starts an effort. Identifying the little relationships without the framing of the best practices and scope of the sim is a staggeringly complex activity, which while satiates the purists, can take huge amounts of time and overwhelms all but the most intrepid. Projects that start here seldom see the light of day. Even if they do survive, there is so much wasted effort.

Step 2.13 Identify Genre and Techniques

We have already identified the high level rules and the mounds of tiny relationships. Now, find an existing simulation or game that comes close to the framework or spirit of some or all of what you want to accomplish (if you possibly can).

Is it a first person shooter? Tower defense? Branching story? Then, borrow the established format as much as possible. If an engine exists, such as Second Life or Adventuremaker, figure out how to use it (this can save more than 80% of the development time). Regardless, use the gameplay and level design conventions. In many cases, you will also draw models from other genres as well, glomming them together. Even the most narrow toolset allows for importing of great ideas.

There are times when there are no appropriate games or sims, let alone toolsets. In these cases, find a perfect example or microcosm that can serve as the model for the interaction.

This step must come after the development of the sim content model.

A common worst practice is starting with the identification of the genre or engine and filling in the blanks, before a complete sim content model is identified. I often see this when either a vendor has a pre-built engine they are using for a new project, or when an organization has invested in a platform or authoring environment themselves and are trying to push more programs onto it. The results is quick (weeks instead of months), cost effective, and efficient development. The course is spit out on time.

The problem is that the content is flat. Designers end up merely reskinning rather than teaching anything of note. Two or three different programs, ostensibly covering different topics, starting from the vantage of same engine all look the same, and more importantly, basically "teach" the same thing.

We are seeing this in abundance with sims in Second Life, but also from small specialty vendors. From a business perspective, this makes sense for them – the vendor’s internal cost, time frame, unpredictability, and quality of talent needed are five to ten times greater if they are creating a new engine than using an old. But it can result in a substandard or forced student experience.

Note: The Importance of Playing Games for a Sim Designer

There is always an uneasy relationship between sim designers and computer games. Computer games represent simultaneously a vision, a trap, and the embodiment of a near-damning misrepresentation.

Still, it is up to any sim designer to play quite a few computer games, at least a few hours a week. We must look at and understand the current repertoire of mechanisms, including complexity and functionality of interface schemes, solutions to visualization challenges, even level and game length.

I like to look at popular games, critically acclaimed games, and examples of new genres. To accomplish this, I try to download as many free demos as I can for different systems, including iPhones, PCs, and PS3's. Even the old coin-operated arcade games have a lot to teach us.

Step 2.135: Look at Sim Authoring Tools

Clients often hire me to evaluate e-learning and other content authoring tools. They rightfully see authoring tools as they key to the knowledge kingdom. Authoring tools transfer the critical ability to quickly create or modify content from an external party to internal. Further, the tools selected by an organization ultimately shape the types of content captured and distributed.

I have developed a framework for assessing authoring environments. Here's what it looks like (and click to enlarge):

The label descriptions are as follows:

First, all authoring tools have affordances. These are things that it can do:

• easily and to great effect (so-called sweet spots, in the upper right of the chart),
• easily and better than other tools (efficient)
• with difficulty (hard-to-do) but to powerful ends,
• not as easily as other tools but sufficient in context (tangential areas), and
• not at all.

Hard-to-do almost always has cost implications (simply being more expensive), but also in some cases has stability and stacking (one project might only allow three hard-to-do things, for example) implications. Having said that, hard-to-do also presents an opportunity for content vendors or individuals to differentiate from an otherwise vanilla acceptance or rejection of tool functionality. In other words, if it is built into the tool, anyone can do it.

Second, we must look at the tools ability to deliver both learning objectives and program objectives. Learning objectives are new capabilities developed in the students (or otherwise end-users) through the media, and thus may include specific skills or broader situational awareness. Program objectives, in contrast, are requirements of the sponsor and the students of the media outside of learning goals, such as high engagement, low cost, or ease of deployment.

It is through both of these lenses that we have to consider authoring tools and engines.

Step 2.14: Synchronize and Mesh the Content Model

Now, we have to bring everything together. Use the rules to organize the tiny relationships, and then use the genre from step three to frame everything. You will work in from the three corners to the middle. Ultimately, all three should converge (even if there is fear at first that they won't).

Reconciling Broad Rules and Tiny Relationships

During this process, we start seeing plenty of places where the broad rules and the tiny relationships do not necessary align.

One example is when you are given a list of different possible successful approaches, especially when given superficially illustrative examples. For example, (the old training might read):

To influence someone, a leader can tell someone what to do, but the leader can also bribe then, threaten them, appeal to their sense of purpose, ask them as a favor, or make a logical case for a request. To illustrate (the old training might further read): consider a documented case where a CFO was asked to postpone her retirement, and the new CEO was successful because he appealed to her loyalty to the company.

This may be a sufficient for a PowerPoint slide, but like philosophy, it begs more questions than it answers for a sim designer. Things that need to reconciled include:

• Why did the expert (the CEO) use that approach? Was that his favorite influence strategy? Had that worked before with the CFO?
• Did the CEO consider two or three different approaches, and what was the criteria that won out?
• Did the CEO switch approaches midstream, and if so, why?

At a higher level, I ask, is there a common underlying model of tiny relationships that aligns most (I am not naïve enough to hope for all) of the identified approaches?

There are other situations as well, you will find when sifting through the body of linear content, when different experts have different and contradictory pieces of advice. A classic contradictory construct: Are you turning the other cheek (good) or are you appeasing (bad)? As with the above situation, a goal is to find common mechanics that allows for both.

Creating Strategic and First Person Perspective

The most effective sims use two or more parallel and mutually-reinforcing perspectives. This approach is consistent with generations of computer games and flight simulators. These have traditionally featured a first person perspective and a strategic (aka a radar or "mini map") perspective. The protypical example is of a driving game, where the screen is used to show the world from the driver perspective looking out at the highway and other nearby cars, while also showing a tops-down perspective on the entire track with all competitors. Players made decisions based on both perspectives simultaneously.

The first-person perspective presents the actual decisions that the student will see and make in the real world. This often involves interpersonal conversations. The strategic perspective presents the “big picture” and involves a visualization of a system and interactions often invisible in the real world.

Other Steps in the Synchronization Process

As one closes in on a final design, some tough questions have to be answered. Here are some.

How broadly can the identified actions be abstracted? For our computer security example, all of the actions, in both the target set and the contextual set, can be generally abstracted into the core three actions of accept the incoming request and act upon it, probe the request, or reject the request.

This is critical, as most sims work best in real time, where the computer does not wait for the student. Ideally a few actions are applied repeatedly, in different orders and sensitive to timing. Further, abstracting actions can increase the applicability of the sim to wider groups.

Another type of problem we have to answer: how does the sim handle little failures? Being inappropriately aggressive to a subordinate, for example, is a bad idea in a leadership sim. But does it stop the whole sim? In real life, plenty of successful people have little slips. Is it cumulative? (I am jumping ahead to Step Three, now but….) Arcade games often had a “three lives” model. Is that appropriate?

You may have a few outlier rules at the end of the process that fall outside of the system and level designs that you have created, but that still need to be included. Here you might use traditional pedagogical technique such as slides or pop-ups to convey this content. But hopefully this is minimal, and you know you have done a good job when all three perspectives support each other rather than grind. And often, amazingly, you will gain unique and industry valued perspectives through this process.

Step 2.15 Produce Rough Schematic/ Walk Through

Create a visual representation, storyboard, and walk-through of a part of the sim. This is done both to think through the engagement by the designer, but also show outside people to build support.

To build support at this stage, what you have to do is cheaply show something that will be animated and interactive when done, but all in a form that is neither now. This is fraught with risks. If you the show something too simple (i.e. something early on in the sim), the audience might "get it" but note that it is too simplistic. Or, you can show something from the middle or end of the sim which shows complexity and nuance, but the audience will typically grumble that it seems too complicated and no one will ever get it (or my favorite, that the members of the committee get it, but the target audience won't).

I prepare a fifteen to twenty slide PowerPoint slide storyboard.

The first 10ish slides showing a very gradual build up to the first point of student/learner interaction. I storyboard the animation to show how a few things will move and interact. My success criteria for this part is to make it easy for anyone to understand the simplest possible moment of interaction and to leave them actually wanting to engage the simple sim.

Then, in the last section of slides, I like to jump ahead and show a couple of screenshots of moments from deep into the simulation, often with annotations. Here, I risk overwhelming the audience with complexity in order to impress them with the eventual options available to the student/learner. I have to trust that the reviewers will trust my design to get student there gradually.

(From a technical perspective, I draw all of the interfaces in PowerPoint. On most contracts, I then use a graphic artist to turn my images into something that actually looks good.)

Here's an example. (And this works better as a presentation. But I think and hope if you stick with me, click on the images, you will get the feel. You should be able to understand this if I have done my job, because any presentation has to be able to be understood by people who were not at the meeting.)

First, s-l-o-w-l-y introduce the general look and feel of the interface:

Give a simple example of interactivity:

Show some feedback examples:

Then start showing the more advanced levels.

If multiple perspectives are being proposed, introduce them about here.

Finally, wrap up with all of the great ideas that you forced yourself not to visualize in the proposal for fear of overwhelming the audience!

Step 2.16 Produce Concept Document

Create a 4-6 page concept document that describes the basic look and feel of a desired 30 minute to 1 hour sim. In this concept document, include:

• Introduction
• Learning objectives
• Set up/Story
• Basic gameplay
• Basic screen interfaces and interactions
• Basic feedback and reward system
• Overview of underlying mechanisms from a descriptive more than technical perspective
• Basic level structure.

I use a lot of rough PowerPoint sketches as well as existing computer game screen shots as mock-ups. While not pretty, they do show all of the basic components. (If the project is for a proposal, clients may invest three hours of a professional artist/graphic designer to make the three or four screen shots look better, and about two hours for someone to proof and format the entire document.)

For more information, see The Complete Guide to Simulations and Serious Games.

Step 2.18: Establish Story, Characters, and Settings

If the sim needs a story, complete with characters and settings, establish them here.

All of the generic advice for story telling and character creation apply here. More specifically, the story should be established such that the player cares about success during the interactive part. Emotion is necessary for learning, and story, characters, and setting are very useful tools.

Logistically, it is often useful to cut out photographs from magazines to create composite characters.

What is the Role of the Player?

Before creating a story, one has to think about the role the players will take. The biggest question is: will/should a person play themselves?

In this area, like so many, computer games are not overly useful role models. In educational simulations, there are many factors to consider.

• In a corporate environment, people like to play as close to themselves as possible; they want their roles in the sim to match their own situations.
• However, it is easier to program a sim if you can specifically define the character and situation.
• In fact, if you want the sim to be job, role, or context specific, the permutations can be in the hundreds for the typical large organization. Meanwhile, even figuring out five different "skins" in a single sim can be difficult.
• One solution is to use text as much as possible for "skin" level customizations. For example, a player can type in their name, and have that appear where appropriate in word balloons or emails.
• Regardless, you really want players to choose genders, if gender is in anyway presented.

What I often try to do is to create a single story where the character is dropped into a new situation. For example, in an office environment, I might create a new job that the player has just accepted, or even a temporary new assignment where they are filling in.

For a teenager, I might create a situation where the person has just moved to a new school or camp. In these cases, the player does not have to accept a fictional past, but they do get to buy into an alternative future. In one sim aimed at teenagers, I created an alternative mother and father figure (a friend's mother and coach), who did parent types of activities (feed them, pick them up after games) that we could visualize and go through plot arcs. We could then play with these characters without breaking the past of the player (replacing their real-world parents).

Creating the proper identification with the on-screen character from the player is more art than science. But it is necessary, and those that do it well focus at both the "skin" and "tactics and strategies" level.

Step 2.22: Bring in the Final Programming Talent, if not yet done

By this step in the process, the final programming talent must be brought in, if not done previously.

Description of Responsibilities: The lead programmer is responsible for creating all of the code for the sim. This includes prototyping, piloting, creating any authoring or editing environment, and creating a finished simulation. In many projects, this category also includes the different skills set for evaluating and adopting third party technology and tool sets.

They need combinations of (from most to least important):

• Adobe Flash,
• ActionScript 3
• HTML
• CSS
• SCORM compliance
• JavaScript
• DreamWeaver
• Captivate
• Photoshop
• Illustrator

Percentage of Entire Project: 20-30%

One way to find this talent is through the schools that graduate game designers. Lists can be found at both IGDA and Gamasutra. Some of the most notable in the US are

• DeVry
• DigiPen
• Full Sail
• Guild Hall at Southern Methodist University
• Southern Michigan University
• University of Central Florida- Masters and Doctoral degrees in Modeling and Simulation
• Old Dominion University - Masters in Modeling & Simulation - College of Engineering
• Rochester Institute of Technology - Game Design and Development
• Ohio University - Digital Media
• University of Baltimore - Simulation and Digital Entertainment
• Michigan State University

There are also slighly less direct paths.

• There are game centric job boards, such as Gamejobs, and recruiters, such as GameRecruiters.
• There are advertisements in game magazines and game sites.
• One can also look at social networking sites, such as LinkedIn, game blogs, and game industry specific forums, again such as IGDA or Gamasutra.
• Finally, one can look at indie game producers. They know how to pinch a penny, and can focus on highly interactive experiences.

The biggest caveat is that a game designer does not necessarily make a great serious games designer. First, make sure the experience matches the proposed solution. If you want to make Flash based games, for example, don't bring aboard someone used to working in Triple-A level 3D environments (or, on the other side, Flash designers who have focused on highly interactive web pages). Then, make sure the passion is aligned. If one wants to create games where heroes save the world, they may not want to work on serious games. But if they really want to be a hero and save the world, this might be perfect.

Step 2.23: Produce the Design Document

Produce the 30 page design document. This document summarizes all of the key attributes and decisions around the sim. It includes:

• Overview and Rationale
• Intended Audience
• Learning Goals
• Program Goals
• Core Gameplay (including screen shots and storyboards)
• Level Breakouts, Sequence, and Timing (including story, characters, trailer/entice mode)
• Evaluation and Metrics Strategy
• Distribution/Technology Environment/SCORM and LMS Integration
• 508 Considerations
• Milestones/Project Management (including what has been done so far)
• Bibliography
• Appendix: High Level Overview and Examples of Mechanisms, including state charts and equations
• Appendix: Analysis of Target Content

Step 2.25: Finalize Assessment Strategy

Create the final assessment strategy for the sim. This will "prove" to third parties if learning has or has not happened during the deployment of a sim.

Typically, a sim deployment is comprehensively evaluated during the pilot, and then only indirectly evaluated during ongoing roll-out.

There is a theoretical and academic model for testing:

• Create two large random groups.
• Put one group through the sim.
• Objectively measure the the target actions, either in a lab or in practice, from the two groups and compare.

However, there is no perfect real-world assessment strategy. Each approach has compromises around decisions, such as (This list from The Complete Guide to Simulations and Serious Games):

• Post only or also pre? Do you evaluate what the students started off knowing, or just what they know after the end of the program? Any kind of certification program (programming, sexual harassment) just cares what the student knows at the end of the program. But any Sim program manager also wants to know what the magnitude of the shift is.

• Control group or just students? Do you evaluate a group of people who did not go through the program, or just the students who did go through the program? Do you evaluate students who went through an alternative program? When multiple components (such as mentors and communities) are used, is the impact of each permutation measured?

• Students only or people around students? Do you just use the students’ own knowledge or insights, or do you tap the knowledge and insights of the people around the student (so-called 360-degree reviews)? Students themselves tend to overvalue new knowledge and undervalue new actions, while the communities around the student are exactly the opposite.

• Indicating or direct measurement? Do you look at some sort of objective measure? Success of student? Retention of student? Promotion of student?

• Evaluate right after the program, or weeks or months out? Do you administer any postprogram evaluation the moment the program is over? Again, this radically overemphasizes the fast-decaying new knowledge acquired, while making impossible the evaluation of new behavior (unless the program deployment itself has occupied several weeks, so new behavior might have already shown up). The further out you go (one week, five weeks, five months), the greater the value of any recorded impact, but the harder it is to get results (compliance rate goes down), the harder it is to use those results to shape the evolution of the program itself, and the more other variables get in the way. The larger the group involved in evaluating the student (especially with 360-degree feedback), the more time you need (if someone only meets with the student twice a month, it takes longer for them to see real changes).

• Standardized multiple choice or short answers? Do you ask everyone the same multiple choice questions? This makes it easy to compare results. Or do you allow short answers? This adds more personality, and allows for factors beyond the standardized. It gathers anecdotes of success, which are great if (shudder) subjective.

• Do you want students to look at their own experience and draw conclusions? This subjective reflection in a pre and post situation can backfire. I have had students think they were a 4 out of 5 in leadership skills in the pre-test. Then they went through the program, learned a lot, but also realize how much more they had to learn, and then in the post-test claim they were a 3 out of 5. Through the program, they seemingly lost a point of leadership skills! (One option around that is to have student re-evaluate their pre program opinions in a post-test: Rather than just asking, “How good a leader are you?” in both the pre and post, I also like to ask in the posttest, “How good a leader were you before the program?” This can be compared against the respondent’s original answer, often revealing interesting levels of change.)

Example of Real-World Compromise Approach: The Bristol Method for Measuring Skill Adoption in a Large Community

The core assessment methodology is a simple model: A student gets sets of screens, with the task of quickly connecting boxed items from the left with the correct corresponding items on the right by drawing a line with a mouse.

The directions to the students are as follows:

• You will get five sets of screens; in each you will connect an item from the left column with an item on the right column. You will do this by using the mouse to click on the item on the left, which will highlight, and then click on the most corresponding item on the right. A line will be drawn showing that they are connected. If you make a mistake, reclick on an item in the left column, and do the process again. There is a 90 second timer for each screen. If you are done before the time allowed, you can press the “done” button. If you finish early (before the timer runs out), you will get some bonus points.
  
• After each screen, you will see which are the right answers and a score. Your score is based on both the number correct, as well as any bonus points for hitting “done” before the timer goes off. The first of the five screens will be a practice round, which is not scored.

To launch the assessment in the first place, the student has to demonstrate a basic understanding of the connecting action, as such:

Then, again per directions, the first screen is a practice screen. For example:

This will be followed by a review screen, showing which answers are right or wrong (this is important feedback to reward students), the time left if any, and a score comprised of the two. Specifically, students get 60 points as a base, and then seven points for every right answer, plus one point for ever ten seconds left over on the clock, up to a maximum of 100 points (these points may be adjusted).

Students then have the ability to go back and replay the test screen if they want. When they are ready, the students can go through the four "real" assessment screens and sets. Each of the four real assessment screens would show five scrambled questions and answers, drawn randomly from a pool of about fifteen possible pairs. The tone of the assessment is like a computer arcade game, with “fun” animations. At the end of the assessment they also get a final score with an average of all four sets.

The Questions

Obviously, the Bristol Method requires good questions. They can vary, although be organized by screens. So screens could include:

• Define the terms;
• Given brief overviews of situations, identify the best strategy;
• Given quotes, identify the meaning.

The test questions line up with the learning objectives of the course.

Process

This same test (although with randomized questions each time) is applied before and after a student goes through a simulation, and can also be applied before and after a student went through a non-simulation class, as well as no class at all for baselines.

Result

If the simulation was effective, the result should show that students answered more question correctly, and in less time, and in a way that is directly comparable (and favorable) to a shift in other methods or no methods. An aggregated graph may look like:

Rational

The Bristol Method solves a handful of traditional problems.

First, it moves quickly, and is enjoyable (or less miserable than a traditional test). It has some great bells and whistles. This is critical because any student taking a pre-test in a subject in which they don't know much can necessarily be miserable, and a harsh way to start any educational program. (Starting a two hour long simulation course having to answer questions to which a student doesn't know the answer can create defensiveness that puts a damper on the entire experience.) By matching (which rewards some knowledge by reducing the number of possibilities with each correct answer), by having a short timer (win, loose, or draw, the experience is over quickly), and by learning what they got right and wrong in the review, the experience can be as painless as possible. Further, the lower the student resistance to an assessment, the easier it is to create base cases.

Second, by measuring speed, a test asks questions that in an untimed methodology, the student can "figure out" given more time (which are often the types of questions one wants to ask anyway). It also minimizes the cheating options of an open-book tests. And given that an advantage of a simulation is instinctive, intrinsic knowledge, looking at speed is relevant.

Third, the content is fair and objective, and can show improvements (or lack thereof) in a way that is convincing to the outside world. And the long term time frame and low completion rates of 360 assessments, while more fair, are avoided.

Step 2.26: Develop Architecture, including Content and SCORM and LMS Integration

Content Architecture

When designing a simulation, one has to be mindful of three different types of content:

• The rigid content, which is nearly impossible to change once built (or once an authoring environment is procured);
• The semi fluid content, which can be changed but with expense and difficulty, requiring a certain level of expertise, and in a way that may create seams in the simulation; and finally
• The fluid content, which can be changed easily, without seams, at almost any point including post-employment, and by someone without specialized experience.

Typically, the rigid content is the structure of the display itself, the framework of the goals, and the nature of all interactivity. I often refer to this as the genre and is framed by any authoring environment. When looking at the program of a sim, the rigid content is often compiled in a large inaccessible executable file.

The semi fluid content includes all of the drawings, animations, voice recordings, and sets or backgrounds. Usually, this content is stored in folders off of the main directory, hopefully in industry standard formats such as JPGs and MP3s.

The fluid content includes all text, which should be easy to identify and change by anyone. For example, the words that make up the story go here. This content should be captured and accessed in XML files. Fluid content also includes level design, and branching links, including branching dialogs and state based diagrams, and weighting of algorithms, which also be stored in XMLs (albeit more cryptically and in a perfect world well documented). Finally, the most important fluid content should be tips that help the end user navigate the sim, including what user activity triggers them, and what they say and do (such as pause the sim or highlight a box). No matter how smart we all wish that we were, we can never fully anticipate when these will be needed until real people use the sim in real situations.

Therefore, as one is considering simulation design, one has to consider which content will need to be changed on an ongoing basis and need constant maintenance. Likewise, as one is building a simulation, one needs to nail down first all of the rigid content, next all of the semi fluid content, and only last the fluid content.

SCORM and LMS Integration

Almost all corporate, government, and military organizations need to launch sims from a learning management system, and often complying with various SCORM standards. In this step, develop the strategy and approach to do that.

Here's an example:

Let's assume that the sim consists of seven levels, each with a briefing, debriefing, and core interaction. Let's also say that each level is between three to seven minutes long for a single complete play-through. (Obviously, failed play-throughs are typically shorter.)

Once could visualize this progression as such:

From an integration with the LMS perspective, here are the critical points.

• The core sim is often Flash/Actionscript
• This core sim has an HTML wrapper. This HTML wrapper can but does not have to be the engine for the briefing and debriefing segments. However, the wrapper has to be accessed between levels of a sim.
• It is the HTML wrapper that communicates with the LMS, and contains the SCORM data.
• Typically, this means that the sim remembers where the user is at the granularity of level, not place in the level. If the user closes a sim halfway through the core interaction of a sim, the LMS will not record that progress, and restart the user next time at the beginning.

The resulting architecture looks something like this:

With the accompanying note that:

• It tends to be the server side data including xml sheets and media that allow the Flash engine to create a level.
• Either a browser cookie or some type of LMS integration is necessary to save the user's place in the sim.
• Often, a user has to demonstrate a level of competence in order not to fail a level.

Then, at a minimum, the LMS tracks:

• Name
• Position (by level)
• Any relevant score in each level (or even just the boolean of pass or fail).

Step 2.27: Bring in the Lead Artist

Engage the lead artist (If an organization has an internal artist, that person will have helped on several stages up to this point. If an organization has to bring in an external artist, they will typically wait as long as possible, which is at this step.). The lead artist is responsible for all of the visuals of the project. This includes the aesthetics of the interface, any and all color schemes, drawings, and, very importantly, animations (such as a character running or a button being clicked). The artist contributes greatly to the look and feel of the sim, from serious to light hearted. As a result they should be picked out extremely carefully. The artists may also have to work within the corporate identify guidelines created by a communications group. Ultimately, the art will consume about 10% of the total resources needed for a complete sim.

(An artist may have been brought in earlier to create mock-ups for proposals.)

Step 2.29: Finalize Key Simulation Actions and Mechanisms

Finalize the interface and gameplay, including various sim mechanics. This is the last chance to tinker with the interface.

Here is a sketch of a corporate financial sim, that uses a workbench like interface to allow managers to "play around" with various attributes of their value to chain to see what the impact would be in the marketplace. Note that the player can change prices. The shape of the widget shows it projected profitability.

Step 2.31: Meta-Code Algorithms

Finalize all of the high level mathematical or logic relationships needed for the sim. An average sim may have between 50 and 200 of these, including how scores are determined, how animations are played, how AI reacts, and how onscreen objects behave. With algorithms, the final values may be tweaked through scaling, but the key components have to be identified at this stage.

(See The Complete Guide to Simulations and Serious Games for more details.)

State-Based Meta-code Describing Dialogue Flow

Algorithm-based Meta-code for Behavior of Onscreen Avatar

Step 2.33: Create all of the Final Illustrations

Create all of the final illustrations for use in the sim. This includes backdrops and pedagogical illustrations, as well as organization logos.

These may change a little bit. But it is easier for the programmers to have "final" art to structure them, even if the art will eventually get nipped and tucked.

3.0 Code

In Part 3, the programmers/coders (hopefully well briefed and otherwise involved during Part 2) will program the material in the design document. They will produce much of the core sim engine itself, and provide the links to the fluid content, such as graphic files, videos, sound files, text, and entire level designs and sim flow, using industry standard media and xmls. The program sponsor, lead designer, graphic designer, and client liaison will be peripherally involved, making decisions, and helping flesh out the numerous parts of the sim engine that need refining. Near the end of this process, the lead designer will begin inputting as much of the final content as possible. About 40% of the project budget is spent in this stage.

Step 3.01: Prototype in Code Each New Segment/Genre

The most effective sims involve creating some new interface and game play tailored for the learning objectives. In this step of the sim design process, create some quick and dirty code to show how this basic new interaction is going to work. Validate that it will be effective and usable.

(Note: While the final sim will also involve more traditional pieces, such as the presentation of static content, these do not need to be prototyped here because there is a high comfort with that type of interaction. In short, if you can point to another program that does it well enough, you don't need to prototype it here.)

Step 3:02 Complete all Dialogue and Pedagogy Text

Complete the writing of all dialogue and all pedagogical content. While some of this will have been done from previous steps, it none the less requires the rigor of a computer programmer with the flair of a writer.

I use a basic rule for estimating the amount of text needed for a sim. First, create a best guess estimate based on how much dialogue and other words you think you will need (don't forget to include in-game tips), and then multiply that number times 10 if you have built a lot of sims, and 50 if this is your first sim.

Complete all of the dialog.

Complete all text and diagrams that introduce or wrap up concepts. (I did not work on this sim - this is a great and thorough project management sim from Double Masters.)

Step 3.03: Build Out the Various Content Engines

Continue working on the various engine prototypes to bring them to stability and connectitivity.

This is often a thankless activity, time and other resource consuming, relying almost completely on the few programmers, and without new things to put in front of the client. It can be best to focus attention on the story and even early deployment plans while so much intensive work is being done.

Step 3.04: Create One Complete Level

Combine the various pieces to create one beginning-to-end level. This gives you a first real chance to calibrate the timing and feedback for all levels. (This also gives you a preview of Step 3.06.)

Sim designed by me and produced by Carney

One of the biggest challenges is not the technical but the political. This level will be widely circulated among sponsors and other stake-holders. If the sim level is too early on, people will complain the sim is too easy and doesn't teach enough. If the sim level is too late, people will complain either it is too hard for them or that they get it but no one else will.

Ideally, out of a seven level sim, I choose either level 2 or 3.

Step: 3.05: Finish the Separate Self-Contained Engines Pieces and Fill in the Level Details

Complete the engine programming and fill in the content for the separate, individual levels of the sim. This includes broken up by:

• Different content engines/structures (such as overview slides, quizzes, turn based strategic segments, action oriented real-time segments, after action reviews), and the
• Different levels for each.

This is the last opportunity for quick access to the individual pieces, so do as much debugging as possible here.

The same sim has several different types of content, in this case: real time action, pedagogical framing, and fast review games: