Why Use Grades, Anyway?: An Excerpt from Simulations and the Future of Learning

What follows is the third and last excerpt from Simulations and the Future of Learning, a 2004 text book on simulations that is structured around a first-hand account of the creation of the 2002 simulation, Virtual Leader:

---

Almost all formal learning programs have grades. And we were creating a formal learning program. So we needed grading, right? On the other hand, we were using computer games to teach, and we made up our own leadership theory, so what did we care about convention? Could we push our luck a bit further?

Cultural assumptions aside, was scoring inherent in learning? Did it help or hurt the process?

How Do You Score an Open-Ended Simulation?

I have to admit that one reason for my reluctance to incorporate scoring into Virtual Leader was that I really did not know how to do it. How would you score The Sims if you had to?

If an experience is open-ended, how do you take into account that one player focuses on wealth and another on social networks? Do you make an equation to say that every $10 thousand in annual income equals two close friends?

How would you score SimCity? Fiscal restrain or expansionary vision? Or the establishment of cultural landmarks?

Should the score be running all of the time, or only at the end? If it is ongoing, how would you measure someone who is executing a more complicated plan than the norm? In open-ended simulations, all of the pieces have to be in place before any results are seen. If scoring is held until the end, who would care? The game is over. Sid Meier’s Alpha Centauri gives players a comprehensive score at the end, but at that point, it just doesn’t matter.

There is the philosophical question of what to reward. But there is also a very technical question of what values you assign. As you add complexity, gameplay becomes harder to predict. Scores seem impossible to create in a way that does not seem arbitrary. How do you take into account all variables? The more complicated and intertwined a system is, by the way, the harder it is.

Here is a real example. In Roller Coaster Tycoon, players have the goal of building their own amusement park, with one core metric being the satisfaction of all of the people at your park. Different people have different needs, and therefore your park may sit better with people who want slow rides, balloons, and leisurely walks by the pond, as opposed to those who crave fast rides and sugary foods. So measuring and scoring customer satisfaction sounds simple enough. But here is a real hint from a site called gamewinners.com on how to increase your park rating:

“Find all of the guests who are unhappy or angry. Drown them. Eventually your park rating will go up 100 to 200 points.”

Tight metrics and open-ended play seemed like impossible bedfellows. It wasn’t getting any easier.

---

This article has been excerpted from the 2004 classic, Simulations and the Future of Learning by Clark Aldrich.

For the design of an educational sim, how accurate is accurate enough?: An Excerpt from Simulations and the Future of Learning

What follows is an excerpt from Simulations and the Future of Learning:

---

How accurate do simulations have to be to be valid teaching tools? Said more technically, to what degree does a simulation have to be predictive and/or of very high fidelity in order to be instructive?

As with other questions concerning realism in simulations, the answer to this question will change with time. Simulations will become increasingly realistic. But they will never perfectly replicate reality. How can I be sure? Because across time and technologies, once a simulation in a given area becomes perfect, you could take the person out of the equation altogether and just automate the system.

Simulations are there to help people, not replace them—or as General George S. Patton, Jr., once said, “All of this talk about super-weapons and push-button warfare is a pile of junk. Man is the only war machine. Man has to drive tanks, fly planes, crawl through the mud, pull the triggers, and push the buttons.”

In fact, there are several reasons you might not want a simulation to perfectly reflect real life. For instance, the environment provided by a simulation may actually be a better one for learning than real-life situations, which almost always contain a fair number of distractions. The example that Stephen Alessi and Stanley Trollip give in Computer-Based Instruction: Methods and Development is a good one:

The cockpit of a modern airplane is one of the worst learning environments possible. Not only are there many instruments confronting the novice pilot but also constant messages being relayed to and from the traffic controllers from all aircraft in the vicinity. . . . the novice pilot is usually apprehensive about being up in the air and is also concerned about other aircraft nearby. All this creates a situation in which most attention is being concentrated on aspects actually irrelevant to the immediate task at hand, which is learning to control the plane. (p. 173)

Given that people are part of the equation, in simulation design, perfection is not always as perfect as you might hope. Part of the goal of any simulation is to focus the end-learner on a finite, not infinite, set of relationships. While the number of relationships will grow both as simulations become more powerful and as we become more used to learning from them, simulations will never reach the infinite subtlety of life, nor should they.

One analogy is bird guides. Some guides use photographs of birds, such as the National Audubon Society Field Guide to North American Birds. These pictures are, by all standards, as accurate as possible. They are nearly perfect. Yet the lists of best-selling bird books are dominated by guides using illustrations, such as the classic books by Roger Tory Peterson, the recent best-selling Sibley Guide to Birds by David Sibley, Kaufman Focus Guides, National Geographic Field Guide to the Birds of North America, and even the best-selling coffeetable book, Birds, by
Robert Bateman.

How can this be if accuracy is key? Clearly there are other issues. Illustrations provide an editorial ability of the expert both to highlight certain aspects and to pull out some of the background. Cooking books, carpentry manuals, and similar guides in fact use carefully rendered illustrations rather than photographs, despite the fact that illustrations are actually more expensive to produce.

---

This article has been excerpted from the 2004 classic, Simulations and the Future of Learning by Clark Aldrich, available from Jossey-Bass.

The Myth of Subject Matter Experts: An Excerpt from Simulations and the Future of Learning

What follows is an excerpt from Simulations and the Future of Learning:

---

If you were to design a leadership simulation, something that had both valid learning and computer-game-like interactivity, how would you do it?

Other than playing more than my fair share of computer games, I was way outside my domain of expertise now. I knew the existing e-learning market inside and out. I could draw market diagrams on a whiteboard in multicolor. But going this far out rendered much of my own hard-earned specifics useless. Not only did I not have any answers to the tough questions, but I didn’t even know what the tough questions were. Some of the issues we did know that we had to solve were

• What would a leadership simulation look like when it was finished?
• Where should we start?
• How long would it take?
• How much would it cost?
• What would the perfect team to pull it off look like?

The Shrinking Role of Linear, Branded Experts

Our first goal, we agreed, was to find a leadership expert from whom we could license some established content. Our focus would be on the creation of the simulation, not on the leadership theory. We wanted to stick to our core competency. We figured this should take about a month to do, two months on the outset. Once we had done that, we could get to the business of making any necessary tweaks to the material, and then we would bring in the people to develop the software.

There were a few problems.

We visited with twenty or so academics, authors, and consultants. Some you would have heard of, most probably not.

They all had modern glass and chrome offices or comfortable, Georgetown homes. They all could not have been more gracious when we walked in the door. They were well-dressed, and I am sure they all played golf. They moved quickly and efficiently, belying their age. The conversations started off with pleasantries, although salted with numerous reminders of how busy they were, frequent checking of their watches, and reiterations of how much of a favor
they were doing by meeting with us.

It would have been more encouraging if I could write that the experts fell into one of the following groups. In fact, everyone we talked to fell into ALL of these groups.

Group 1: The Figureheads

The experts wanted a huge amount of money for no work and no accountability. The first thing they made clear is that they expected five to ten thousand dollars a day in consulting fees, as well as huge amounts of equity in the company. A project like this would take weeks of their time, they calculated gleefully, maybe months. And they were very busy.

Group 2: The Control Freaks

These experts wanted complete control over the project. Or they wanted veto power at every point along the way, from product to marketing to packaging. They did not want their names misused, but they also wanted their names on everything, in the largest letters. It would really be their simulation, they decided, and we would help them make it. They would use their names in the title, and we would get a “powered by” mention on the back. And of course, they should have the right to infinitely hold up the project. It was their project, after all.

Group 3: The Pass-Off Masters

Could they help us? Well, maybe. They wanted to clarify what they meant by “their” time. They were very busy. What would be better, they explained, was to have their assistants/grad students help us. The experts assured us that these junior players, although only part of the team for nine months, were masters of the topic and would be our primary contacts for work going forward. The experts would still be there, however, to sign off on everything, when they could find the time.

Group 4: The Time Challenged

These experts let us know that they were on a different time schedule than we were. Returning a call after a month was hurrying for them. They all had critical projects, critical clients, critical family obligations, and critical month-long vacations where they would be out of touch. The academics thought in terms of semesters and five-year grants. Weeks were rounding errors.

Group 5: The Linear Headliners

Essentially, these experts viewed themselves as star athletes, and we were Wheaties. They wanted their pictures on the box and a check with lots of zeros for the honor. Anything that didn’t fit that mold didn't quite fit with them.

That mentality was difficult for all of us. But, we conceded, all of those were ultimately manageable.

But when we went a little further with a few of them, something that should have been obvious was staring us in the face. These people, self-proclaimed “experts in the leadership area,” were all linear experts. Their knowledge consisted of case studies and high-level charts. They spoke in terms of sequences. They returned to linear content like a magnet to north.

To launch the simulation, they mulled, maybe a fifteen-minute video of them talking would be best. Or maybe longer.

After someone made a mistake, they could come back in and give some comforting comments. They had some white papers that could be pre-reading, so players could read fifteen pages if they made the same mistake twice.

Meanwhile, the more probing questions we asked, the more they shrugged their shoulders. After hearing one person’s dramatic case study, for example, we asked, “What determines a person’s loyalty between friendship and authority?”

“No,” she corrected. “As I said, friendship was more important. The authority never had a chance.”

We persisted, “Under what conditions would those people have followed the authority, and not the personal relationship?”

“Well, that would be a different example. I would use an example from IBM in that case.”

And on it went.

Today’s experts are in the business of producing linear content—be it a speech, a book, or a lecture series. This linear framework has influenced how they gather information for decades. Everything they have ever studied was broken into paragraphs. Their mental note cards were ordered and re-ordered depending on the output. Getting any of them to think of content non-linearly would be a huge undertaking, possibly impossible. Because for them to accept The Simulation Way, they would have to accept that they were no longer experts. And that would mean no huge fees.

---

This article has been excerpted from Simulations and the Future of Learning by Clark Aldrich, available from Jossey-Bass.

Actions: only include them if the learning matters or the wisdom is worth recording

A critical part of the instruction design process (or creating any intellectual property designed to change behavior, from research reports to web sites to sims) is to identify actions.

Actions are what a person in an experience actually does at the most tactical, or basic level, be it "pat person on the back" in one scale or "buy small company" in another.

Having said that, this step is often glossed over. Further, even our traditional vocabulary to describe actions is remarkably thin. Here are some examples of terms to describe actions.

Contextual Actions

Many actions only are available based on context, such as where on a Map one is. One can't open a door unless one is next to it.

And many contexts are conceptual, and have to be earned. One can't promote an employee unless one is at the right organization level. One can't buy a date dinner unless one is out for dinner.

Cyclical Actions: Like notes of music or words in a sentence

Some actions are cyclical; as with tennis or negotiating or writing. The same finite set of actions is available throughout an experience, and the actions typically have two analogue components, timing and magnitude.

For example, in driving a car, it is not enough to "turn right," but to turn the steering wheel the right amount, and at the right time. Further, drivers will have to "turn right" not once a trip, or ten times, but an almost indefinite amount of times. Likewise, in managing a project, it is not enough to increase the budget, but to increase the budget the right amount, and at the right time. In eating well, it is not enough to eat protein. One has to eat the right amount, and at the right time.

Cyclical actions are effective in contexts of each other and combinations. The collection of actions is ultimately as logical and fluid as collections of words in a sentence, or notes of music in a concerto, and needs as much practice to master. Certain "hard wired" succesful combinations of actions are called combo moves.

Cylical actions themselves tend to be recursive. Further:

• Actions have to, in combination, enable Middle Skills, which then enable Big Skills.
• Actions can be done in the context of activities (the action is the what, the activity is the why), which themselves may be part of a process.
• Actions may get one close to a long range goal via conceptual dead reckoning.

Actions in Learning

A focus on actions is necessary for a formal learning program to change behavior. In a simulation, Actions are accomplished by Basic Inputs. These can include pressing buttons, moving sliders, entering numbers, clicking on a text window or icon, or moving a joystick.

Actions. Some actions impact activities and process, and some actions impact big and middle skills. Some actions impact both simultaneously.

The granularity and scope of the simulation also determines the granularity of the actions. For a country president, an action might be [to send an assertive letter to another country's leader]. But if the simulation is about the staff writer, an action might be [to start off the letter recalling a shared positive experience].

The presentation and availability of actions through basic inputs can greatly influences the look and functionality of the simulation display. And the right mapping of actions to basic inputs to interface greatly facilitates the transfer of knowledge from the program to the real-world.

The educational challenge of actions is building a new awareness of students’ real-world options. Conquering the actions challenge sometimes means seeing things at a higher level (when I think I am doing x, I really am doing y.). Sometimes it means not allowing students to do things the way they have in the past to break bad habits.

The challenge of actions is also the challenge of applying what one learned in the world. Most traditional courses or research leave the task of applying the material learned for the students to figure out after the class, which means that most do not do it, and the learning is wasted. By forcing the practice of the application of the material to the front of the program, it paves the way for the materials’ productive use after the program.

Research Questions

Question to ask when researching actions are:

• What does an expert actually do all day? What are the discrete activities?
• What are the same collection of options an expert is constantly evaluating? Why do they do one over another?
• How does an expert know if they have done an activity too soft or too hard?

Questions to ponder:

• What does a company's top sales person do when talking to a perspective client?
• What did you CEO or President do at 10:15 today? Why?
• Before and during the battles of America’s Revolutionary War, what were the ten or fifteen options that General George Washington repeatedly considered? What sequence did he do each time, and with what relative intensity?
• Every week, what group of discretionary actions did Joseph Stalin consider while in
  power? Of those choices, which did he take? How did those actions (both considered and taken) shift over his years in power?
• What options did the Ottoman Empire have to deal with its neighbors?

This type of action-content lends itself to a simulation, and drives improvements in active behavior. But it can take many forms, and should be part of any knowledge capture and transfer.

(For more information, see The Complete Guide to Simulations and Serious Games, from where this entry was taken.)

Game Elements: A spoonful of sugar if you can avoid hypoglycemic shock

Game elements are techniques that motivate people to want to engage an experience, outside of any intrinsic motivation.

Game elements include fantasy, whimsy, competition, beauty, and a great story. It is a game element, for example, that positions the participant as the hero.

Games elements can do a lot of good in a serious games or educational simulations. Game elements drive engagement.They can make a boring simulation much more tolerable. They can build good will, which is often then transferred to the content itself. They can lower tension (so assessing with a game show might actually be more accurate than assessing with a test). In fact, the Genres of frame games and mini games are the result of almost pure examples of game elements, although any sim genre uses some game elements.

But game elements are also very controversial.

• Game elements surround and dilute the learning. They take up developer time and they take up end-learner time, ultimately taking resources away from the primary content.
• They are subjective. What is fun for one person, such as gambling or treasure hunts, can be tedious for someone else. Different cultures, ages, genders, experiences, and needs all create complex demographics.
• Game elements can also sometimes subvert the learning. A developer can make things happen faster, or more dramatically (see exaggerated response), or abstract tedious steps, which increases the fun but at the expense of lack of accuracy.
• “Fun” elements can often also be leveraged by a learner to “game” the sim.
• Certain game elements, like competition, can focus users on getting a high score rather than learning the material.

Having too little game elements results in a boring, dry experience, and having too many game elements creates something that is silly and distracting. As a rule, the more one cares about content, the more one is intolerant of game elements. If the orphanage is burning and you need to learn how to put out the fire, the last thing you want is to play Wheel of Fortune to get the information.

Balancing Simulation Elements, Game Elements, and Pedagogical Elements is unquestionably an art, not a science. But understanding the trade-offs is critical to aligning sponsor, developer, implementer, and student.

Obviously what is fun for one person is not necessarily fun for another. You can envy the fun someone else is having, but know that if you were the same situation, you would not be having any fun at all.