Building the holodeck
I’ve been in and out virtual worlds since I was an undergraduate in the mid ’90s and man how have things changed between now and then. This series of essays outline some of the historical context for some ideas for augmented additions to the real world and fully immersive virtual environment.
In 1987 we were introduced to Star Trek: The Next Generation and the multiple new technologies it introduced, the holodeck chief among them.
A Holographic Environment Simulator, or holodeck for short, was a form of holotechnology designed and used by Starfleet. They were installed aboard starships, space stations, and at Starfleet institutions for entertainment, training, and investigative purposes. A typical holodeck consisted of a room equipped with a hologrid containing omnidirectional holographic diodes, enabling holographic projections and creation of holodeck matter through the manipulation of photons contained within force fields.
The most obvious function of a holodeck is to provide entertainment and diversion for the crew. (TNG: “11001001”, “The Big Goodbye”, “Elementary, Dear Data”, “A Fistful of Datas”; VOY: “The Cloud”, “Homestead”)
A holodeck can be used to create training simulations and exercise environments not otherwise available or safe, including star ship battle simulations and the Bridge Officer’s Test. (TNG: “Code of Honor”, “Where Silence Has Lease”, “The Emissary”, “New Ground”, “Firstborn”; VOY: “Learning Curve”, “Extreme Risk”, “The Fight”)
The holodeck can be used as a laboratory to aid in analysis, such as recreating the scene of a crime or accident to aid in forensic investigations. (TNG: “A Matter of Perspective”) They can be used to visualize a 3D scene from alternate data sources for analysis (TNG: “Identity Crisis”, “Phantasms”; VOY: “Distant Origin”) or used as a brainstorming tool. (TNG: “Schisms”, “Booby Trap”)
Holodecks can create both open ended worlds (like the worlds Jake Sisko ans his father shared in DS9’s Emissary) and specific scenarios (like the Dixon Hill Series of pulp novels that Captain Picard likes in TNG’s “The Big Goodbye”, “Manhunt”, “Clues”) or more open ended scenarios like the opening of Deep Spance Nine’s Emmisary or the Victorian novels of Voyager’s captain Janeway.
While we don’t have the holodeck quite like the one in the series but in the last 4 years or so virtual technologies have matured tto the point where an early version of the holodeck may be possible… nothing like Star Trek or the Full Dive Gear from Sword Art Online but immersive enough to trick our brains into thinking we’re already there.
In these essays we’ll explore Beacons as storytelling devices, Augmented Reality gear to tell stories and fully immersive environments and the stories we can tell in and with these 3D virtual worlds.
Digital worlds then and now
The progress to a fully 3D immersive holodeck is not a new endeavor or something that has appeared in a single discipline. In order to better understand what we want to do in the VR world it pays to get an idea of where we’ve already been. I’ll break the old virtual worlds into three categories:
- Text-based virtual environments
- SIMNET and military simulations
- Early attempts at virtual reality: Lucasfilms Habitat
Text-based virtual environments
The text-based environments I refer to the in this section are the TinyMUD created by James Aspnes in 1989 as the first in a line of successors to the original MUD created by Roy Trubshaw and Richard Bartle in 1978.
What differentiated TinyMUD and its children was a move away from combat as emphasized in the original MUD into a more social and communication environment. It also allowed al players to create objects and spaces in the world and then link it to existing content, whether player created or not.
I still remember my time as an anthropology undergrad (1994 – 98) and how I became involved in the early forms of immersive storytelling environments. MUDs (Multi User Dungeons) and their derivatives provided free-form interactions in a fully immersive textual experience.
For me it was Lord of The Rings and Battlestar Galactica. These style of worlds known as MUSH (Multi User Shared Hallucination) provides for a fluid style of writing that describes both the users’ speech and their interactions.
The example below, taken from Elendor MUSH illustrates the type of interaction players write.
Below is a snippet from an RP scene in Gondor recently played.
The two players involved are Tathar and Turlach Nimothan, aunt and nephew.There is not much to be seen here, for the weather is dark and rainy and
the bookshelves unlit. But at a table is set a candle, its wick nearly
drowned in melted wax, and beside it, Turlach’s dark head rests upon an
open book. His eyes are closed.The door swings open quietly, and the wavering light of another candle
shedding its soft golden glow into the room. Tathar comes in, stopping
as she sees Turlach, then moving to the bookshelves, where she lights
a lamp, and pulls out one of the books, taking it and her candle to a chair nearby.[fast-forward to a bit of their conversation.]
“It means …” Turlach looks up cautiously, eyeing the slightly-ajar
door. “She may have been taken by the Black Company. But why, I wonder?”The door is pushed further open as an elderly butler comes in. “Yes,
m’lady?”“Something hot, please,” Tathar instructs him. “Turlach and I are growing
chilled, doing nothing but sitting and reading. A hot drink would be very
welcome.” Beldin nods and vanishes again, and Tathar turns her attention
back to her nephew. She is frowning.“Taken…? But of what avail would that be to them? The Ephalkhir have nothing
to do with Mormegil or its estates.”
The idea that attracted a large number of people (myself included) to MUDs was the chance to recreate the worlds of our favorite novels or TV series and push them beyond their conclusion. It wasn’t enough to know how the last episode of the single season of Battlestar Galactica ended, we wanted to build the story of what happened afterwards. Did the Galactica find Earth? How did it happen?
MUD creators provided us with very detailed worlds built on text and the possibilities to interact with the environment and with objects created by the system operators or by other players. We recorded the stories we played in to have a record of our collective performances for posterity.
The experiences were not always positive. Some of the first conversations about online communities, punishment and enforcement originated from less than pleasant, some may say terrifying, events that happened in these early text-baed communities.
Wherever you see an MMORPG you’re seeing the children of MUD and when looking at AR and VR Stories it is no different. We are all children of MUD.
Distributed Training Simulations: SIMNET
SIMNET was the first successful implementation of large-scale, real-time, man-in-the-loop simulator networking for team training and mission rehearsal in military operations.
— SIMNET: The Advent of Simulator Networking
SIMNET stands for SIMulator NETworking. Initiated in 1983, it was the first “shared virtual
reality” distributed simulation system, which continues to have significant influences. It was
sponsored by DARPA, the Defense Advanced Research Projects Agency, the Department of
Defense’s principal high-risk, high-payoff research and development organization, established in 1958. […]— SIMNET and Beyond: A History of the Development of Distributed Simulation
SIMNET was created in the early 1990s to provide training simulations for large scale forces without the cost associated with moving such large forces to training sites like The National Training Center in Fort Irwin California.
There were two revolutionary aspects of SIMNET: The real time long distance networking infrastructure and the simulator pods built to participate in the simulations.
SIMNET Networking
SIMNET provided a fully decentralized architecture for each of the simulator pods representing anything from an M1 Abrahams tank to an AH-64 Apache attack helicopter.
The biggest improvement was the lack of a central coordinating unit. Each simulator pod was responsible for tracking its status, the status of any unit it interacted with directly and any result of interaction (did it hit another object such as a mine, did a gun fired hit something or was the unit hit by an object fired by another unit) and of reporting its state to units it interacts with in the network. This was defined as each unit being an “autonomous node”.
This made SIMNET scalable to several hundred units spread around military bases in the United States (Fort Benning, GA, Fort Rucker, AL, Fort Knox, KY, and Fort Leavenworth, KS) and Grafenwoehr, Germany. To accommodate for the increased bandwidth requirements and potential latency, SIMNET adopted two additional principles:
First, a simulation node would send updates only when its state changed. For example, a stationary entity would not send state updates (except for minimum “heart-beat” updates that assured other nodes that network communications had not been lost).
Second, an entity moving in a straight line at a steady speed would send state updates only when it changed its speed or direction.
SIMNET Simulator Pods
The simulator pods each represented a fully crewed vehicle (a 4-man tank or a 2-man helicopter). It would have been very difficult to fully reproduce a combat vehicle in the simulation environment SIMNET adoped a “selective fidelity” (also known as “the 60% solution”) where only parts relevant to the functioning of the vehicle in the simulation were kept. Commanders and other mission supervisors could decide what systems and aspects of the 60% were carried over. An example of this was the decision to induce “jiggle” in the vision block views and a “rumble” in the crew’s seats to give them clues as to how fast they were moving and what kind of terrain they were traversing.
Some of the vehicle simulator pods (taken from SIMNET and Beyond):
M1 Abrams Main Battle Tank. The M1 has been the Army’s primary heavy combat vehicle since entering service in 1980. It carries a crew of four: the driver, who reclines in a very tight space inside the forward hull compartment, plus the tank commander, the loader, and the gunner, who sit inside the turret compartment. The driver’s orientation is fixed, but the turret can rotate 360 degrees (quite rapidly), so the turret’s orientation can be independent of which direction the tank is moving. The tank commander’s role is to control and coordinate the tank’s overall operation, as well as to coordinate tactically with other commanders. He tells the driver where to go, designates targets for the gunner, and tells the loader what type of ammunition to pull from the rack behind the blastproof doors and load into the breech of the 105 mm or 120 mm cannon. The crew communicates with each other over a local intercom, and the commander communicates over a multi-channel radio.
Similar to the real vehicle, the M1 simulator has two separate compartments: one for the driver and one for the other three crew members, each with the most essential controls and displays (chosen in accordance with the “selective fidelity” principles previously noted). The commander sits above the gunner, with the loader located to their left between the ammunition rack and the gun breech. A full description of the simulator crew compartments and controls from the SIMNET era can be found in (Chung, 1988). The driver has three “vision block” displays, the gunner has a stabilized gunsight with high/low magnification, the commander has three vision blocks plus an auxiliary sight that duplicates the gunner’s sight, and the loader has a periscope display. Both the commander’s and loader’s cupolas can rotate to simulate additional vision blocks in a closed-hatch M1.
Rotary-Wing Aircraft (RWA). Generic attack/scout helicopter, approximating the AH-64 or OH-58. Includes pilot and copilot/gunner stations with flight controls and displays. Includes a slewable electrical-optical sensor for use in target acquisition and engagement. It is armed with HELLFIRE laser-guided anti-tank missiles and air-to-air STINGER missiles, in addition to a 30-mm chain gun. The pilot and co-pilot share visual access to eight out-the-window displays, arranged in a “three over five” configuration, plus a single sensor channel, switchable between a thermal image and a daylight TV image. The RWA uses typical helicopter cyclic and collective controls, along with a throttle, for flight maneuvers. (Harrison, 1992)
SIMNET Impact and Legacy
The simulators were not a marvel of high technology but they did their job well enough that SIMNET evolved from a training system into mission planning and, eventually, technology development simulator. The technologies pioneered in SIMNET are now part of a family of Distributed Interactive Simulation standards that have grown past their origins in wargaming and war fighting simulations; space exploration and medical fields have also embraced simulation as a lower cost, lower risk alternative to live activities.
The events are reproducible. The Battle of 73 Easting was fully recreated in collaboration with the member of the unit involved and a very thorough analysis. They captured GPS coordinates from the American vehicles involved in the battle and reconstructed the event as accurately as possible. People can now review the simulated battle and see the action as it happened more than 20 years ago. Simnet became a study (and marketing) tool for the Army and the basis for a new family of training environments, the latest of which is the Synthetic Training Environment.
Several members of the original teams that worked in SIMNET moved on to gaming and distributed networking endeavors after SIMNET was over. Some of these startups include:
- RTIME, Inc. (Rolland Waters) game network engines
- MetaVR, Inc (W. Garth Smith), simulation and training, GIS systems
- MaK Technologies (Warren Katz and John Morrison), simulation software
- Reality by Design, Inc (Joanne West Metzger and Paul Metzger), simulation and training systems
- Zipper Interactive (Brian Soderberg), game developer (SOCOM game series)
- Wiz!Bang (Drew Johnston), game developer
Early attempts at graphic VR: Habitat
One of the first attempts at creating a graphical online virtual world was Habitat. Created by F. Randall Farmer and Chip Morningstar for Quantum Link an early Internet Service Provider targeted to Commodore 64 and Commodore 128 systems using dial up connections.
The lead creators of Habitat, Chip Morningstar and F. Randall Farmer have written extensively about what Habitat is and what lessons they have learned in the process.
The essential lesson that we have abstracted from our experiences with Habitat is that a cyberspace is defined more by the interactions among the actors within it than by the technology with which it is implemented. While we find much of the work presently being done on elaborate interface technologies — DataGloves, head-mounted displays, special-purpose rendering engines, and so on — both exciting and promising, the almost mystical euphoria that currently seems to surround all this hardware is, in our opinion, both excessive and somewhat misplaced. We can’t help having a nagging sense that it’s all a bit of a distraction from the really pressing issues. At the core of our vision is the idea that cyberspace is necessarily a multiple-participant environment. It seems to us that the things that are important to the inhabitants of such an environment are the capabilities available to them, the characteristics of the other people they encounter there, and the ways these various participants can affect one another. Beyond a foundation set of communications capabilities, the details of the technology used to present this environment to its participants, while sexy and interesting, are of relatively peripheral concern.
It is interesting to see the parallels between Habitat and its modern descendants like Destiny and World of Warcraft. The characters and the backgrounds and the interactions are different but how you move and how you interact with people hasn’t really changed significantly since the early days of online multi player games.
Some things the authors warn us that central planning is impossible. In a fully interactive multi player environment makes it impossible to plan every little contingency. We’ll discuss this when we talk about storytelling and planning versus spontaneity.
The emphasis has always been in community building. Habitat was immersive not because of its technology, the graphics don’t hold a candle to today’s massive 2D and 3D game environments, textures and animations, but because of the community and the interactions between people who were in different geographical locations yet were able to live and interact together in a community that was free of trolls and free of the harassment that permeates our modern online communities.
They also argue for a world that is consistent with itself. If something happens handle in the world. Don’t step out of the world to handle it. The two examples they give very strongly illustrate the point.
In the first instance it would have been easy to just take the money away from the ‘cheaters’ and restore the economy to what it should be. They were lucky that the avatars were good and used their resources for the betterment of the community.
In order to make this automated economy a little more interesting, each Vendroid had its own prices for the items in it. This was so that we could have local price variation (i.e., a widget would cost a little less if you bought it at Jack’s Place instead of The Emporium). It turned out that in two Vendroids across town from each other were two items for sale whose prices we had inadvertently set lower than what a Pawn Machine would buy them back for: Dolls (for sale at 75T, hock for 100T) and Crystal Balls (for sale at 18,000T, hock at 30,000T!). Naturally, a couple of people discovered this. One night they took all their money, walked to the Doll Vendroid, bought as many Dolls as they could, then took them across town and pawned them. By shuttling back and forth between the Doll Vendroid and the Pawn Shop for hours, they amassed sufficient funds to buy a Crystal Ball , whereupon they continued the process with Crystal Balls and a couple orders of magnitude higher cash flow. The final result was at least three Avatars with hundreds of thousands of Tokens each. We only discovered this the next morning when our daily database status report said that the money supply had quintupled overnight.
We assumed that the precipitous increase in “T1” was due to some sort of bug in the software. We were puzzled that no bug report had been submitted. By poking around a bit we discovered that a few people had suddenly acquired enormous bank balances. We sent Habitat mail to the two richest, inquiring as to where they had gotten all that money overnight. Their reply was, “We got it fair and square! And we’re not going to tell you how!” After much abject pleading on our part they eventually did tell us, and we fixed the erroneous pricing. Fortunately, the whole scam turned out well, as the nouveau riche Avatars used their bulging bankrolls to underwrite a series of treasure hunt games which they conducted on their own initiative, much to the enjoyment of many other players on the system.
The second example is of how to handle incidents “in world” as opposed to handling it as a technical exercise and frustrate users when we violate the contract between the users and the fantasy in their world.
One of the more popular events in Habitat took place late in the test, the brainchild of one of the more active players who had recently become a QuantumLink employee. It was called the “Dungeon of Death”. For weeks, ads appeared in Habitat’s newspaper, The Rant, announcing that that Duo of Dread, DEATH and THE SHADOW, were challenging all comers to enter their lair. Soon, on the outskirts of town, the entrance to a dungeon appeared. Out front was a sign reading, “Danger! Enter at your own risk!” Two system operators were logged in as DEATH and THE SHADOW, armed with specially concocted guns that could kill in one shot, rather than the usual twelve. These two characters roamed the dungeon blasting away at anyone they encountered. They were also equipped with special magic wands that cured any damage done to them by other Avatars, so that they wouldn’t themselves be killed. To make things worse, the place was littered with cul-de-sacs, pathological connections between regions, and various other nasty and usually fatal features. It was clear that any explorer had better be prepared to “die” several times before mastering the dungeon. The rewards were pretty good: 1000 Tokens minimum and access to a special Vendroid that sold magic teleportation wands. Furthermore, given clear notice, players took the precaution of emptying their pockets before entering, so that the actual cost of getting “killed” was minimal.
One evening, one of us was given the chance to play the role of DEATH. When we logged in, we found him in one of the dead ends with four other Avatars who were trapped there. We started shooting, as did they. However, the last operator to run DEATH had not bothered to use his special wand to heal any accumulated damage, so the character of DEATH was suddenly and unexpectedly “killed” in the encounter. As we mentioned earlier, when an Avatar is killed, any object in his hands is dropped on the ground. In this case, said object was the special kill-in-one- shot gun, which was immediately picked up by one of the regular players who then made off with it. This gun was not something that regular players were supposed to have. What should we do?
It turned out that this was not the first time this had happened. During the previous night’s mayhem the special gun was similarly absconded with. In this case, the person playing DEATH was one of the regular system operators, who, accustomed to operating the regular Q-Link service, had simply ordered the player to give the gun back. The player considered that he had obtained the weapon as part of the normal course of the game and balked at this, whereupon the operator threatened to cancel the player’s account and kick him off the system if he did not comply. The player gave the gun back, but was quite upset about the whole affair, as were many of his friends and associates on the system. Their world model had been painfully violated.
When it happened to us, we played the whole incident within the role of DEATH. We sent a message to the Avatar who had the gun, threatening to come and kill her if she didn’t give it back. She replied that all she had to do was stay in town and DEATH couldn’t touch her (which was true, if we stayed within the system). OK, we figured, she’s smart. We negotiated a deal whereby DEATH would ransom the gun for 10,000 Tokens. An elaborate arrangement was made to meet in the center of town to make the exchange, with a neutral third Avatar acting as an intermediary to ensure that neither party cheated. Of course, word got around and by the time of the exchange there were numerous spectators. We played the role of DEATH to the hilt, with lots of hokey melodramatic shtick. The event was a sensation. It was written up in the newspaper the next morning and was the talk of the town for days. The Avatar involved was left with a wonderful story about having cheated DEATH, we got the gun back, and everybody went away happy.
These two very different responses to an ordinary operational problem illustrate our point. Operating within the participants’ world model produced a very satisfactory result. On the other hand, taking what seemed like the expedient course, which involved violating the world-model, provoked upset and dismay. Working within the system was clearly the preferred course in this case.
Some of these experiences continue to shape our technological communities, how to balance freedom and creativity with keeping people safe. Others have to do with community rules, user types and managing behavior and expectation.
Single user stories, like those I’m most interested in crafting, don’t have these types of problems but, as soon as you bring more than two people together and one or more are women, online you have to deal with the potential for attacks, trolling and sexist remarks.
