Utopia

“Well, Unit Dee is the only one likely to suffer damage. Unit Dum is a non sentient computation device, not a robot. He has a pseudo-self-aware interface that allows him to converse, to a limited extent, but he’s not a robot and he’s not subject to the Three Laws. Unit Dee is a different story. She’s really not much more than an enormous positronic brain hooked up to a large number of interface links. A robot brain without a conventional robot body-but she is, for all intents and purposes, a Three-Law robot. Just one that can’t move.”

“So what is the difficulty?” Kresh demanded, clearly on the verge of losing his patience again.

“That should be obvious,” Soggdon replied, realizing just a second too late how rude a thing that was to say. “That is-well, my apologies, sir, but please consider that Unit Dee is charged with remaking an entire planet, a planet that is home to millions of human beings. She was designed to be capable of processing truly huge amounts of information, and to make extremely long-range predictions, and to work at both the largest scale and the smallest level of detail.”

“What of it?”

“Well, obviously, in the task of remaking a planet, there are going to be accidents. There are going to be people displaced from their homes, people who suffer in floods and droughts and storms deliberately produced by the actions and orders of these two control systems. They will, inevitably, cause some harm to some humans somewhere.”

“I thought that the system had been built to endure that sort of First Law conflict. I’ve read about systems that dealt with large projects and were programmed to consider benefit or harm to humanity as a whole, rather than to individuals.”

Soggdon shook her head. “That only works in very limited or specialized cases-and I’ve never heard of it working permanently. Sooner or later, robotic thinking machines programmed to think that way can’t do it anymore. They bum out or fail in any of a hundred ways-and the cases you’re talking about are robots who were expected to deal with very distant, abstract sorts of situations. Unit Dee has to worry about an endless series of day-by-day decisions affecting millions of individual people-some of whom she is dealing with directly, talking to them, sending and receiving messages and data. She can’t think that way. She can’t avoid thinking about people as individuals.”

“So what is the solution?” Kresh asked.

Soggdon took a deep breath and then went on, very quickly, as if she wanted to get it over with as soon as possible. She raised her hand and made a broad, sweeping gesture. “Unit Dee thinks this is all a simulation,” she said;

“What?” Kresh said.

“She thinks that the entire terraforming project, in fact the whole planet of Inferno, is nothing more than a very complex and sophisticated simulation set up to learn more in preparation for a real terraforming project some time in the future.”

“But that’s absurd!” Kresh objected. “No one could believe that.”

“Well, fortunately for us all, it would seem that Unit Dee can.”

“But there’s so much evidence to the contrary! The world is too detailed to be a simulation!”

“We limit what she can see, and know, very carefully,” Soggdon replied. “Remember, we control all of her inputs. She only receives the information we give her. In fact, sometimes we deliberately introduce spurious errors, or send her images and information that don’t quite make sense. Then we correct the ‘mistakes’ and move on. It makes things seem less real-and also establishes the idea that things can go wrong. That way when we do make mistakes in calculations, or discover that we’ve overlooked a variable, or have just plain let her see something she shouldn’t have, we can correct it without her getting suspicious. She thinks Inferno is a made-up place, invented for her benefit. So far as she knows, she is actually in a laboratory on Baleyworld. She thinks the project is an attempt to lea-n how to interact with Settler hardware for future terraforming projects. “ Soggdon hesitated for a moment, and then decided she might as well give him the worst of the bad news all at once. “In fact, Governor, she believes that you are part of the simulation.”

“What!”

“It was necessary, believe me. If she thought you were a real person, she would of course wonder what you were doing in the made-up world of her simulation. We have to work very hard to make her believe the real world is something we have made up for her.”

“And so you had to tell her that I did not really exist. “

“Precisely. From her point of view, sapient beings are divided into three groups-one, those who exist in the real world, but don’t have anything to do with her; two, real-world people here in the lab and in the field who talk with her and interact with her-and three, simulants, simulated intelligences.”

“Simulants,” Kresh said, very clearly not making it into a question. He was ordering her to explain the term, not asking her to do so.