Utopia

“Ah, yes, sir. That’s the standard industry term for the made-up humans and robots placed in a simulation. Unit Dee believes that the entire population of Inferno is really nothing more than a collection of simulants-and you are a member of that population.”

“Are you trying to tell me I can’t talk to her because she’ll realize that I’m not made-up?” Kresh asked…

“Oh, no, sir! There should be no problem at all in your talking with Unit Dee. She talks every day with ecological engineers and field service robots and so on. But she believes them all to be doing nothing more than playing their parts. It is essential that she believe the same thing about you.”

“Or else she’ll start wondering if her simulated reality is actually the real world, and start wondering if her actions have caused harm to humans,” said Kresh.

“She has actually caused the death of several humans already,” Soggdon replied. “Unavoidably, accidentally, and only to save other humans at other times and places. She has dealt reasonably well with those incidents-but only because she thought she was dealing with simulants. And, I might add, she does have a tendency to believe in her simulants, to care about them. They are they only world she’s ever known.”

“They are the only world there is,” said Kresh. “Her simulants are real-life people.”

“Of course, of course, but my point is that she knows they are imaginary, and yet has begun to believe in them. She believes in them in the way one might care about characters in a work of fiction, or the way a pet owner might talk to her nonsentient pet. On some level Unit Dee knows her simulants are not real. But she still takes a genuine interest in them, and still experiences genuine, if mild, First Law conflict when one of them dies and she might, conceivably, have prevented it. Causing the death of simulants has been extremely difficult for her.

“If she were to find out she had been killing real people-well, that would be the end. She might simply experience massive First Law conflict and lock up altogether, suffer brainlock and die. Or worse, she might survive.”

“Why would it be worse if she survived?” Kresh asked.

Soggdon let out a long weary sigh and shook her head. She looked up at the massive hemisphere and shook her head. “I don’t know. I can guess. At best, I think she would find ways to shut down the whole operation. We’d try to stop her, of course, but she’s too well hooked in, and she’s awfully fast. I expect she’d order power shutdowns, find some way to deactivate Unit Dum so he couldn’t run the show on his own, erase computer files-that sort of thing. She’d cancel the reterraforming project because it could cause injury to humans.”

“The best sounds pretty bad. And at worst?”

“At worst, she would try to undo the damage, put things back the way they were. “ Soggdon allowed herself a humorless smile. “She’d set to work trying to un-reterraform the planet. Galaxy alone knows what that would end up like. We’d shut her down, of course, or at least try to do so. But I don’t need to exaggerate the damage she could do.”

Kresh nodded thoughtfully. “No, you don’t,” he said. “But I still need to talk with her-and with Unit Dum. You haven’t said much about him, I notice.”

Soggdon shrugged. “There’s not much to say. I suppose we shouldn’t even call him a he-he’s definitely an it, a soulless, mindless, machine that can do its job very, very, well. When you speak with him, you’ll really be dealing with his pseudoself-aware interface, a personality interface-and, I might add, it is quite deliberately not a very good one. We don’t want to fool ourselves into thinking Unit Dum is something he is not.”

“But it sounds as if he could handle the situation if Unit Dee did shut down.”

“In theory, yes, Unit Dum could run the whole terraforming project by himself. In practice, all of us here believe you were quite wise not to put all your trust in a single control system. We need redundancy. We need to have a second opinion. Besides which, the two of them make a good team. They work well together. They are probably three or four times as effective working together as either would be alone. And anyway, we’re only a few years into a project that could take a century or more. It’s way too early to think about risking our primary operating procedure and trusting the whole job to backups. What if the backup runs into trouble?”

“All your points are well taken,” said Governor Kresh. “So-what are the precautions I should take in talking to them?”