CHAPTER SIXTEEN
The arguments that had begun during Dyer’s first meeting in Washington all boiled down to this: if nothing that the system on Janus was capable of doing could prevent its being deactivated, then all the risks associated with allowing TITAN to grow further reduced to some form that was acceptable; if the system succeeded in devising some form of “unpullable plug,” then the risks were unacceptable.
The object of the experiment was not to find an effective means of destroying the system. After all, given a machine with no previous knowledge or experience of having its survival threatened and without having had any opportunity even to become aware that it was vulnerable, that end could have been achieved with almost absurd simplicity: just switch it off and that would be the end of the exercise. The objective was to find out how effectively its defenses would evolve in response to repeated demonstrations of its vulnerability. It was hoped to simulate the effects of things that might occur in the normal environment on Earth, things that could be insignificant to Man but which a system that had developed a survival drive might interpret as potential threats—power cuts, for example.
The only way to bring it to the system’s attention that it was vulnerable at all would be to go ahead and switch it off, and let its reasoning abilities figure out the implications. But obviously if it was switched off, it would be incapable of reasoning about anything at all, never mind taking any action to protect itself. Which said they’d have to switch it on again.
With power restored the machine would, because of the way it had been programmed, react to the knowledge that it had been halted, becoming mildly concerned and somewhat curious. Repetitions of the process—simply switching key parts of the machine off and then on again—would reinforce its reaction to the “discomfort” until, like a dog with an itch, it would begin experimenting to find ways of making the discomfort go away.
Kim’s group was responsible for developing the programming that would produce this behavior, and work had continued throughout the final weeks at CUNY and later in the computer lab set up at Fort Vokes. Progress in this area was on schedule.
But all that this would result in so far would be a computer that worried. Even if it suffered agonies of paranoia, what could it do about them? As Ron had said, computers were not equipped to carry rifles or throw grenades at suspected assailants. This was where Fred Hayes and his group came in. In a makeshift lab in the “Egghead Block”—a building that had been allocated to the scientific team for work space—Hayes described some of the techniques that the system was expected to experiment with in devising methods of self-defense.
“Here’s an example of one of the structural modules used in the construction of buildings on Janus,” he said, gesturing for the others to follow him over to the open area by the door of the lab. They formed a rough circle around an eight-foot-high panel, formed from some coated sheet material and reinforced by a sturdy-looking frame of aluminum sections. It was standing vertically in a supporting jig away from the walls on a clear part of the floor, and allowed them plenty of room to study it from all angles. Dyer walked slowly round the panel, casually taking in details of the alignment lugs and securing catches along all four sides, and came to where Frank Wescott was leaning forward to run a finger experimentally across part of its surface.
Frank had a pale thin face whose planes came together at sharp angles. He wore his hair short and parted in an old-fashioned style and his tight-lipped mouth had a permanent downturn at the corners, which gave him the appearance of being somewhat humorless and fussy. In fact, he could be just that at times, but he was first-rate at pinpointing elusive bugs in horrendously complex programs and that was what mattered.
“I thought it would be plastic and cheap,” Frank said, looking up. “But it’s not. It’s difficult to tell exactly what it is. Feels quite good and strong, though,” He sounded mildly disappointed.
“If Janus is made out of moonrock I don’t think you’ll find much plastic there,” Dyer remarked. “I don’t think you’ll find much of anything that needs carbon. It’s probably some silicon-based stuff.”
“Is this made out of lunar material?” Chris asked Hayes, a few feet away from them.
“Yes it is,” Hayes replied. “It was one of the ones churned out in Detroit while Janus was being built.” He raised his voice to address all of the half-dozen or so persons present.
“This is an example of one of several kinds of standard wall module,” he said. “No two buildings on Janus look alike, yet they’re all constructed by putting together a comparatively few types of standard module like this one. There are modules for walls, floors, roofs, ceilings and so on, and some special types such as see-through panels for sun porches or windows or whatever. Anybody can put ’em together and you can design your own house and put it up in a day. The number of possible combinations is more than the whole population could get through in a lifetime.”
The faces around him were polite but not really all that interested. They knew that Hayes was not there to talk about aesthetics and architecture.
“Modular buildings aren’t really new,” he went on, as if reading their minds. “But here’s something that you won’t find in any modular buildings down here on Earth . . . not yet anyway.” He indicated a flat rib encapsulated in an insulation coating that transversed the rear of the panel fully from one side to the other. Both ends terminated in identical blocks at the edges, suggesting connectors of some kind.
“Datastrip,” Wescott guessed. Hayes nodded. Somebody gave a low whistle of approval.
Datastrip was something that had been under development for a few years and which was reported in the professional journals from time to time. Essentially it meant that every structural module of every building of Janus carried a length of integral electrical bus to which any device designed to communicate into TITAN could be coupled, either by direct connection or by radiated energy as was the case with portable things like viewpads. As all the modules were assembled together to form structures, the strips connected up automatically to provide plumbed-in TITAN wherever you happened to be. The network thus formed a tree which grew as the building grew, all its twigs finding their way eventually back to one of the trunks of the primary data highways.
“Not just passive wiring either,” Hayes told them. “It’s got its own switching and routing microprocessors built into the connector blocks.” That meant the buildings themselves would constitute functional extensions of the network’s total switching hierarchy. It heralded the day when whole cities could be designed as living cells in the planetary organism, not just as boxes of inanimate steel and concrete through which electronic neural tissue was threaded afterward.
“This is the first line of defense that we expect the system to try,” Hayes went on. “The Datastrip distributes power lines throughout Janus as well as network intelligence. When we start breaking its connections it’ll almost certainly start creating bypass links to neutralize the breaks. With billions of combinations to choose from and thousands of computers available to do the figuring, it ought to have no trouble finding ways through a lot faster than we can block them.” He shrugged. “That, of course, is one of the things we’d very much like to find out more about.”
While Hayes was talking, Dyer studied the expression on Wescott’s face. Frank’s features were blank but his eyes betrayed a lack of conviction. It was the look of somebody in the position of having to listen to a sales pitch on something he’d already made his mind up not to buy. Frank was still convinced that the risks had been outrageously exaggerated, that the whole exercise would prove an expensive and pointless waste of time, and he had said so.
“If you route the primary power of a machine through a manually controlled switch there’s no way in hell it’s gonna stop you from unplugging it,” he had told Dyer and Krantz during a debate shortly after his arrival. “If you decide to throw the switch then that’s the end of it. You don’t need an army and your own private world to prove that. We’ll just wind up looking like the biggest bunch of assholes in the business with millions of dollars gone down the tubes to account for.”
Frank had maintained his stance ever since Dyer’s first trip to Washington with Richter, so his attitude had come as no surprise. At first Krantz had expressed doubts at the wisdom of having Frank along at all, but Dyer defended the choice on the grounds that in science, as with most things, good ideas flourished best on a diet of varied opinions. People who all thought the same way tended to expend a lot of time and energy merely reinforcing one another’s prejudices instead of solving problems. Schroder had agreed with Dyer and in the end Krantz accepted the vote. Dyer had guessed that Frank’s morbid streak would compel him to come along if only to see his prophesies fulfilled, and that was exactly what happened.
“If the system turns out to be incapable of bypassing our attempts to shut it down, then we’ll have won and there’ll be nothing more left to do,” Hayes said. “If that happens we can carry on and upgrade TITAN without any further worries. But if that did happen, I must confess I’d feel very disappointed in it. I’m sure it could do better.” Wescott sniffed pointedly but didn’t take the matter any further. “So now we’ve reached a point, hypothetically, where the system has successfully neutralized the built-in breakpoints that were supposed to guarantee we’d always have final control over it,” Hayes went on. “What do we do then?”
He cast an eye around the group to invite suggestions and began moving slowly toward the end of the lab, where a demonstration of some kind seemed to have been set up. On a bench was a stripped-down electronics mounting box which contained a battery of standard honeycomb blocks—the high-density receptacles used universally for holding the scores and often hundreds of molecular-circuit cartridges that were interconnected to form computers and practically every other kind of complex system. A second bench about ten feet to the right carried a smaller assembly of honeycombs held in a metal frame and coupled to an elaborate mechanism of shafts, cylinders, linkages and motors. Three sections of Janus-style wall module stood edge to edge to provide a backdrop for the display, presumably affording a connection between its two parts via Datastrip.
“Well, if you found you couldn’t control the net, you’d have to start isolating sections of it until it lost integrity,” a short, balding, pink-faced man offered in reply to Hayes’s question. His name was Eric Jassic. He was one of Schroder’s CIM scientists from Washington, a specialist in communications techniques who had made significant contributions to ultra-high-frequency optical multiplexers.
“Why screw around?” Ron demanded. “Just go to where the goddam processors are and unplug ’em there.”
“Very well,” Hayes agreed in a pleasant voice. “Let’s try.” A few heads exchanged quizzical looks as the group formed a loose gaggle between the two benches. Hayes tapped a rapid command into the touchboard of a flatscreen panel hanging on an arm at one end of the left-side bench. At once the mechanism on the other bench came to life, with a flurry of whines, clunks and hisses. After watching it for a few seconds they realized that it was nothing more than an automatic component-forming machine, the kind used in thousands of manufacturing plants around the world. An injection moulder ejected cast blanks at the rate of one every couple of seconds, which then passed through a series of cutting and drilling operations, eventually finding their way through to a spring-loaded magazine in which the finished parts were being stacked. The magazine would normally convey them onward to the next stage of whatever assembly process they were intended for.
“It doesn’t really matter what those widgets are,” Hayes commented cheerfully. “But if you want to know, they’re part of the end-bearing for a room-temperature superconducting clutch. You’re all probably familiar with this kind of machine, at least in principle.” He indicated the right-hand bench with a vague gesture of his arm. Most of them were. It was just one example of many types of general-purpose machining robots in widespread use. Such machines were general-purpose in the sense that they were programmable and could produce a virtually unlimited variety of parts depending on the commands loaded into them. They were descended from the specialized machine tools that had been used for many years in mass-production plants, but were far more versatile. Presumably the small honeycomb next to the machine was the local computer that stored and interpreted the programs.
“The large computer here on the bench is the remote supervisor,” Hayes informed them, tapping his fingers against the larger honeycomb. “It’s coupled into the machine’s own processor in the usual way, except that to make things a little more authentic we’ve used Datastrip à la Janus. The supervisor downline loads the programs of what’s wanted and the local processor does the rest. Also, the supervisor performs remote diagnostics via the link to make sure that all’s going well at the other end. Okay?” Nobody had any queries or comments. Everything that Hayes had described was standard practice. They waited, curious to see what would come next. The machine clunked and whirred, churning out its widgets with obvious contentment.
“If the machine packed up, the normal thing to do would be to get a diagnosis from the supervisor and send someone to fix it,” Hayes continued. “At least, if it happened today it would. But that takes time and people would rather be doing more interesting things, so probably in years to come we wouldn’t bother. What we’ll probably have is something like we’re putting into Janus today.” His eyes twinkled as he looked from face to face around him, as if he were enjoying some joke and were waiting for them to see it. He was evidently amused, but at the same time he seemed to be waiting for a response to some implied challenge that he hadn’t voiced. “Well?” he asked after a while. He caught Dyer’s eye for an instant and winked almost imperceptibly. Dyer had seen this demonstration about a week earlier. He didn’t want to spoil Fred’s fun, so said nothing.
“Aw, quit the fooling around, Fred,” Ron exclaimed at last. “What the hell are you waiting for us to say? Okay, we’re making widgets. So what?”
Hayes couldn’t contain a smile any longer.
“You’re not supposed to say anything,” he replied. “You’re supposed to stop it.”
“Stop what?” Ron looked confused.
“The machine,” Hayes said. “See if you can stop it making widgets. In other words put a fault into it. That’s the game.” Dyer grinned to himself as he saw a crimson tide of exasperation boiling up out of Ron’s collar. Chris was standing next to him, frowning thoughtfully and looking from Hayes to the machine and back again.
“Stuff all this,” Chris said suddenly. He stepped forward and stood in front of the bench. “Just stop it,” he repeated. “That’s all we have to do, right? Any way we like.”
“Yes,” Hayes answered.
“Right.” Chris opened one of the drawers below the edge of the bench, cast a quick look around inside, closed it and tried another one, grunted to himself and lifted out a tray containing tools. He selected a dental probe, put on a pair of binocular magnifiers and leaned forward to peer closely at the face of the honeycomb. “Let’s see now . . .” he mumbled to himself. “There’s a row of oh-eight-sevens here . . . probably the main processor array. Must be part of the guts of it anyhow. Let’s have a go anyway . . .” He probed delicately into the honeycomb and with a smooth practiced motion extracted one of the microcartridges. The widget-maker promptly clunked to a halt. Chris deposited the cartridge carefully on a watchglass and stepped back with a shrug. “One widget whatsit bites the dust,” he announced.
“What’s that supposed to prove?” Ron asked.
“I know him. He’s up to something,” Jassic said. Hayes continued smiling.
A sudden rushing sound, like that of high-velocity ducted air, mixed with a fainter electric whine, came from halfway up the wall to their right, causing all heads to whirl around in unison. The metal racking against the wall there had looked like ordinary storage shelving and nobody had taken much notice of it. But now that their attention was drawn to it, they could see that it was something else. It was an array of open compartments that looked like pigeon holes for mail, except that each was a foot or more square. There must have been at least two dozen compartments. A few were empty but each of the others contained one of an assortment of unidentifiable objects. Some gleamed bright and silvery and were about the shape and size of ordinary toasters; others were dull and cylindrical, while still others suggested nothing familiar at all but with their sprouted tangles of rods, hooks, antennas and claws resembled, if anything, gigantic mutant insects.
The noise was coming from one of these objects. The object that it was coming from was a dull-gray cylinder about six inches across, lying on its side on top of a flat tubular framework that contained a mass of tightly packed gadgetry and wiring. The near end of the cylinder was distinctly insectlike, with a profusion of miniature probes and jointed arms, and a circle of recessed windows that could have been lens apertures.
The whole thing was starting to move.
As they watched speechless, it slid smoothly out of its cell like a metal wasp emerging from its nest, and hung in midair a foot or so in front of the pigeonholes. Then it dropped vertically for a short distance, aligned itself in the direction of the right-hand bench, and began moving at about chest height off the ground. Chris jumped out of the way in sudden alarm.
“It’s okay,” Hayes said with a laugh. “Stay there.”
“Up yours, mate,” Chris breathed shakily.
The wasp homed unerringly on the face of the honeycomb. It extended three of its tiny arms sideways to lock onto the registration pins located at intervals across the face and then, holding itself quite steady in the air, traversed slowly sideways until its axis was aligned with the array element from which Chris had taken the cartridge. Nobody could see quite what happened next because the wasp was flush against the face, but suddenly the widget-maker clicked into life again. The wasp detached itself and turned back to point at its cell. Just as it started moving, Hayes stepped forward and placed himself in the way. The wasp paused for a split second, then made a smooth arc around him, reversed itself back into its cell, and died.
A burst of excited chattering suddenly broke out to greet the performance. There was no need for Hayes to explain what had happened. It didn’t take much thought to see that other wasps, equipped with suitable tools and carrying the right selection of parts, could replace far more things than just electronic microcartridges, provided of course that the equipment being serviced had been designed for it.
“They’re called drones,” Hayes told them. “I’m sure I don’t have to spell out the idea. There’s a whole zoo of them to cover lots of different special functions. Most of the work on them has been done in Japan. This’ll be the first time anybody’s seen ’em outside a few R & D labs. How d’you like ’em?”
“I’m still not sure what it’s supposed to prove,” Frank Wescott said. “What are you trying to tell us . . . that the Janus system will be able to fix itself even if we try deactivating it? I don’t believe it, Fred. All it says is that routine repairs are going to become more automated. Okay, that’s good. But there are still lots of ways I can think of to shut a machine down that things like that couldn’t handle . . . ways that need people.”
He looked across at Dyer. “That’s my whole point, Ray. We can always bust it in ways that only people can fix. As long as that’s true I can’t see what there is to get worried about. I’m sorry, but I just can’t see the point.”
“Go ahead and show us,” Dyer invited. A hush of interest descended on the room. Wescott moved forward to survey the system before him for a few seconds. The widget-maker clacked away happily while its now-full magazine was whipped away and replaced by an empty one.
“Mmm . . .” Wescott said. “The supervisor here runs the diagnostics . . . so the supervisor must be able to figure out the fault to be able to call in the right drone. There must be a comm channel from the supervisor to the drones somehow . . . probably radio via the Datastrip,” He cocked an eye at Hayes. “How’s that? Am I about right?”
“Right on the ball,” Hayes said approvingly.
“And you did say we could try anything we like.”
“Yes I did.”
A gleam of unabashed malevolence came into Wescott’s eye, He was going to enjoy this exercise. He rubbed his palms together and stooped to use the tools that Chris had left on top of the bench.
“Want a hammer, Frank?” Ron called.
“That’s crude,” Wescott called over his shoulder. “I’ve never found a machine yet that I couldn’t outsmart.” And that’s why you’re here Frank, Dyer thought to himself.
Frank began with a simple trick. He removed a cartridge just as Chris had, and then located the connection from the Datastrip to the widget-maker’s processor and jerked it out. Then he removed a second cartridge. With the connection to the supervisor broken, there was no way that the supervisor could deduce that the second cartridge had gone. Frank wanted to know what it would do when the drone replaced the first cartridge and nothing happened.
The same drone as before emerged from its cell and did its party-piece. The widget-maker remained paralyzed.
“Don’t tell me it’s quit,” Wescott said scornfully. “A kid of two could have thought that one up.”
“Not on your life,” Hayes replied.
An electric-toaster drone came out, hovered alongside the computer-in-distress and plugged itself into an auxiliary test socket. Silently it communicated its findings back to the supervisor and the supervisor thought about the situation.
“It’s called in the flying doctor,” Chris mused.
A spherical drone, bristling with lenses, joined in the act next and proceeded to drift slowly back and forth a few inches from the honeycomb, rotating turrets to switch in different viewers while it studied the scene from all angles. The doctor unplugged itself and backed off to hover a couple of feet back, uncovering the point where the cable that Frank had disconnected was hanging an inch away from its socket. The scanning drone zeroed in immediately and a few moments later a crab drone descended and restored the connection. After that it was pure routine for the cartridge-injecting drone to do its thing again and the widget-maker was back in business.
“Come on, Frank, what’s the matter with you,” Ron jeered. “I thought you were gonna outsmart it.”
“Electrons one, humans nil,” Chris declared. “Round two coming up.”
Frank gritted his teeth and turned back to study the layout with a new respect. “Okay you bastard, you’ve asked for it!” he growled.
This time he didn’t bother trying swapping cartridges around. Obviously the drone would be designed to extract duds as well as inject replacements, and would therefore be just as capable of swapping them back again. He disconnected every cable he could find both at the machine’s control computer and within the mechanical labyrinth of the machine itself. But after a brief conference between the supervisor, the sphere drone and a scurrying-crab drone, all the cables were plugged in again. He traced the main data cable that connected the local computer to the machine, disconnected it at both ends, undid its restraining clips, removed the cable completely and threw it in the trash bin; a tubby, jolly-looking drone bustled down to attach a new cable and the crab showed off its versatility by nudging the cable into the restraining clips and snapping them shut. The audience were joining in the spirit of the game and suggested cutting cables, filling disconnected sockets with resin cement and flattening the pins of plugs with pliers, but after some debate they concluded that things like that would probably be of no avail; if the drones could replace cables, they could no doubt replace broken cables and fouled-up components just as easily. Hayes confirmed it and they believed him.
Frank tried a new approach. He disconnected the main drive motor of the machine, then used a spanner to dismount the whole motor and heaved it out of the machine completely. It was heavy and he needed both arms to hoist it. Everybody watched with rising suspense as the sphere drone fussed back and forth around the machine inspecting the damage. Surely there was little that the drones could do about that. The supervisor worried in silence for a long time, and Frank began to look grimly satisfied.
And then a drone larger than any they had seen so far trundled itself out along the floor. Obviously this one didn’t fly. It was about the size of an upright chair and looked something like a cross between a lawn mower and a forklift truck. As they watched in astonishment, it rolled across to the far side of the lab and slid its lift underneath a spare motor that was lying on a low shelf while the sphere drone followed it and hovered nearby, presumably to act as eyes. Then fork lift trundled back to the machine, jacked the motor up to the height of the mounting flange, and an extending ram pushed the motor off the lift and slid it neatly onto the studs. While the fork lift held it another drone secured it with a rotating nut-driver bit, after which crab drone restored the electrical connections and immediately the widgets started flowing again.
Frank got mad, stamped around to the rear of the wall modules and put an axe through the Datastrip rib. A garbage-disposal drone walked along the wall astride the rib, lifting it from the surface, cutting it into twelve-inch lengths and stacking them on its back while a strip-laying drone spun a new one behind it, followed by a chattering crab drone which fastened the new strip securely to the surface. Frank used an RF probe to measure the field being radiated from the strip to control the drones, and tried jamming it with an oscillator-fed antenna, but the supervisor simply changed frequencies faster than he could match.
In the end he did win, but only by deactivating the supervisory computer. It was a hollow victory, however; if the machine had been controlled through a network the size of TITAN instead of from a single computer isolated on a lab bench, the controlling processor could have been any one of thousands located anywhere. Deactivating that would have achieved nothing since the network could simply have substituted another.
When the demonstration was over, Dyer addressed the faces around him, which had become very quiet.
“Some of you might be wondering why we should bother giving the System things like this to defend itself with at all if it’s going to make our job more difficult. Why not simply lay off making drones in the first place?” A few puzzled heads nodded. “But there really isn’t anything especially significant about the drones,” he told them. “They just represent a situation in which the System has greater control over resources and over itself than anything we’ve seen with TITAN. In years to come there will be lots of other things around besides drones. The question is, if we ever gave a system autonomy comparable to this on a global scale, how far could it go in using it in ways that we never intended it to? That’s what we want to find out.”
“Is there any chance that these things could end up being used as weapons if the System turned nasty?” one of the CIM people asked.
“It’s a possibility that we have to allow for,” Dyer replied. “Fred and his crew have been working with the Japanese on developing some specially modified versions that use a number of methods to deactivate or destroy other drones. They can be operated independently of the System if need be, for example via lasers or wires as well as by radio. So if it does turn nasty, we can send in our own antidrone drones after its troops anywhere they can go. I think that when you’ve had a chance to see what we’ve got and play with them, you’ll find we’re in pretty good shape.
“You all know that the Janus System will be a step ahead of TITAN in terms of managing a whole planet. It will have control over the life-support, power distribution, transportation and that kind of thing, so obviously it could play a lot of unfriendly tricks if it ever recognized us as adversaries and discovered our weaknesses. Well, we’ve put a lot of work into analyzing the kind of things it might possibly do, and we’ve built in all manner of safety overrides to make sure we always have ultimate control over it. You’ll be seeing more of those over the next few weeks and until then don’t worry too much about it. Janus will be full of things that we’ll know about because we put them there, but the System won’t.”
A chorus of mixed murmurings broke out on all sides as he finished speaking. In the middle of it.
“You’re telling us the dumb bastard isn’t going to outsmart us, right?” Frank called out.
“Right!” Dyer told him and grinned. Everybody laughed and the atmosphere at once became more cheerful.
Later on, when some of them were having a nightcap in the Officers’ Mess, Ron turned to Chris. “What if Fred’s antidrone menagerie isn’t up to it? D’you figure an M25 could stop one of those things?”
“No problem,” Chris told him. “It’d drill straight through one of those tin cans.” Somehow his tone failed to echo the confidence of his words. He sat back and rubbed his chin thoughtfully, then added, “Although to be honest, I wouldn’t mind having a Gremlin handy as well . . . just in case.”