Spinning Out of Control

It’s a space scientist or engineer’s worst nightmare. Your mission is working flawlessly and then, due to oversight or unforeseen events, its starts tumbling out of control. Several hundred million dollars’ worth of hardware turns a blind eye to its scientific mission. Deaf to telemetry and far beyond the reach of astronauts, it languishes uselessly in deep space.

On a June evening in 1998, mission controllers at NASA God­dard Space Center in Maryland took note as SOHO put itself into Emergency Sun Reacquisition mode. This mode is a safe “holding pattern” autonomously entered by the spacecraft any time it en­counters anomalies. From this point, the ground controller sends a special sequence of commands to recover normal science opera­tion, the first step of which involves pointing the spacecraft at the Sun so it knows where it is. Since SOHO had entered this mode five times since its December 1995 launch, the controllers weren’t too worried. But this time was different. Over the next few hours, a series of mistakes seemed to doom the spacecraft, first causing loss of attitude control, followed by an interruption in crucial te­lemetry, then loss of power, and finally loss of all thermal control. It was a bleak day, and it appeared to be the death knell for the young mission. In the control room, the tension grew in waves.20 At the first entry into Emergency Sun Reacquisition (ESR) Mode, engineers calmly issued the commands to point SOHO at the Sun by carefully orchestrating its three roll gyroscopes. Project scientist Bernhard Fleck said there was no undue concern. Even when a sec­ond ESR was triggered a few hours later, there was no panic; engi­neers had seen this before too. But as the roll thrusters were firing to point SOHO at the Sun, a third ESR was encountered. Out in deep space, a million miles from Maryland, SOHO was spinning faster and faster. Then the communication link went down, pre­sumably disrupted by the spacecraft’s wild motion. SOHO was unreachable.

The subsequent investigation showed that the spacecraft per­formed as designed; all the errors were human. In hindsight, Fleck thinks they gained a false sense of security after two and a half years of operations.21 Like pilots who had done many takeoffs and landings and who had flown on sunny days and through thunder­storms, they thought they had seen it all. They didn’t realize they were in a nosedive.

With the spacecraft seemingly lost, NASA and ESA quickly con­vened a review board to issue a diagnosis and post mortem.22 The review board concluded that seat-of-t he-pants decision making in the control room exacerbated the problem. Controllers erro­neously removed the functionality of SOHO’s normal safe mode and misdiagnosed the state of two of the three gyros. At any time during the mishap, if they had verified that Gyro A was not con­trolling the roll angle properly, they could have avoided the serious problem. The mundane and somewhat sheepish conclusion: when a very complex system is made and operated by human beings, sooner or later they’re going to do something wrong.

Ground controllers kept trying to send messages to SOHO using the Deep Space Network, but they had heavy hearts. The spacecraft was in an uncontrolled spin, possibly at a rate that could cause structural damage. Engineers believed it was spinning with its solar panels edge-on to the Sun and so not generating any power. As a result, the batteries and the onboard fuel would have frozen into a state from which they could not be recovered. But there was a window of opportunity that admitted a sliver of hope. SOHO’s steadily changing orbit with respect to the Sun was increasing the illumination on the solar panels for a few months and they would give the batteries a chance to recharge. Twelve hours a day, controllers “pinged” SOHO in the hopes of getting a response. Then, a month after contact with the spacecraft had been lost, a break. The huge 305-meter radio dish at Arecibo was able to bounce radar off SOHO and the refection was picked up by the Deep Space Network dish at Goldstone. SOHO was spinning at a modest rate of one revolution per minute. Communicating with the frozen spacecraft was not easy, but six weeks after contact had been lost a feeble signal was received. SOHO was alive.

Over the next two months, engineers clawed SOHO back from the brink. They thawed out its batteries and fuel lines and were gratified that none of the scientific instruments seemed worse for wear after having been subjected to temperature variations from +100°C to -120°C. Some of the instruments even improved their performance after experiencing this bracing range of temperatures. SOHO was back, but two of the three gyros were not respon­sive due to the series of mishaps. As luck would have it, the third gyro failed a few months later. Although the gyroscopes were not needed to gather science data, maintaining a Sun-oriented attitude used valuable fuel and lowered the margins of safety if anything else should go wrong. Undaunted, the engineers made lemonade with their lemons. They developed special software to enable the spacecraft to point using the control reaction wheels, the first time an ESA spacecraft had ever been operated without gyroscopes. For all the detailed planning, SOHO ended up living by its wits in a series of knife-edge decisions.