Program alarms: the cause

Over the decades, many stories have been told about why. when they already had enough to contend with just to land Eagle on the Moon. Armstrong and Aldrin found themselves dealing with what appeared to be a balky computer. The rendezvous radar became deeply implicated in the problem and stories abounded about whether or not it should have been powered at this point. Some said it was a procedural error, or a crew7 error. In fact, there was no problem with it being powered. It was the mode that it was in that was related to the actual problem.

In the 2006 documentary In the Shadow of the Moon, Aldrin. who w ent by the nickname ‘Dr. Rendezvous’ by virtue of his thesis on the subject, saw it as a conflict between the checklist and his operational needs. “Being Dr. Rendezvous,

… I was going to leave the rendezvous radar on and active so if we had to abort, it was on and working and we could reacquire Mike as soon as possible if we had to go up."

Probably the best account of the alarms is told by Don Pyles, then a young engineer at MIT who specialised in the new7 field of software. lie was an integral part of the team who w7rote the computer code for the landing. Given the close relationship the computer had with the spacecraft’s other systems, he knew a lot about them too.

Eyles agrees with Aldrin. "Many explanations have been offered for why the [rendezvous radar] was configured in this way for the lunar landing. For example, a fanciful scheme for monitoring the landing by comparing [radar] data to a chart of expected readings may have been considered by some people in Houston. However, a simpler explanation is sufficient to explain the facts: The [radar] was on for no other purpose than to be warmed up if there were an abort.”

The tale of the program alarms that Eyles tells is quite nuanced. It centres on a little electrical funny associated with two of the LM’s major systems, the computer and the АТС A (attitude and translation control assembly), ‘flic latter mediated between the spacecraft’s controls and the computer. These systems synchronised themselves by way of 28V. 800 IIz AC signals which were meant to match in frequency, which they did. However, the relative phase of the signals was not defined and w7hen the computer was powdered up, which was after the ЛТСЛ had been powered, the phase angle between the two systems could be of any value. If it was near 90 or 210", there were odd consequences. In this condition, data about the pointing angle of the rendezvous radar would make no sense to the rest of the G&1S system if it were in either its Slew or Auto modes (the third mode, ‘LGC. was not implicated). The result of this circumstance was that counters in the computer were continuously being incremented or decremented, an operation that took up valuable cycles of processing time. Unfortunately, upon power-up, this was the situation that Eagle found itself in.

During PDI. the computer was already very busy with all that it had to do to keep the LM on a safe flight path to landing. Ever)’ two seconds, it would run through its list of jobs. These included updating the state vector, controlling the LIVTs attitude, controlling the descent engine’s throttle, adjusting the engine’s gimbals to maintain its aim through the centre of mass, and flying the desired trajectory to the surface. The extra cycles required to deal with the errant counters took the computer very near to the end of its available time before it ran out.

What Look it over the edge in the first instance was a task that Aldrin had to perform when he instructed the computer to display delta-II. the difference between what the computer thought their height was, and the accurate value determined by the landing radar. This task caused the computer to run out of time before it could complete all its allotted jobs, at which point it threw’ up an error code and performed a reset. Thankfully, the software had been written in such a way that, after each reset, it could pick up the threads of all the tasks it had been executing and then continue as if nothing had happened. ‘This resilient approach to software writing is attributed to Hal Laning, another of the software engineers at MIT.

‘The load on the computer increased again when P64 began to run. It had the additional task of calculating LPD angles for the commander and with the errant counters still eating up computer cycles, the time ran out once more. The crew were presented with their second series of alarms. Although the numerical error code was different, the guidance officers recognised it to be of the same basic type as the first, and therefore harmless so long as it did not become continuous.

In 2004, Eyles summed up Laning’s achievement thus, “When Hal Laning designed the Executive and Waitlist system in the mid 1960s, he made it up from whole cloth with no examples to guide him. The design is still valid today. [It] still represents the state of the art in real-time GN&C computers for spacecraft.’’

Since the computer was still doing its primary job flawlessly, despite the alarms, the crewr returned to their roles; Armstrong looking out, and Aldrin keeping him abreast of the numbers. “35 degrees. 35 degrees. 750 [feet]. Coming dowrn at 23 [feet per second].”

“Okay.”

“700 feet. 21 [feet per second] down. 33 degrees.”

“Pretty rocky area," said Armstrong. The erratic LPD angle had swung by a huge amount to 33 degrees and it w’as indicating that they were heading towards an area just outside a large crater known informally as West Crater. It was so named because it was situated on the western end of the landing ellipse. It w:as common for the ejecta blanket around such a crater to include a scattering of large blocks. ‘This

did not look like a place he wanted to set down. Armstrong never got to use the ability of P64 to redesignate his landing site. He was too preoccupied with computer alarms and by the inability of the LPD to give him a trustworthy idea of where the computer was aiming. Instead, he took control, made his decisions and carried them out.

"PICKING UP SOME DUST”: P66 "600 feet, down at 19.” Aldrin continued his litany of data while Armstrong weighed up his prospects. The computer was still behaving and otherwise the descent seemed to be going well. But he had to decide what to do about the block у ejecta around West Crater.

“I’m going to…” he told Aldrin. and assumed manual control of the LM’s attitude by changing to P66. He then pitched forward to an almost vertical attitude that allowed Eagle to maintain its horizontal speed and let him fly over the boulder field of West Crater. Once clear, he pitched the LM backwards again to resume cancelling the craft’s horizontal speed, and he searched for somewhere safe to bring it dowm.

P66 looked after the LM’s vertical speed, also known as its rate of descent (ROD), by adjusting the throttle to maintain a desired value. The commander had a ROD switch that he could flick up or down momentarily to increase or reduce the rate of descent by fixed increments. At the same time, his hand controller let him adjust the vehicle’s attitude, which gave him control of horizontal speed, very much in the manner of a hovering helicopter, ‘l ilt to the left and the engine would aim slightly to the right, pushing the LM towards the left.

“100 feet, 7>Vi down, nine forward.’’ called Aldrin. “Live per cent. Quantity light.” he added.

A light had come on to indicate that they had only 5.6 per cent of their propellant remaining. From pre-flight analysis, planners had decided that, from this point, they could fly safely for only another 114 seconds before they must either land the LM or abort. A 94-second countdown began in mission control that would lead to a call for the crew either to abort or land. If the commander felt he could get the ship dowm within the remaining 20 seconds, he could continue, otherwise he had to get out of there by punching the abort button.

However, Apollo ll’s slosh problem had fooled them again. By triggering the quantity warning light early, it made them believe they had less propellant than was actually available and it came very near to causing an unnecessary abort. A set of fold-out baffles were retro-fitted to Apollo 12’s LM but they were not very effective. It wasn’t until Apollo 14 that the slosh problem was resolved.

Tindallgrams

The manner in which the team decided how to deal with this low-level quantity warning light taps into one of Apollo’s most interesting side stories, because it illustrates the management style of How ard Wilson (Bill) Tindall, one of the senior
engineers. He was an expert on the subtleties of rendezvous and trajec­tories, and became head of the Mission Planning and Analysis Divi­sion. In the hectic days that led up to Apollo’s successes, he coordinated the planning process that threaded together the disparate systems and people to create the bureaucratic edifice that was an Apollo mission.

His method of decision making touched just about every facet of a flight, from the dumping of urine to the position of the Navy’s recovery force or any other thing that was intertwined with the trajectory, and he is considered by many to be a major reason for the success of the programme.

There were two sides to his style.

The first was the manner in which he handled large meetings that involved engineers, programmers, mathematicians, crews or whoever in order to get this diverse mass of people to reach a decision – “knocking people’s heads together”, as one engineer described it. David Scott attended lots of these meetings and shares the admiration that many have for his abilities. “Tindall would control the debates in terms of giving people the opportunity to talk, and then mix and match and make the trades. Then he would make a decision and say, ‘I’m gonna recommend this to management. Anybody have any really strong objections?’ And the guy who lost the debate may say, ‘Yeah, it won’t work!’ And Tindall would say, ‘OK, fine. We’ll go this way and if it won’t work, we’ll come back and re-address it, but we’ll make a decision today.’ They were good debates and anybody could stand up and debate the issue. But he kept it moving. He didn’t get bogged down because he himself was a brilliant engineer. I think Tindall was a real key to the success of Apollo because of how he brought people together and had them communicate in very complex issues. He was very good at it. He’d have them explain it, and in front of all their peers.”

The second side to Tindall’s ability was in the extraordinary memos he wrote, now fondly called Tindallgrams. NASA often displayed the formal stuffiness of a government bureaucracy, yet the memos from this particular senior engineer not only showed how he tied the project’s final stages together, but they revealed a unique chatty, easy to understand style that historians thought was quite remarkable. For example, a memo that discussed the possible reasons for Apollo ll’s overshoot had as its subject line, ‘Vent bent, descent lament!’ Another, written
before the Apollo 11 mission, concerned the LM’s low-level warning light, and was sent to a large list of addressees. It had this wonderful section:

‘"I think this will amuse you. It’s something that came up the other day during a Descent Abort Mission Techniques meeting.

“As you know, there is a light on the LM dashboard that comes on when there is about two minutes’ worth of propellant remaining in the DPS tanks with the engine operating at quarter thrust. This is to give the crew an indication of how much lime they have left to perform the landing or to abort out of there. It complements the propellant gauges. The present LM weight and descent trajectory is such that this light will always come on prior to touchdown. This signal, it turns out. is connected to the master alarm how about that! In other words, just at the most critical time in the most critical operation of a perfectly nominal lunar landing mission, the master alarm with all its lights, bells and whistles will go off. This sounds right lousy to me. In fact. Pete Conrad tells me he labelled it completely unacceptable four or five years ago, but he was probably just an ensign at the lime and apparently no one paid any attention. If this is not fixed, I predict the first words uttered by the first astronaut to land on the moon wall be ‘Gee whiz, that master alarm certainly startled me."’

Sheer engineering magic.