The NASA School of Hard Knocks

In the summer of 1961, JPL engineers prepared the first Ranger spacecraft for flight. Having successfully passed through a series of structural, electrical, and environmental tests, Ranger 1 was scheduled for launch in July. After two hardware component failures in the spacecraft and one in the launch vehicle delayed the launch, the Atlas-Agena took off in August. Within minutes, engi­
neers found that the Agena upper stage did not ignite, stranding Ranger 1 in Earth orbit. The spacecraft operated properly but burned up in Earth’s atmo­sphere on August 30. Ranger 2 launched in November. Its Agena also failed, placing Ranger 2 in a low orbit, from which it soon disintegrated in the Earth’s atmosphere. The air force launched an investigation of the failures and pres­sured Lockheed for solutions.54

Ranger 3 was the first of JPL’s Block 2 design, which included new science experiments as well as a television camera to take lunar surface photographs for Apollo. In January 1962, the air force’s launch vehicles placed Ranger 3 on a trajectory that would miss the Moon, but JPL decided to operate the spacecraft as close as possible to the normal mission. While performing the first-ever trajectory correction maneuver (firing thrusters to change the space­craft’s course), a reversed sign between the ground and flight software re­sulted in the spacecraft’s course changing in exactly the opposite direction from that desired. Two days after launch, the spacecraft computer failed, end­ing the mission prematurely.55

Launching in April 1962, Ranger 4’s computer failed almost immediately after separating from Agena. This made the spacecraft blind and dumb-able to send no data to Earth and unresponsive to commands. Its trajectory was nearly perfect; the spacecraft crashed onto the Moon’s surface three days later. Although NASA and the newspapers proclaimed the mission a great success, which it was for the air force’s launcher, it was a complete technical failure for JPL. Engineers hypothesized that the failure was due to a metal flake floating in zero gravity that simultaneously touched two adjacent electrical leads.56

JPL next scheduled its two Mariner R spacecraft for launch. The first launch on July 22 failed after five minutes, when Atlas’s guidance system malfunc­tioned. Air force and General Dynamics engineers traced the problem to a missing hyphen in a line of software code. Mariner 2 launched in August and successfully flew past Venus in December. Its outstanding performance kept Congress and NASA headquarters from losing all confidence in JPL.57

Burke and his team made several changes to Ranger’s design and organi­zation between Ranger 1 and Ranger 5. On early flights, division chiefs inter­fered with spacecraft operators, so Burke reserved authority to command the spacecraft to project managers and engineers. Engineers also added redun­dant features to the spacecraft. Despite proof that heat sterilization compro-

Ranger. The failures of the first six Ranger flights led to NASA and congressional pressure to strengthen project management. Courtesy NASA.

mised electrical component reliability, lunar program chief Cliff Cummings maintained NASA’s sterilization policy. In any case, Ranger 5’s components had already been subjected to sterilization. Project engineers formed investi­gation groups to study interfaces and spacecraft subsystem interactions. They also added a system test to check interfaces and interactions between the spacecraft and flight operations equipment and procedures.58

These changes did not completely resolve the project’s problems. Ranger 5 launched in October 1962 and began to malfunction within two hours. This

time, the power system failed, losing power from the solar panels. Shortly thereafter the computer malfunctioned, and the battery drained within eight hours, causing complete mission loss. The spacecraft’s disastrous failure con­vinced JPL and NASA headquarters managers that the project was in serious trouble. Both launched investigations.59

The headquarters investigation board’s findings left no doubt that JPL’s organization and management of Ranger were at fault. They criticized JPL’s dual status as a contractor and a NASA field center, noting that JPL received little or no NASA supervision. Project manager Burke had little authority over JPL division chiefs or the launch vehicles and no systems engineering staff and showed little evidence of planning or processes for systems engineering. JPL’s approach-using multiple flight tests to attain flight experience-had strong traces of its army missile background. The board believed this “mul­tiple shot’’ approach was inappropriate. Spacecraft had to work the first time, and testing had to occur on the ground, not in flight. The board character­ized Ranger’s approach as ‘‘shoot and hope.’’ Supervisors left design engi­neers unsupervised, and design engineers did not have to follow quality as­surance or reliability recommendations.60 The Board’s position was clear: ‘‘A loose anarchistic approach to project management is extant with great empha­sis on independent responsibilities and individual accomplishment. . . This independent engineering approach has become increasingly ingrown, with­out adequate checks and balances on individual actions. Pride in accomplish­ment is not a self-sufficient safeguard when undertaking large scale projects of international significance such as JPL is now undertaking for NASA.’’61

JPL’s rapid growth also caused problems. From 1959 to 1962, JPL’s budget grew from $40 million to $220 million, while its staff grew from 2,600 to 3,800. Most of the new budget went into subcontracts, yet JPL’s business manage­ment capability did not expand accordingly. Often JPL personnel ‘‘failed to penetrate into the business and technical phases of subcontract execution.’’ This was due in part to JPL’s ‘‘two-headed’’ nature, as a contractor to the gov­ernment, and as a de facto NASA field center. NASA did not require JPL to use NASA or Department of Defense regulations, treating it as a contractor. On the other hand, NASA headquarters did not supervise JPL tightly like a con­tractor, because of its quasi-field center status.62 Ranger managers had neither

the authority and resources to carry out their mission nor the oversight that might lead to discovery and resolution of problems.

Failure in the race for international prestige was no longer acceptable to NASA headquarters.63 The board recommended strengthening project man­agement, establishing formal design reviews, eliminating heat sterilization, assigning launch vehicles either to NASA or to the air force, instituting a fail­ure reporting system, clarifying Ranger’s objectives, and eliminating extra­neous features. To spur JPL into action, the board recommended that NASA withhold further projects until it resolved Ranger’s problems.64

JPL Director Pickering quickly replaced lunar program chief Cummings with Robert Parks and project manager Burke with former Systems Divi­sion chief Harris Schurmeier. He gave them the authority that Cummings and Burke only dreamed of: power over division chiefs in personnel matters. He also created an independent Reliability and Quality Assurance Office with more than 150 people and gave this office authority over the engineers.65

Schurmeier instituted process changes to strengthen Ranger’s systems en­gineering. He levied an immediate design review, adopted Mariner’s failure reporting system, and ensured that engineers took corrective action. Schur­meier also instituted Mariner’s system of engineering change control and de­sign freezes. Ranger expanded the use of its Design Evaluation Vehicle to test component failure modes and inter-subsystem ‘‘cross-talk and noise.’’ Schur- meier required that engineers plan and record each test on new test data sheets. Ranger 6 also adopted conformal coating, a method to cover all ex­posed metal wiring with plastic to preclude short circuits from floating debris. NASA headquarters and JPL delayed the next Ranger flight until early 1964 to ensure that the changes took effect.66

During 1963, JPL scheduled no launches but concurrently worked on Ranger 6, Surveyor, and Mariner Mars 1964 (MM64). MM64 was JPL’s next planetary project, planned to launch in November 1964 to fly by Mars. Ad­ministrative mechanisms for Ranger 6 and MM64 converged during this time, while managers paid relatively little attention to Surveyor, which JPL con­tracted to HAC.67

The first test of Ranger’s enhanced management was Ranger 6’s launch in January 1964. Whereas Ranger 5 had fourteen significant failures in testing prior to launch, Ranger 6 had only one subsystem failure during its life cycle testing, boding well for its future. The only hardware that caused trouble was the single new major element, the television camera from Radio Corporation of America (RCA). Ranger 6’s launch was flawless, except for an unexplained telemetry dropout. So too were its cruise and midcourse correction. Expecta­tions rose as Ranger 6 approached its final minutes, when the spacecraft was to take pictures just prior to crashing onto the surface. Reporters, engineers, managers, and scientists waited anxiously in a special room at JPL during the last hour for the first pictures to be broadcast. None ever came. Pickering, who arrived just prior to Ranger 6’s impact, was humiliated, saying ‘‘I never want to go through an experience like this again—never!”68

Investigations ensued, followed this time by congressional hearings. JPL’s board isolated when the failure had occurred and which components had failed. Board members could not determine the cause of the failure but none­theless recommended engineering changes to deal with the several possible causes. NASA Associate Administrator Robert Seamans assigned Earl Hilburn to lead the headquarters investigation of the troubled program. Hilburn’s re­port claimed that the spacecraft had numerous design and testing deficien­cies and called for the investigation to be expanded to the entire program. Project personnel vehemently disagreed with the tone and many specific rec­ommendations. JPL incorporated some minor changes but refused to modify its testing practices.69

Minnesota congressman Joseph Karth headed the congressional investiga­tion, which focused on JPL’s status as a contractor and NASA field center as well as its apparent refusal to take direction from NASA headquarters. Con­gress recommended that JPL improve its project and laboratory management. Pickering complied by strengthening project management, but he hedged on hiring a laboratory operations manager to assume some of his responsibili­ties. When NASA Administrator James Webb refused to sign JPL’s renewal contract unless it complied, JPL finally hired Maj. General Alvin Luedecke, retired from the air force, as general manager in August 1964. While the suc­cessful flight of Ranger 7 in July 1964 finally vindicated the troubled project, NASA managers became concerned with JPL’s third program, Surveyor.70

By late 1963, JPL managers spotted trouble signs as Surveyor moved from design to testing. Despite indications of escalating costs and slipping sched­ules, personnel limitations and preoccupation with Ranger and Mariner pre­vented JPL managers from adding personnel to Surveyor.71 However, NASA headquarters managers held a design review in March 1964 to investigate.

The headquarters review uncovered difficulties similar to those on Ranger. Surveyor’s procurement staff at JPL was ‘‘grossly out of balance’’ with needs, far too small given that the project consumed one-third of JPL’s budget. The review team recommended that both JPL and HAC give project managers more authority over the technical staff. It also recommended that JPL have ‘‘free access to all HAC subcontractors,’’ that JPL schedule formal monthly meetings with HAC and the subcontractors, and that JPL’s Reliability and Quality-Assurance Office closely evaluate hardware and testing. HAC’s man­agement processes also received criticism. The HAC PERT (Program Evalua­tion and Review Technique) program did not account for all project elements, leading to inaccurate schedule estimates. HAC’s change control system, in­herited from manufacturing, was unduly cumbersome. Most critically, HAC did not ‘‘flag impending technical problems and cost overruns in time for project management to take corrective action.’’72

JPL engineers began their own technical review in April, using twenty ex­perienced Ranger and Mariner engineers. Their primary finding was the in­adequacy of HAC’s systems engineering. HAC divided the spacecraft’s tasks into one hundred discrete units, instead of the eight to ten subsystems typical for JPL. JPL found that many groups ‘‘showed a surprising lack of informa­tion or interest’’ about the impact their product had on adjacent products or on the spacecraft as a whole. The spacecraft’s design showed HAC had per­formed few trade-offs between subsystems, leading to a complex design. This led to reliability problems because HAC’s design had more components and critical failure points than necessary.73

The design reviews came too late to fully compensate for three years of in­attention. In April 1964 the first lander ‘‘drop test’’ came to a premature end when the release mechanism failed and the lander crumpled upon ground impact. JPL management responded by increasing JPL’s Surveyor staff from under one hundred in June 1964 to five hundred in the fall of 1965. In the meantime, five independent test equipment and spacecraft failures doomed the second drop test in October 1964.74

HAC partially complied with NASA recommendations — it strengthened

its project organization in August 1964. JPL resisted the headquarters pres­sure to change, but after some pointed letters from Office of Space Sciences and Applications head Homer Newell, and not-so-subtle pressure from Con­gress and Administrator Webb, JPL relented and “projectized” Surveyor and its other programs. Both JPL and HAC added personnel and improved bud­geting, scheduling, and planning tools. JPL created the Project Engineering Division, which assisted flight projects in ‘‘launch vehicle integration, system design and integration, system test and launch operations and environmental requirements.’’75

JPL reassigned engineers from Ranger and Mariner to Surveyor, and it in­stituted an intense program of contractor penetration. While JPL stated that this led to a clearer picture of problem areas and a better relationship be­tween JPL and HAC, HAC engineers and managers frequently viewed it as an exercise in educating JPL personnel, instead of letting them fix the pro­gram’s myriad problems. In 1965, HAC estimated that approximately 250 JPL engineers were at HAC’s facility on any given day. Numerous changes led to months of intense contract negotiations between the two organizations.76

Surveyor’s problems caught Congress’s attention, leading to a House in­vestigation in 1965. The subsequent report identified “inadequate prepara­tion’’ and NASA’s inattention as the primary problems, resulting in ‘‘one of the least orderly and most poorly executed of NASA projects.’’ Congress did not believe further changes to be necessary but severely criticized NASA for its past performance. The investigation concluded, ‘‘NASA’s management per­formance in the Surveyor project must be judged in the light of a history of too little direction and supervision until recently.’’ Nonetheless, five of Surveyor’s seven flights eventually succeeded.77

Ranger and Surveyor were JPL’s trial by fire. Embarrassing failures and cost and schedule overruns plagued early efforts in both programs. They humbled JPL and provided leverage for Congress and NASA headquarters to impose their will on JPL. JPL managers and engineers learned that they needed strong project management, extraordinary attention to design details and manufac­turing, change control, and much better preliminary design work before com­mitting to a project.

When Pickering finally agreed to change JPL and its processes, Jack James’s

Mariner project provided the model. JPL survived its early managerial and technical blunders on Ranger and Surveyor primarily because of its solid suc­cesses on Mariner. Jack James’s strong project organization, backed by pro­gressive design freezes and change control, made Mariner a much different organization than Ranger or Surveyor. Based on the Mariner model, later projects would earn JPL its reputation as the world’s leader in the art of deep space exploration.