Category The Secret of Apollo

Technical failures of aerospace projects are hard to hide. Rockets and missiles explode. Satellites stop sending signals back to Earth. Pilots and astronauts die. To the extent that systems management helped prevent these events, it must be deemed a technical success. Systems management methods such as quality assurance, configuration control, and systems integration testing were among the primary factors in the improved dependability of ballistic missiles and spacecraft. Missile reliability in air force and JPL missile programs in­creased from the 50% range up to the 80% to 95% range, where it remains to this day. JPL’s spacecraft programs suffered numerous failures from 1958 to 1963, but after that JPL’s record dramatically improved, with a nearly perfect record of success for the next three decades. The manned programs suffered a number of testing failures at the start but had an enviable flight record with astronauts, with the one glaring exception of the Apollo 204 fire. A strong cor­relation exists between systems management and reliability improvements.

The nature of reliability argues for the positive influence of systems meth­ods. For aerospace projects to succeed, there must be high-quality compo­nents, proper integration of these components, and designed-in backups in case failures occur. Only the last of these is a technology issue in the design sense. The selection and proper integration of components has more to do with rigorous compliance with design and manufacturing standards than it does with new technology. High component quality comes through unflag­ging attention to manufacturing processes, backed by testing and selection of the best parts. In a nutshell, it is easy to solder a joint or crimp a connec­tor pin but extremely difficult to ensure that workers perform thousands or millions of solders and crimps correctly. Even a worker with the best skills and motivation will make occasional mistakes. In systems management, so­cial processes to rigorously inspect and verify all manufacturing operations ensure high quality across the thousands of workers involved in the process.

Similarly, ensuring proper integration is a matter of making sure that each and every joint is properly soldered, every pin and connector properly crimped, every structure properly handled at all times, and all of these opera­tions rigorously tested. On top of this, ‘‘systems testing’’ checks for design flaws and unexpected interactions among components. In all of these issues, procedures and processes—not new technology—are the keys to success. Sys­tems management provided these rigorous processes and tests.

Once organizations dealt with component problems, they ran into the next most likely cause of failure: interface problems caused by mismatches between designs. By the mid-1960s, both the air force and NASA obsessively concen­trated on interface problems, which resulted ultimately from poor communi­cation, poor organization, or both. Engineers and managers recognized that differences in organizational cultures and methods made communication be­tween organizations more difficult than communication within an organiza­tion. Miscommunication led to incompatibilities between components and subsystems — incompatibilities often found when components were first con­nected and tested. More technology was not the solution. Instead, engineers needed improved communication through social processes.

Engineers enforced better communication by creating standard documents and processes. They required that one organization be responsible for ana­lyzing both sides of an interface and that the specifications and analyses be documented in a formal Interface Control Document. Many interface prob­lems were subtler than simple mismatches between physical or electrical com­ponents.

For example, engineers at Marshall Space Flight Center found that a ‘‘non­liftoff’’ of a Mercury-Redstone test vehicle occurred because the Mercury cap­sule had a different weight than the Redstone’s normal warhead, changing the time it took for the launch vehicle to separate from the launch tower. Because the combined launch complex-launch vehicle electronics required that the ve­hicle lift off at a certain rate, the changed rate led to a shutdown of the launch vehicle as emergency electronics kicked in to abort the launch.

Problems such as these were solvable not through technology but through better engineering communication and better design analysis. Once engineers understood all of the factors, the design solution was usually simple. The problem was making sure the right people had the right information and that someone had responsibility for investigating the entire situation. As ELDO’s history shows, getting an organization to pay for a change in an interface was often more difficult than formulating a technical solution. Authority and communication matter most in interface problems and solutions. Better orga­nization and better systems, not better technology, made for reliability in large aerospace projects by standardizing the processes and providing pro­cedures to cross-check and verify each item, from solder joints to astronaut flight procedures. These methods essentially provided insurance for technical success.

How much did this insurance cost? Did systems management lower costs or speed development compared to earlier processes and methods? Concurrency in the 1950s was widely believed to shorten development times, but at enor­mous cost. The secretary of the air force admitted that the air force could af­ford only one or two such programs. Schriever contended that concurrency saved money because it shortened development time. Because R&D costs are spent mostly on engineering labor, Schriever argued, shortening development time would reduce labor hours and hence cost. Most other experts then and later disagreed with him. Political scientist Michael Brown contends that con­currency actually led to further schedule slips because problems in one part of the system led to redesigns of other parts, often several times over.3

On any given design, having systems management undoubtedly costs more than not having systems management, just as buying insurance costs more than not buying insurance. The real question is whether systems management reduced the number of failures sufficiently, so that it counterbalanced the re­placement cost. For example, a 50% rate of reliability for a missile system such as Atlas in the late 1950s meant that every other missile failed. With this failure rate, the air force and its contractors could afford to spend up to the cost of an entire second missile in improvements to management processes, ifthese pro­cesses could guarantee success. In other words, at a 50% reliability rate and a cost of $10 million per missile, each successful launch costs $20 million. Thus, if process improvements can guarantee success, then spending $10 million or less per missile in management process improvements is cost-effective.

In fact, the early Atlas, Titan, and Corporal projects achieved roughly 40­60% reliability. Reliability improvement programs — that is, systems manage­ment processes — improved reliability into the 60-80% range during the 1950s and early 1960s and into the 85-95% range thereafter.4 The reliability im­provement meant that roughly nine out of ten launches succeeded, instead of one out of two. Therefore, systems management could easily have added more than 50% to each missile’s cost and still been cost-effective. NASA’s efforts to ‘‘man-rate’’ Atlas and Titan could have added 100% to costs for Atlas and Titan and still been cost-effective, because success had to be guaranteed. In fact, considering the potential loss of not only the launchers but also the manned capsules and astronauts, NASA could likely spend 200-500% on launcher im­provements and still be cost-effective, considering the low reliability of these vehicles at that time. Pending detailed cost analysis, systems management was probably cost-effective if costs were measured for each successful launch.

Another way to assess systems management is to compare missile and space programs that implemented systems management methods with programs that did not. ELDO provides the most extreme example of little or no systems management. None of its rockets ever succeeded, despite piecemeal intro­duction of some systems management methods. Comparison of JPL’s Ranger program with the contemporary Mariner program provides another example, because the Mariner design was a modification of the Ranger spacecraft. With less systems management, Ranger’s first six flights failed, whereas Mariner achieved a 50% success rate, with later Mariner spacecraft performing al­most perfectly. After strengthening systems management, Ranger’s record was three successes out of four launches.5 Assuming Ranger and Mariner costs per spacecraft were roughly equal, Mariner cost less per successful flight than early Ranger.

Aside from pure cost considerations, failures hurt an organization’s credi­bility. In the rush to beat the Soviets, early space programs lived the old adage ‘‘There is never time to do it right, but there is always time to do it over.’’ They had many failures, but in the early days executive managers were not terribly concerned. By the early 1960s, however, failures mattered; they led to con­gressional investigations and ruined careers. Systems management responded to the need for better reliability by trying to make sure that engineers ‘‘did it right’’ so they would not have to ‘‘do it over.’’

It is no coincidence that engineers developed systems management for mis­siles and spacecraft that generally cannot be recovered. When each flight test and each failure means the irretrievable loss of the entire vehicle, thorough planning is much more cost-effective than it is for other technologies that can be tested and returned to the designers. This helps to explain why the bu­reaucratic methods of systems management work well for space systems but seem too expensive for many Earth-bound technologies. For most technolo­gies, building a few prototypes and performing detailed tests with them be­fore manufacturing is feasible and sensible. Lack of coordination and plan­ning (each of which costs a great deal) can be overcome through prototype testing and redesign of the prototype. This option is not available for most space systems, because they never return.

The evidence suggests that systems management was successful in improv­ing reliability sufficiently to cover the cost per successful vehicle. Although systems management methods were not the only factor involved in these im­provements, from the standpoint of reliability, they were critical. Process im­provements, not technology improvements, ensured the proper connection and integration of thousands of components. Systems management increased vehicle costs on a per vehicle basis compared to previous methods but re­duced costs when reliability is factored in.

In technical terms, NASA’s organization and procedures proved successful on Mercury and Gemini. Both programs boasted enviable flight records, com­pleting their objectives and successfully returning their astronauts on every flight. On the other hand, both programs featured large cost increases. Mc­Donnell’s Mercury capsule contract grew at an astounding rate, from an initial contract of $19,450,000 in 1959 to a total of $143,413,000 by the program’s end in 1963. Within months of its start, Gemini was headed for large cost increases as well. NASA funded the increases, although Congress began to ask questions in the fall of 1962.52

OMSF’s Holmes, overseeing the three manned space flight programs, saw the escalating cost trends as well as anyone. A hard-nosed project manager from Radio Corporation of America’s (RCA’s) Ballistic Missile Early Warn­ing System, Holmes realized that something had to be done if Apollo was to meet Kennedy’s 1969 deadline to land a man on the Moon. Holmes also rec­ognized that Kennedy had staked his reputation on the manned program and would go to great lengths to ensure its success. Based on this, he felt no par­ticular need to control costs and instead requested more funds from NASA Administrator Webb.53

Holmes first asked Webb to go back to Congress for a supplemental appro­priation. Despite Kennedy’s support, Webb recognized that Congress was be­coming uneasy about NASA’s burgeoning costs, and he refused. Holmes next demanded that Webb strip other NASA programs to support the manned pro­gram. When Webb again refused, Holmes went over his head, appealing di­rectly to President Kennedy. This infuriated Webb, now placed in the uncom­fortable position of justifying why he should not strip other NASA projects to fund Kennedy’s priority program. Although Kennedy was not sure that he ‘‘saw eye-to-eye’’ with Webb on this issue, he backed his chosen administrator. Webb replaced his insubordinate OMSF director with STL executive George Mueller (pronounced “Miller’’).54

Under the circumstances, further cost pressures were unwelcome. In March 1963, Gemini project manager James Chamberlin admitted that the project required another huge infusion of money. The unwelcome news led to his replacement by Charles Mathews. Mathews immediately set up a commit­tee to improve cost estimation at NASA and McDonnell. After another year of strained budgets and organizational crises — invariably related to novel portions of the project — Gemini’s budget stabilized, leading to an enviable record of flight success. However, cost estimates had risen from $531,000,000 to $1,354,000,000 between 1962 and 1964.55

With Congress increasingly concerned with costs, Webb instigated an in­ternal investigation of NASA projects in early 1964.56 Deputy Associate Ad­ministrator Earl Hilburn chaired the investigation to study NASA’s project scheduling and cost estimating methods. In confidential reports submitted in September and December of 1964, Hilburn described NASA’s abysmal record on cost and schedule control. Engine development showed by far the largest schedule slips, at 4.75 times the original estimate. Hilburn found significant slips in launch vehicles requiring new engines (3.09), manned spacecraft (2.86), ‘‘simple’’ scientific satellites (2.85), and astronomical observatories (2.83). Classified by field center, he found that the Jet Propulsion Laboratory (JPL) had the smallest schedule slips (1.6) and MSC the largest (3.06). The newest centers, MSC and Goddard, showed the largest slips, and the older centers (JPL, Langley) showed the smallest. Costs varied similarly.57

Hilburn noted that the DOD experience was also ‘‘one of over-runs and schedule slippages in the majority of cases.’’ He concluded that these results were behind recent changes in DOD procurement, ‘‘including program defi­nition, and incentive contracting.’’ Based on DOD precedents, NASA had al­ready begun converting contracts from cost-plus-fixed-fee contracts to incen­tive contracts that awarded firms a higher fee when they performed well and a lower fee when they did not. The Hilburn Report lent support to contract con­version at NASA, which NASA implemented from 1964 through 1966.58 NASA also adopted the DOD practice of phased planning, requiring contractors to better predict costs and schedules during a definition phase of development. The directive had little initial impact because of disagreements about imple­mentation, and more importantly, because few new projects started in 1966 and 1967.59

NASA’s manned space programs were among the worst offenders as NASA’s budget exploded between 1962 and 1964. Having grown accustomed to gener­ous, even extravagant budget increases without justification or congressional concern, NASA’s continued cost excesses were understandable, if not justifi­able. As in other organizations, the time came when its initial growth surge and justification had to slow and managers had to predict and hold to fund­ing profiles. At the highest levels, NASA could replace recalcitrant executives and implement new standards and processes, but to truly predict and control costs, NASA would have to implement rigorous procedures on its research and development (R&D) programs.

Missiles, particularly ballistic missiles,38 disrupted the air force’s culture, oper­ations, and organization in several important ways. First, and most obviously, missiles had no pilots, relegating humans to only pushing a button. Second, maintenance and long-term operations of missiles amounted to storage and occasional refurbishment, as opposed to the ongoing repairs typical for air­craft. Third, because missiles were used just once, missile testing required the creation of a missile production line. Unlike aircraft, where a few prototypes could be built and tested with dozens or hundreds of flights each, every mis­sile test required a new missile. This implied that the fly-before-you-buy con­cept, where aircraft could be tested before instigation of full-scale production, no longer applied. For missiles, testing required a production line. Finally, missiles involved a variety of challenging new technical issues, as described in the previous chapter. Simply put, many of the air force’s existing organiza­tional and technical processes did not work for missiles.

Ballistic missile programs languished at a low priority during and after World War II, as the air force concentrated its efforts first on manned bomb­ers, and then on jet fighters for the Korean War.39 The rapidly escalating Cold War provided the impetus to transform the loosely organized missile projects. Successful testing ofthe Soviet atomic bomb in 1949 spurred the United States to develop a fusion weapon. In March 1953, Assistant for Development Plan­ning Bernard Schriever learned of the success of American thermonuclear tests from the SAB. Recognizing the implications of this news, within days Schriever met renowned mathematician John von Neumann at his Princeton office. Von Neumann predicted that scientists would soon develop nuclear warheads ofsmall enough size and large enough explosive power to be placed on ICBMs. Because of their speed and in-flight invulnerability, ICBMs were the preferred method for nuclear weapons delivery, ifthe air force could make them work. Realizing that he needed official backing, Schriever talked with James Doolittle, who approached Chief of Staff Vandenberg to have the SAB investigate the question.40

The Nuclear Weapons Panel of the SAB, headed by von Neumann, reported to the air force staff in October 1953. In the meantime, Trevor Gardner, assis­tant to the secretary of the air force, volunteered to head a Department of De­fense (DOD) Study Group on Guided Missiles. Gardner learned of Convair’s progress on its Atlas ICBM and met with Dr. Simon Ramo, an old friend and head of Hughes Aircraft Company’s successful air-to-air missile project, the Falcon. Based on the results of his study group, Gardner and Air Force Sec­retary Talbott formed the Strategic Missiles Evaluation Committee, or Teapot Committee, to recommend a course of action for strategic ballistic missiles.41

Von Neumann headed the group, and Gardner selected Ramo’s newly cre­ated Ramo-Wooldridge Corporation (R-W) to do the paperwork and man­age the day-to-day operations of the study. Ramo had partnered with fellow Hughes manager Dean Wooldridge to form R-W.42 In February 1954 the Tea­pot Committee recommended that ICBMs be developed ‘‘to the maximum extent that technology would allow.’’ It also recommended the creation of an organization that hearkened back to the Manhattan Project and Radiation Laboratory of World War II: ‘‘The nature of the task for this new agency re­quires that over-all technical direction be in the hands of an unusually com­petent group of scientists and engineers capable of making systems analyses, supervising the research phases, and completely controlling the experimental and hardware phases of the program — the present ones as well as the subse­quent ones that will have to be initiated.’’43

On May 14, 1954, the air force made Convair’s Atlas its highest R&D pri­ority. Because Convair and the majority of the aircraft industry hailed from

Southern California, the air force established its new ICBM development or­ganization, the Western Development Division (WDD), in a vacant church building in Inglewood, near Los Angeles airport. Air force leaders placed newly promoted Maj. General Bernard Schriever in command on August 2, 1954. Because the Teapot Committee had recommended creation of a ‘‘Man­hattan-like” project organization, one of Schriever’s first tasks was to see if this made sense and determine who would oversee the technical aspects ofthe project.44

Schriever rejected the Manhattan Project organization because ICBMs were significantly more complicated than the atomic bomb.45 Because neither he nor the scientists believed that the air force had the technical expertise to manage the program, Schriever could hire Convair as prime contractor, or he could hire R-W as the system integrator, with Convair and other contrac­tors as associate contractors. The air force used the prime contractor proce­dure on most programs, but this assumed that the prime contractor had the wherewithal to design and build the product. Schriever was already unhappy with Convair because he believed Convair kept “in-house” elements such as guidance and electronics in which it had little experience, to the program’s detriment.46

Scientists with whom he had worked for nearly a decade also deeply influ­enced Schriever. Von Neumann and his fellow scientists believed the Soviet threat required a response like the Manhattan Project a decade earlier, bring­ing together the nation’s best scientists to marry ballistic missiles to thermo­nuclear warheads. Schriever later explained:

Complex requirements of the ICBM and the predominant role of systems engineering in insuring that the requirements were met, demanded an across – the-board competence in the physical sciences not to be found in existing orga­nizations. Scientists rated the aircraft industry relatively weak in this phase of engineering, which was closely tied to recent advances in physics. The aircraft industry, moreover, was heavily committed on major projects, as shown by existing backlogs. Its ability to hire the necessary scientific and engineering tal­ent at existing pay-scales was doubted, and with the profit motive dominant, scientists would not be particularly attracted to the low-level positions accorded to such personnel in industry.47

Organization of the Inglewood complex: the Western Development Division, the Special Aircraft Projects Office, and Ramo-Wooldridge.

Many years later Schriever described his admiration of the scientists: ‘‘I be­came really a disciple of the scientists who were working with us in the Penta­gon, the RAND Corporation also, so that I felt very strongly that the scientists had a broader view and had more capabilities. We needed engineers, that’s for sure, but engineers were trained more in a, let’s say a narrow track having to do with materials than with vision.’’48

To capitalize on the vision and expertise of physical scientists and mathe­maticians such as von Neumann and von Karman, Schriever created an or­ganizational scheme whereby the leading scientists could guide the ICBM program. Following the von Neumann committee recommendations, Schrie­ver selected R-W for systems engineering and integration.49 Free of civil ser­vice regulations, R-W could hire the requisite scientific and technical tal­ent. The air force could more easily direct R-W than Convair, because R-W had few contracts and no production capability. The aircraft industry dis­puted this unusual arrangement, fearing that it established a precedent for ‘‘strong system management control’’ by the air force and also that it might create a powerful new competitor with inside information about air force con­tracts and contractor capabilities. On both counts, the aircraft industry was correct.50

Selecting the best and brightest technical officers from ARDC, Schriever’s talented staff quickly took charge of ICBM development. Because AMC re­tained procurement authority, it set up a field office known as the Special Air­craft Projects Office (SAPO) alongside Schriever’s ARDC staff in Inglewood. By September 1954, air force headquarters approved Schriever’s selection of R-W, confirming the triumvirate of the WDD, the SAPO, and R-W. Schriever’s next battle would be to establish the authority and credibility of his team in the face of skepticism at air force headquarters and the outright hostility of the aircraft industry.51

Firms have shown themselves anxious to collaborate with ESRO as a means of gaining useful experience of the newer management techniques which are indispensable for the effec­tive control of the financial as well as the technical aspects of large and complex projects.

— J. J. Beattie and J. de la Cruz, 1967

The European Space Research Organisation (ESRO) presented a welcome contrast to the ongoing embarrassments of the European Space Vehicle Launcher Development Organisation (ELDO). Created as a service organiza­tion for European space scientists, ESRO overcame its initial organizational difficulties and developed a successful series of scientific satellites. Its achieve­ments proved that effective European space cooperation was possible. Al­though ELDO had been the Europeans’ prime organization to develop space technology, its failure paved the way for ESRO to become the route of choice across the management gap between the United States and Europe.

ESRO’s success owed a great deal to its greater contractual authority (com­pared to ELDO) and to American assistance. While European industry and powerful interest groups focused on the military and economic significance of launchers, ESRO’s scientific satellites seemed insignificant. ESRO’s Conven­tion and procedures consequently had fewer provisions to protect national economic interests than did ELDO, giving ESRO authority that ELDO never had. In addition, whereas Americans did not want to aid Europeans in rock­etry or communications satellites, they cheerfully gave technical and financial assistance to European science.

These factors help explain ESRO’s rise from a small service organization

to the core of Europe’s integrated space organization. Through the authority of its Convention, the ability of its engineering and managerial staff, and the help of the United States, ESRO and its descendant, the European Space Agency (ESA), mastered the art of systems management.1