Category The Secret of Apollo

Missiles were developed from simple rocketry experimentation between World Wars I and II. Experimenters such as Robert Goddard and Frank Ma – lina in the United States, von Braun in Germany, Robert Esnault-Pelterie in France, and Valentin Glushko in the Soviet Union found rocketry experimen­tation a dangerous business. All of them had their share of spectacular mis­haps and explosions before achieving occasional success.5

The most obvious reason for the difficulty of rocketry was the extreme volatility of the fluid or solid propellants. Aside from the dangers of handling exotic and explosive materials such as liquid oxygen and hydrogen, alcohols, and kerosenes, the combustion of these materials had to be powerful and controlled. This meant that engineers had to channel the explosive power so that the heat and force neither burst nor melted the combustion chamber or nozzle. Rocket engineers learned to cool the walls ofthe combustion chamber and nozzle by maintaining a flow of the volatile liquids near the chamber and nozzle walls to carry off excess heat. They also enforced strict cleanliness in manufacturing, because impurities or particles could and did lodge in valves and pumps, with catastrophic results. Enforcement of rigid cleanliness stan­dards and methods was one of many social solutions to the technical problems of rocketry.6

Engineers controlled the explosive force of the combustion through care­fully designed liquid feed systems to smoothly deliver fuel. Instabilities in the fuel flow caused irregularities in the combustion, which often careened out of control, leading to explosions. Hydrodynamic instability could also ensue if the geometry of the combustion chamber or nozzle was inappropriate. Engi­neers learned through experimentation the proper sizes, shapes, and relation­ships of the nozzle throat, nozzle taper, and combustion chamber geometry. Because of the nonlinearity of hydrodynamic interactions, which implied that mathematical analyses were of little help, experimentation rather than theory determined the problems and solutions. For the Saturn rocket engines, von Braun’s engineers went so far as to explode small bombs in the rocket ex­haust to create hydrodynamic instabilities, to make sure that the engine de­sign could recover from them.7 For solid fuels, the shape of the solid deter­mined the shape of the combustion chamber. Years of experimentation at JPL eventually led to a star configuration for solid fuels that provided steady fuel combustion and a clear path for exiting hot gases. Once engineers determined the proper engine geometry, rigid control of manufacturing became utterly critical. The smallest imperfection could and did lead to catastrophic failure. Again, social control in the form of inspections and testing was essential to ensuring manufacturing quality.

Rocket engines create severe structural vibrations. Aircraft designers rec­ognized that propellers caused severe vibrations, but only at specific frequen­cies related to the propeller rotation rate. Jet engines posed similar prob­lems, but at higher frequencies corresponding to the more rapid rotation of turbojet rotors. Rocket engines were much more problematic because their vibrations were large and occurred at a wide range of nearly random frequen­cies. The loss of fuel also changed a rocket’s resonant frequencies, at which the structure bent most readily. This caused breakage of structural joints and the mechanical connections of electrical equipment, making it difficult to fly sensitive electrical equipment such as vacuum tubes, radio receivers, and guidance systems. Vibrations also occurred because of fuel sloshing in the emptying tanks and fuel lines. These ‘‘pogo’’ problems could be tested only in flight.

Vibration problems could not generally be solved through isolated tech­nical fixes. Because vibration affected electrical equipment and mechanical connections throughout the entire vehicle, this problem often became one of the first so-called system issues — it transcended the realm of the structural engineer, the propulsion expert, or the electrical engineer alone. In the 1950s, vibration problems led to the development of the new discipline of reliability and to the enhancement of the older discipline of quality assurance, both of which crossed the traditional boundaries between engineering disciplines.8

Reliability and quality control required the creation or enhancement of so­cial and technical methods. First, engineers placed stronger emphasis on the selection and testing of electronic components. Parts to be used in missiles had to pass more stringent tests than those used elsewhere, including vibra­tion tests using the new vibration, or ‘‘shake,’’ tables. Second, technicians as­sembled and fastened electronic and mechanical components to electronic boards and other components using rigorous soldering and fastening meth­ods. This required specialized training and certification of manufacturing workers. Third, to ensure that manufacturing personnel followed these pro­cedures, quality assurance personnel witnessed and documented all manufac­turing actions. Military authorities gave quality assurance personnel indepen­dent reporting and communication channels to avoid possible pressures from contractors or government officials. Fourth, all components used in missiles and spacecraft had to be qualified for the space environment through a series of vibration, vacuum, and thermal tests. The quality of the materials used in flight components, and the processes used to create them, had to be tightly controlled as well. This entailed extensive documentation and verification of materials as well as of processes used by the component manufacturers. Orga­nizations traced every part from manufacturing through flight.9

Only when engineers solved the vibration and environmental problems could they be certain the rocket’s electronic equipment would send the signals necessary to determine how it was performing. Unlike aircraft, rockets were automated. Although automatic machinery had grown in importance since the eighteenth century, rockets took automation to another level. Pilots could fly aircraft because the dynamics of an aircraft moving through the air were slow enough that pilots could react sufficiently fast to correct deviations from the desired path and orientation of the aircraft. The same does not hold true for rockets. Combustion instabilities inside rocket engines occur in tens of milliseconds, and explosions within 100 to 500 milliseconds thereafter, leaving no time for pilot reaction. In addition, early rockets had far too little thrust to carry something as heavy as a human.

Because rockets and satellites were fully automated, and also because they went on a one-way trip, determining if a rocket worked correctly was (and is) problematic. Engineers developed sophisticated signaling equipment to send performance data to the ground. Assuming that this telemetry equip­ment survived the launch and vibration of the rocket, it sent sensor data to a ground receiving station that recorded it for later analysis. Collecting and processing these data was one of the first applications of analog and digital computing. Engineers used the data to determine if subsystems worked cor­rectly, or more importantly, to determine what went wrong if they did not. The military’s system for problem reporting depended upon pilots, but con­tractors and engineers would handle problem reporting for the new technolo­gies — a significant social change. Whereas in the former system, the military tested and flew aircraft prototypes, for the new technologies contractors flew prototypes coming off an assembly line of missiles and the military merely witnessed the tests.10

Extensive use of radio signals caused more problems. Engineers used radio signals to send telemetry to ground stations and to send guidance and de- struct signals from ground stations to rockets. They carefully designed the electronics and wiring so that electromagnetic waves from one wire did not interfere with other wires or radio signals. As engineers integrated numerous electronic packages, the interference of these signals occasionally caused fail­ures. The analysis of ‘‘electromagnetic interference’’ became another systems specialty.11

Automation also included the advanced planning and programming of rocket operations known as sequencing. Rocket and satellite engineers de­veloped automatic electrical or mechanical means to open and close propul­sion valves as well as fire pyrotechnics to separate stages, release the vehicle from the ground equipment, and otherwise change rocket functions. These ‘‘sequencers’’ were usually specially designed mechanical or electromechani­cal devices, but they soon became candidates for the application of digital computers. A surprising number of rocket and satellite failures resulted from improper sequencing or sequencer failures. For example, rocket stage sepa­ration required precise synchronization of the electrical signals that fired the pyrotechnic charges with the signals that governed the fuel valves and pumps controlling propellant flow. Because engineers sometimes used engine turbo­pumps to generate electrical power, failure to synchronize the signals for sepa­ration and engine firing could lead to a loss of sequencer electrical power. This in turn could lead to a collision between the lower and upper stages, to an engine explosion or failure to ignite, or to no separation. The solution to se­quencing problems involved close communication among a variety of design and operations groups to ensure that the intricate sequence of mechanical and electrical operations took place in the proper order.12

Because satellites traveled into space by riding on rockets, they shared some of the same problems as rockets, as well as having a few unique features. Satellites had to survive launch vehicle vibrations, so satellite designers ap­plied strict selection and inspection ofcomponents, rigorous soldering meth­ods, and extensive testing. Because of the great distances involved-particu – larly for planetary probes-satellites required very high performance radio equipment for telemetry and for commands sent from the ground.13

Thermal control posed unique problems for spacecraft, in part because of the temperature extremes in space, and in part because heat is difficult to dissipate in a vacuum. On Earth, designers explicitly or implicitly use air currents to cool hot components. Without air, spacecraft thermal design re­quired conduction of heat through metals to large surfaces where the heat could radiate into space. Engineers soon designed large vacuum chambers to test thermal designs, which became another systems specialty.

Unlike the space thermal environment, which could be reproduced in a vacuum chamber, weightlessness could not be simulated by Earth-based equipment. The primary effect of zero gravity was to force strict standards of cleanliness in spacecraft manufacturing. On Earth, dust, fluids, and other contaminants eventually settle to the bottom of the spacecraft or into corners where air currents slow. In space, fluids and particles float freely and can dam­age electrical components. Early spacecraft did not usually have this problem because many of them were spin stabilized, meaning that engineers designed them to spin like a gyroscope to hold a fixed orientation. The spin caused particles to adhere to the outside wall of the interior of the spacecraft, just as they would on the ground where the spacecraft would have been spin tested.

Later spacecraft like JPL’s Ranger series used three-axis stabilization whereby the spacecraft did not spin. These spacecraft, which used small rocket engines known as thrusters to hold a fixed orientation, were the first to en­counter problems with floating debris. For example, the most likely cause of the Ranger 3 failure was a floating metal particle that shorted out two adjacent wires. To protect against such events, engineers developed conformal coating to insulate exposed pins and connectors. Designers also separated electrically hot pins and wires so that floating particles could not connect them. Engi­neers also reduced the number of particles by developing clean rooms where technicians assembled and tested spacecraft.

Many problems occurred when engineers or technicians integrated com­ponents or subsystems, so engineers came to pay particular attention to these interconnections, which they called interfaces. Interfaces are the boundaries between components, whether mechanical, electrical, human, or “logical,” as in the case of connections between software components. Problems between components at interfaces are often trivial, such as mismatched connectors or differing electrical impedance, resistance, or voltages. Mismatches between humans and machines are sometimes obvious, such as a door too high for a human to reach, or an emergency latch that takes too long to operate. Others are subtle, such as a display that has too many data or a console with distract­ing lights. Finally, operational sequences are interfaces of a sort. Machines can be (and often are) so complicated to operate that they are effectively unusable. Spacecraft, whether manned or unmanned, are complex machines that can be operated only by people with extensive training or by the engineers who built them. Greater complexity increases the potential for operator error. It is probably more accurate to classify operator errors as errors in design of the human-machine interface.14

Many technical failures can be attributed to interface problems. Simple problems are as likely to occur as complex ones. The first time the Ger­mans and Italians connected their portions of the Europa rocket, the diame­ters of the connecting rings did not match. Between the British first stage and the French second stage, electrical sequencing at separation caused com­plex interactions between the electrical systems on each stage, leading ulti­mately to failure. Other interface problems were subtle. Such was the failure of Ranger 6 as it neared the Moon, ultimately traced to flash combustion of propellant outside of the first stage of the launch vehicle, which shorted out some poorly encased electrical pins on a connector between the launch ve­hicle and the ground equipment. Because the electrical circuits connected the spacecraft to the offending stage, this interface design flaw led to a spacecraft failure three days later.15

Some farsighted managers and engineers recognized that interfaces repre­sented the connection not simply between hardware but also between indi­viduals and organizations. Differences in organizational cultures, national characteristics, and social groups became critical when these groups had to work together to produce an integrated product. As the number of organiza­tions grew, so too did the problems of communication. Project managers and engineers struggled to develop better communication methods.

As might be expected, international projects had the most difficult prob­lems with interfaces. The most severe example was ELDO’s Europa I and Europa II projects. With different countries developing each of three stages, a test vehicle, and the ground and telemetry equipment, ELDO had to deal with seven national governments, military and civilian organizations, and national jealousies on all sides. Within one year after its official inception, both ELDO and the national governments realized that something had to be done about the ‘‘interface problem.’’ An Industrial Integrating Group formed for the pur­pose could not overcome the inherent communication problems, and every one of ELDO’s flights that involved multiple stages failed. All but one failed because of interface difficulties.16

By the early 1960s, systems engineers developed interface control docu­ments to record and define interfaces between components. On the manned space projects, special committees with members from each contributing or­ganization worked out interfaces between the spacecraft, the rocket stages, the launch complex, and mission operations. After the fledgling European Space Research Organisation began to work with American engineers and managers from Goddard Space Flight Center, the first letter from the American project manager to his European counterpart was a request to immediately begin work on the interface between the European spacecraft and the American launch vehicle.17

Systems management became the standard for missile and space systems because it addressed many of the major technical issues of rockets and space­craft. The complexity of these systems meant that coordination and commu­nication required greater emphasis in missile and space systems than they did in many other contemporary technologies. Proper communication helped to create better designs. However, these still had to be translated into techni­cal artifacts, inspected and documented through rigid quality inspections and testing during manufacturing. Finally, the integrated system had to be tested on the ground and, if possible, in flight as well. The high cost and “nonreturn” of each missile and spacecraft meant that virtually every possible means of ground verification paid off, helping to avoid costly and difficult-to-analyze flight failures. All in all, the extremes of the space environment, automation, and the volatility of rocket fuels led to new social methods that emphasized considerable up-front planning, documentation, inspections, and testing. To be implemented properly, these social solutions had to satisfy the needs of the social groups that would have to implement them.

Pride in accomplishment is not a self-sufficient safeguard when
undertaking large scale projects of international significance.

— Kelley Board, after Ranger 5 failure

The Jet Propulsion Laboratory (JPL), located in Pasadena, California, and managed by the California Institute of Technology, began as a graduate stu­dent rocket project in the late 1930s and developed into the world’s leading institution for planetary space flight. Between 1949 and 1960, JPL transformed itself twice: first, from a small research organization to a large engineering de­velopment institution, and second, from an organization devoted to military rocketry to one focusing on scientific spacecraft.1

JPL’s academic researchers did not initially recognize the many differences between a hand-crafted research vehicle and a mass-produced, easily oper­ated weapon, or highly reliable planetary probes. The switch from research to development required strict attention to thousands of details. Properly build­ing and integrating thousands of components was not an academic problem but an organizational issue. JPL’s engineering researchers learned to become design engineers, and in so doing some of them became systems engineers.

Learning systems engineering on tactical ballistic missiles, JPL managers and engineers modified missile practices to design and operate spacecraft. The most significant missile practices that carried over to spacecraft were organizational: component testing and reliability as well as procedures for change control. A few JPL managers learned these lessons quickly. However, it

took a number of embarrassing failures for JPL’s academically oriented engi­neers and managers to accept the structured methods of systems manage­ment.

JPL independently recreated processes that the air force developed on its ballistic missile programs: systems engineering, project management, and configuration control. The history of the two organizations shows that the processes were the result of not individual idiosyncrasies but larger technical and social forces.

The failure of F11 in November 1971 brought home to the member states — and this was indeed the only positive point it achieved — the necessity for a complete overhaul of the pro­gramme management methods.

— General Robert Aubiniere, 1974

World War II left Europe devastated and exhausted, while the United States emerged as the world’s most powerful nation, both militarily and economi­cally. Western Europeans feared the Soviet Union’s military power and totali­tarian government, but they worried almost as much about America’s im­mense economic strength. Some asserted that American dominance flowed from the large size of American domestic markets or the competitive nature of American capitalism, while others believed that technological expertise was the primary force creating ‘‘gaps’’ between the United States and Europe. By the late 1960s, the “technology gap’’ was a hot topic for politicians and econo­mists on both sides of the Atlantic.

Investigations showed that European technology and expertise did not radically differ from that of the United States. However, a number of studies showed that Americans managed and marketed technologies more efficiently and rapidly than Europeans. Significant differences between the United States and Western Europe existed in the availability of college-level management education and in the percentage of research and development expenditures. In each of these areas, Americans invested more, in both absolute and per capita terms. Some analysts believed the technology gap to be illusory but a management gap to be real.

To close the gaps, Europeans, actively aided by the United States, took a number of measures to increase the size of their markets, to develop ad­vanced science and technology, and to improve European management. The Common Market was the best-known example of market integration. Science and technology initiatives included the Conseil Europeen pour Recherche Nu – cleaire (CERN [European Committee for Nuclear Research]) for high-energy physics research, EURATOM for nuclear power technologies and resources, the European Space Research Organisation (ESRO) to develop scientific satel­lites, and the European Space Vehicle Launcher Development Organisation (ELDO) to create a European space launch vehicle.

Because of the military and economic significance of space launchers, the national governments of the ‘‘big four’’ Western European states — the United Kingdom, France, West Germany, and Italy—all supported the European launcher effort. Seeking contracts, the European aircraft industry also actively promoted the venture. Paradoxically, these strong national interests rendered ELDO ineffective. Each country and company sought its own economic ad­vantages through ELDO, while withholding as much information as possible. This attitude led to a weak organization that ultimately failed. When the Euro­peans decided to start again in the early 1970s, ELDO’s failure was the spur to do better, a prime example of how not to organize technology development.

Systems management became the standard for space and missile technolo­gies because it promoted the goals of the groups involved. Scientists received credit for conceiving novel technologies. Military officers gained control over radical new weapons and their associated organizations. Engineers earned re­spect by creating reliable technologies. Managers gained by controlling orga­nizations within a predictable (but hopefully large) budget.

In the 1950s scientists formed an alliance with the military to rapidly cre­ate novel weapons. Officers desired quick weapon deployment, while scien­tists provided the novelty. Within systems management, the role of scientists became standardized. They were to perform systems analyses at the begin­ning of programs to determine technological feasibility and whether to de­velop a particular mission or weapon. Scientists also used their quantitative skills to assess the reliability of the new technologies as engineers developed them, acting as credible second sources of information. Managers and mili­tary officers frequently used scientists in this capacity, as Ramo-Wooldridge and Aerospace did for the air force against the contractors.

Military officers also gained influence through systems management. Here it is important to distinguish between technical officers and operational offi­cers. The latter led troops into battle and throughout the long history of the military held the reins of power. Technical officers became more important in the 1950s and 1960s. Before the 1950s, air force technical officers lamented their poor career possibilities. With the creation of Air Research and Develop­ment Command (ARDC) in 1950 and Air Force Systems Command (AFSC) in 1961, technical officers won for themselves career paths separate from those of their operational brethren. Systems management became the formal set of procedures that allowed them to maintain a military career in technical R&D. Technical officers gained a stable career path and significant power.

In the late 1950s and early 1960s, NASA was run by engineers and for engi­neers. Although engineers no longer had free rein after that time, NASA re­mained an engineering organization. So too were ARDC, AFSC, and their laboratories and contractors. Aerospace engineering, and particularly the space program, were the glamour jobs of engineering during this period, with interesting tasks, substantial authority and funding, and excellent opportuni­ties for promotion into managerial or technical positions. Systems manage­ment ensured a large role for engineers throughout the design process and kept alive the engineering working groups that allowed engineers to main­tain substantial authority. Engineers developed the testing techniques that ensured that their products operated, and hence they guaranteed their own credibility and success. These testing techniques too became part of systems management. Engineers benefited from the creation of systems engineering, which gave the chief systems engineer nearly as much authority as the project manager, who more often than not was also an engineer by training and ex­perience.

Management credibility and authority stem from control of a large orga­nization with funding to match. The power of the purse was the manager’s primary weapon, but in the 1950s managers had not yet learned how to use it to control the scientists and engineers. As long as scientists and engineers cre­ated novel technologies and funding was plentiful, they could and did claim that they could not predict costs or schedules. Until the technologies reached testing, where failures appeared, managers could not successfully challenge that claim. However, technical failure gave managers the wedge they needed to gain control.

As tasks and projects repeated, managers used past history to predict costs and hold scientists and engineers to estimates based on prior history. Stan­dardizing systems management made costs and schedules more predictable and allowed managers to distinguish between ‘‘normal’’ cost and schedule patterns and ‘‘abnormal’’ patterns that signaled technical or organizational problems. Project managers used this information to control projects and pre­dict outcomes without being completely dependent on technical experts.

Executive managers wanted to know about the current status of projects and about possible new projects. To determine if new projects should be funded, executive managers created ‘‘breakpoints’’ at which they could inter­vene to continue, modify, or cancel a project. Phased planning implemented these breakpoints and ensured that only limited resources would go toward new projects before executive managers had their say.

From a social viewpoint, each of the four professional groups gained im­portant career niches from the institution of systems management. This helps to explain why it has proven to be a stable method in the aerospace indus­try. However, this leaves unanswered whether systems management actually made costs predictable or novel products dependable. In the end, none of the social factors would matter if the end products ultimately failed.