Systems Management, Air Force Style

The informal management methods that characterized the manned program’s first few years did not last. New OMSF chief George Mueller understood very well that he would have to implement rigorous cost prediction and control methods to address NASA’s unruly engineers and R&D projects. To do this, he would turn to the management methods with which he was familiar, those developed in the air force and at Thompson-Ramo-Wooldridge (TRW).

Mueller began his career in 1940 working on microwave experiments at Bell Telephone Laboratories. After the war he taught electrical engineering at Ohio State University, and in 1957 he joined Ramo-Wooldridge’s STL as the director of the Electronics Laboratories. He moved up quickly, becoming program manager of the Able space program, vice president of Space Systems Management, and then vice president for R&D. Mueller helped build the air force’s bureaucratic system to control large missile and space projects.60

Before Mueller started his NASA duties, he performed his own investi­gation of OMSF. His first impression of NASA was that ‘‘there wasn’t any management system in existence.’’ The many interface committees and panels worked reasonably well but did not penetrate ‘‘far enough to really be an effective tool in integrating the entire vehicle.” Most seriously, Mueller found no means to determine and control the hardware configuration, leaving no way to determine costs or schedules. Mueller concluded that he had to ‘‘teach people what was involved in doing program control.”61

In August 1963 Mueller invited each of the field center directors to visit him. He explained his proposed changes and how they would help solve NASA’s problems with the Bureau of the Budget and the President’s Science Advi­sory Committee. Mueller explained, ‘‘If we didn’t work together, we were sure going to be hung apart.’’ Mueller had little trouble convincing Apollo space­craft manager Joseph Shea at MSC that his changes were necessary, given Shea’s prior experience at TRW. However, he met resistance from MSFC lead­ers. When MSFC manager Eberhard Rees challenged Mueller’s proposals, Mueller retorted that ‘‘Marshall was going to have to change its whole mode of operation.’’ Two weeks later, at another MSFC meeting, von Braun gave Mueller ‘‘one of his impassioned speeches about how you can’t change the basic organization of Marshall.’’ Refusing to back down, Mueller told him that MSFC’s laboratories ‘‘were going to have to become a support to the pro­gram offices or else’’ they ‘‘weren’t going to get there from here.’’ Von Braun, in response, reorganized MSFC on September 1, 1963, to strengthen project organizations through the creation of the Industrial Operations branch.62

Webb strengthened Mueller’s position by making the directors and proj­ects at MSC, MSFC, and KSC report to OMSF. Mueller reduced attendance at the Manned Space Flight Management Council to himself and the center di­rectors to ensure coordinated responses to OMSF problems. Borrowing from the air force Minuteman program, Mueller also formed the Apollo Execu­tive Group, which consisted of Mueller and Apollo contractor presidents. The group members met periodically at NASA facilities, where Mueller pressured them to resolve problems.63

Facing the same dilemma Holmes faced—Apollo’s slipping schedules and cost overruns—Mueller concluded that the only way to achieve the lunar landing before 1970 within political and budget constraints was to reduce the number of flights. In his ‘‘all-up testing’’ concept, each flight used the full Apollo flight configuration. This approach, used on the Titan II and Minute – man programs, violated von Braun’s conservative engineering principles. Von Braun’s existing plan used a live Saturn first stage with dummy upper stages for the first test. The second test included a live second stage with a dummy third stage, and so on. By contrast, the all-up concept used all flight stages on the very first test. This reduced the number of test flights and eliminated different vehicle configurations, with their attendant differences in designs, ground equipment, and procedures.64

Saturn V program manager Arthur Rudolf cornered Associate Adminis­trator Robert Seamans at a meeting, showing him a model of the Saturn V dwarfing a Minuteman model, saying, ‘‘Now really, Bob!’’ Seamans got the hint — Mueller’s concept did not apply to the more complex Saturn V. En­couraged, Rudolph showed the same models to Mueller. Mueller replied, ‘‘So what?’’ The all-up concept prevailed.65

In November 1963, Mueller reorganized the Gemini and Apollo Program Offices, creating a ‘‘five-box’’ structure at headquarters and the field cen­ters. The new structure (see figure) ensured that the field centers replicated Mueller’s concept of systems management and provided Mueller with better program surveillance. Inside these ‘‘GEM boxes,’’66 managers and engineers communicated directly with their functional counterparts at headquarters and other field centers, bypassing the field centers’ normal chain of com­mand. As one NASA manager put it, ‘‘Anywhere you wanted to go within the organization there was a counterpart whether you knew him or not. Whether you had ever met the man, you knew that if you called that box, he had the same kind of responsibility and you could talk to him and get communication going.’’67

Mueller’s new organization initially wreaked havoc at NASA headquarters, because the change converted NASA engineers who monitored specific hard­ware projects into executive managers responsible for policy, administration, and finance. For several months after the change, headquarters was in turmoil as the staff learned to become executives.68

NASA’s organizational structure changed as a result of Mueller’s initiatives, but he could not always find personnel with the management skills he desired. Shortly after assuming office, Mueller wrote to Webb, stating that NASA could use military personnel trained in program control. Because of the air force’s interest in a ‘‘future military role in space,’’ Mueller believed that it would agree to place key personnel in NASA, where they would acquire experience in space program management and technology. NASA, in turn, would bene-

George Mueller’s ‘‘five box’’ structure replicated the headquarters OMSF organization through the OMSF field centers. Adapted from “Miscellaneous Viewgraphs,’’ circa January-February 1964, ‘‘Organization and Management,” folder LC/SPP-43:11.

fit from their contractual experience and program control methods. Mueller stated, ‘‘It is particularly worth noting that the Air Force, over a period of years, has developed the capability of managing and controlling the very con­tractors upon whom we have placed our primary dependence for the lunar program.’’ He proposed to place Minuteman program director Phillips in the position of program controller for OMSF and contacted AFSC chief Schriever regarding this assignment. Schriever agreed, but only on the condition that Phillips become Apollo program director. From this position, Phillips would transform NASA’s organization and would become known as Apollo’s Rock of Gibraltar.69

Phillips surmised, ‘‘NASA had developed to be a very, very professional technical organization, but they had almost no management capability nor experience in planning and managing large programs.’’70 Phillips turned to the air force for reinforcements and to his most valuable tool from Minute – man, configuration management.

In a January 1964 letter to Schriever, Phillips asked for further air force per-

George Mueller (left) and Samuel Phillips (right) imposed air force management methods on Apollo by introducing new procedures and bringing dozens of air force officers into NASA’s manned space flight programs. Courtesy NASA.

sonnel to man OMSF’s program control positions. After AFSC assigned two officers to NASA, Phillips created a list of fifty-five positions that he wanted to fill with air force officers. Such a large request entailed formal negotiations between NASA and the air force for Phillips’s ‘‘Project 55.’’ Secretary of the Air Force Eugene Zuckert agreed to consider transfers if NASA better defined the position requirements, so that he could ensure that the positions would en­hance the officers’ careers. In September 1964, the joint NASA-air force com­mittee reported that ninety-four air force officers already worked in NASA and that forty-two of the fifty-five additional positions requested should be filled by further air force assignments. The air force reserved the right to select junior and midgrade officers for NASA tours of at least three years. Eventu­ally, Phillips requested and received assignments for 128 more junior officers, who were mainly assigned to Apollo operations in Houston.71

Mueller and Phillips placed officers in key managerial positions through-

out the Apollo and Gemini programs, particularly in project control and con­figuration management offices. Phillips also requested a few senior officers by name. By December 1964, NASA and the air force agreed to assign Phillips as Apollo program director. NASA assigned Brig. Gen. David Jones as deputy associate administrator for Manned Space Flight (Programs), Col. Edmund O’Connor as director of MSFC Industrial Operations, Col. Samuel Yarchin as deputy director of the Saturn V Project Office, and Col. Carroll Bolender as Apollo mission director.72

Phillips recognized that NASA’s engineers would resist his primary control technique, configuration management. To meet this issue head-on, he sched­uled a Configuration Management Workshop in February 1964 in Los Ange­les. Each NASA and contractor organization could invite two or three people, except MSFC, which could invite six. At the conference, Phillips stated, ‘‘Coming out of Minuteman and into Apollo, I think I’ve been identified as a procedures and methods man with a management manual all written that I intend to force on the Apollo program and down the throats of the exist­ing ‘good people’ base. So I ask myself, ‘WHY PROCEDURES AND METH­ODS?’’’ Phillips noted that good processes and methods made good people’s work even better and that they enabled the manager to communicate and control more effectively because they created a ‘‘high percentage of automatic action.’’ Because of Apollo’s rapid growth, the program needed better com­munications between NASA and contractor organizations, and all parties had to use consistent language. Phillips stated that good procedures had a high probability of preventing oversights or shortcuts that could lead to catastro­phe. He noted, ‘‘The outside world is critical, including the contractors, Con­gress and the GAO [General Accounting Office], and the press.’’73 Solid proce­dures would help to protect NASA from criticism, by preventing failures and by documenting problems as they were found.

Phillips explained his system of design reviews and change control, both of which would help managers control resources. He viewed the field cen­ters as Apollo prime contractors and considered configuration management a contractual mechanism to control industry. Phillips proposed strong project management along with a series ofreviews tied to configuration control. Con­sistent with military doctrine, Phillips believed that ‘‘diffused authority and responsibility’’ meant a ‘‘lack of program control,’’ so he assigned responsi-

bility for each task to a single individual and gave that person authority to accomplish the task.74

To aid his initiative, Phillips modified Apollo’s information system. In the existing headquarters program control and information system, data were collected from the field centers each month, and then headquarters managers reviewed the data for problem areas and inconsistencies, at which time they advised the centers of any problems. Instead, Phillips wanted daily analysis of program schedules and quick data exchange to resolve problems, with data placed in a central control room modeled on Minuteman. He immediately placed contracts for a central control room, which eventually contained data links with automated displays to Apollo field centers.75

At MSFC, Saturn managers already had a control room to track official program activities. Saturn V manager Arthur Rudolf assigned names to each chart on the control room wall, so that whenever he found a problem, he could immediately call someone who understood the chart’s details and im­plications. The charts varied over time, as new ones representing current problems and status appeared, replacing charts addressing problems that had been resolved.

This room, although useful, had its problems. Rudolf did not like PERT and reverted instead to ‘‘waterfall’’ charts (Gantt, or ‘‘bar,’’ charts arranged in a waterfall fashion over time). Project control personnel translated PERT charts into Rudolf’s waterfall charts, located in a small room across the hall. Because the control room displayed the ‘‘official’’ status of items, it did not always have the kind of information Rudolf wanted. A typical problem would be that a hardware item might be completed but the paperwork might still be incomplete. The control room information would not be updated until the paperwork was complete, and hence it did not reflect the true status of the hardware. By contrast, the ‘‘mini control room’’ across the hall had ‘‘grease – penciled’’ charts with more up-to-date information. With MSFC personnel calling contractors to get status updates, this mini control room buzzed with activity.76

Phillips devoted great effort to the promotion of configuration manage­ment. His headquarters group worked with the air force to develop the Apollo Configuration Management Manual. Issued in May 1964, this manual copied the air force’s manual, which AFSC officers were updating at the time. He wrote letters to the field center directors emphasizing the manual’s impor­tance and directed them to develop implementation plans. By fall 1964, the field centers were actively creating configuration control boards (CCBs), de­veloping the forms and procedures, and directing contractors to implement configuration management.77

Configuration management required that NASA have firm requirements and specifications for Apollo, which did not yet exist, despite three-year-old contracts between NASA and its contractors. Phillips ordered NASA head­quarters and systems contractor Bellcomm to develop definitive specifica­tions. Not wanting Bellcomm to take over this task, field center and contrac­tor personnel quickly became involved, resulting in firm specifications for all Apollo contracts.78

NAA recognized that with Phillips as Apollo program manager, failure to enhance configuration management ‘‘could well be a serious mistake,’’ so it led a study of the process. Under Phillips’s plan, NASA placed preliminary speci­fications under change control after the Preliminary Design Review, and the hardware under change control after the First Article Configuration Inspec­tion. NAA’s group discovered that after the Critical Design Review, NASA did not elevate the final design specifications to contractual status, which could lead to contractual disagreements over design specifications changes. NAA raised this to NASA’s attention, resulting in further enhancement of configu­ration management.79

Phillips soon encountered the resistance he expected. Some of it was passive. He tried to educate NASA personnel about systems management through air force project management and systems engineering courses at the Air Force Institute of Technology, where he occasionally lectured. Despite his enthusiasm, only a handful of NASA engineers and managers attended the one-week course.80

Other resistance was overt. At the June 1964 Apollo Executives Meeting, Phillips had his headquarters configuration control manager describe con­figuration management to the field center directors and industry executives. After the presentation, the NASA field center directors argued against Phil­lips’s plan.81 MSC Director Gilruth had heard that configuration manage­ment cost one additional person for every manufacturing team member. If this was true, then configuration management would be far too expensive.

Phillips’s team replied that based on Minuteman experience, configuration control required only one person per one hundred manufacturing team mem­bers. Air force configuration managers described how Douglas Aircraft took four months to confirm the S-IV stage configuration prior to testing and de­livery to NASA, whereas on the Gemini Titan II, a program with configuration management, it took two days.82

MSFC Director von Braun objected: ‘‘This whole thing has a tendency of moving the real decisions up, even from the contractor structure viewpoint, from the one guy who sits on the line to someone else.’’ Boeing President Bill Allen replied, ‘‘That’s a fundamental of good management.’’ After all, he said, ‘‘Who, around this table, makes important decisions without getting advice from the fellow who knows?’’ Von Braun retorted, ‘‘The more you take this into the stratosphere and take the decisions away from the working table — I think the object of this whole thing is to remove it from the drawing board.’’ Allen, whose company had originally created configuration management, re­plied that you merely had to move the ‘‘top engineering guy into the position of the Configuration Manager.’’83

Frustrated in this argument, von Braun retorted that because the mili­tary produced a thousand Minutemen, whereas there was only one or just a few Apollos, ‘‘We have to retain a little more flexibility.’’ Again, Boeing’s Allen disagreed: ‘‘Maybe I don’t understand, but in my simple mind, it doesn’t make any difference with respect to what has been outlined here, whether it’s R&D, Saturn, or whether you’re trying to produce a thousand letters.’’ Von Braun replied, ‘‘If you want to roll with the punches, then you have to main­tain a certain flexibility.’’ OMSF chief Mueller intervened; on the Titan III program, the first with configuration management from the start, Mueller noted, ‘‘Everything is on cost and schedule, even though it married solids and liquids.’’84 Mueller concluded, ‘‘Configuration management — doesn’t mean you can’t change it. It doesn’t mean you have to define the final configuration in the first instance before you know that the end item is going to work. That isn’t what it means. It means you define at each stage of the game what you think the design is going to be within your present ability. The difference is after you describe it, you let everybody know what it is when you change it. That’s about all this thing is trying to do.’’85

Mueller, who had the ultimate authority to force implementation, quelled executive objections, but resistance continued in other forms. MSC assigned only one person to configuration management by October 1964, slowing its adoption there and at MSC’s contractors. There was continued engineering resistance to change control, Phillips noted: ‘‘Engineers always know how to do it better once they’ve done it, and want to make their product better.’’ Yet ‘‘even engineers will admit that changes first of all must be justified,’’ he added.86

Contractors had one last chance to charge additional costs to the govern­ment before NASA could control them in detail using configuration man­agement. They used the opportunity, preventing full implementation into late 1965 by charging high rates to implement configuration control systems. NASA auditors found numerous deficiencies, including incomplete engineer­ing release systems, no configuration management of major subcontractors, uncontrolled test requirements and procedures, poor numbering systems, lack of documentation, and, in a few cases, no system at all.87

With continued exhortation and substantial pressure, Phillips established configuration management on Apollo by the end of 1966, with four different levels of change control and authority: contractor, stage and system manager, program manager, and program office. Contractors could authorize changes that did not affect any other contractors or specifications. Stage and system managers could authorize changes within their own systems. Only program manager offices at the field centers could authorize changes affecting inter­faces between stages, systems, or field centers. Phillips in the Program Office authorized changes to the master schedule, hardware quantities, or the over­all program specifications. The full system included six formal reviews, after which NASA approved or modified the specifications, designs, and hard-

ware.88

Because the Apollo program had been under way for three years before Phillips took control, many of the designs never underwent the initial reviews called for in the new system. NASA had awarded Apollo contracts without accurate specifications, revised the design after contract awards, and designed and even tested system components without specifications. Only one element, the Block II command module for the lunar orbit and landing missions, fol-

Phillips’s review processes for Apollo, which he used to ensure the success of the moon landings. Adapted from Robert C. Seamans Jr. and Frederick I. Ordway, ‘‘The Apollo Tradition: An Object Lesson for the Management of Large-Scale Technological Endeav­ors,” in Frank P. Davidson and C. Lawrence Meador, eds., Macro-Engineering and the Future: A Management Perspective (Boulder, Colo.: Westview Press, 1982), 20.

lowed Phillips’s process in its entirety.89 Other vehicle elements began their first design reviews at their state of maturity when Phillips levied his require­ments.

Phillips augmented configuration control with other information sources. He held daily and monthly meetings with program personnel. In turn, Muel­ler and Seamans reviewed Apollo each month, and Webb reviewed it annu­ally. The project developed a computer system to automate failure reports, cost data, and parts information. Apollo’s Reliability and Quality Assurance organization developed into an important management tool, supplying in­formation on reliability, test results, and part defects as well as current plans, status reports, schedules, funding, and manpower. It forwarded quarterly and weekly highlight reports on each major system element.90

The two years following the hiring of George Mueller in September 1963 marked Apollo’s transition from a loosely organized research team to a tightly run development organization. Mueller made important early decisions, in­cluding instituting his GEM box organization formalizing systems engineer­ing, reliability and quality assurance, and project control on the manned pro­grams. Mueller forced von Braun and Gilruth to adhere to his all-up decision, sharply reduced flight tests in favor of ground testing, and gave more respon­sibility to contractors. Finally, he started the importation of air force officers to implement program control, beginning with Minuteman director Phillips.

Phillips brought in air force officers to implement configuration manage­ment. Configuration management required precise knowledge of the system specifications and design, the baseline against which managers and systems engineers judged changes. This, in turn, required a series of design reviews and managerial checkpoints that progressively elevated specifications and de­signs to controlled status. Despite resistance, by 1966 Mueller and Phillips augmented NASA’s processes by firmly establishing air force methods within OMSF. The new management methods could not prevent all technical prob­lems or make up completely for the earlier lack of management control, nor did contractors uniformly enforce them. For most technical programs, the most difficult times involve testing problems that arise. Apollo would be no different.