First Steps in Proving XVS: A View from the Cockpit

From 1995 through 1999, the XVS element conducted a number of simulator and flight tests of novel concepts using NASA Langley’s and NASA Ames’ flight simulators as well as the Calspan-Air Force Research Laboratory’s NC-131H Total In-Flight Simulator (an extensively modi­fied Convair 580 twin-turboprop transport, with side force controllers, lift flaps, computerized flight controls, and an experimental cockpit) and Langley’s ATOPS Boeing 737.[1158]

In 1995, the first formal test, TSRV.1, was conducted in Langley’s fixed-base Transport Systems Research Vehicle (TSRV) simulator, which replicated the Research Flight Deck (RFD) in Langley’s ATOPS B-737. Under the direction of Principal Investigator Randall Harris, the test was a parametric evaluation of different sensor and HUD presentations of a proposed XVS. A monitor was installed over the copilot’s glare shield to provide simulated video, forward-looking infrared (FLIR), and computer-generated imagery (CGI) for the evaluation. The author had the privilege of undertaking this test, and the following is from the report he submitted after its conclusion:

Approach, flare, and touchdown using 1 of 4 available sensors (2 were FLIR sensors with a simulated selection of the "best” for the ambient conditions) and 1 of 3 HUD presentations making a 3 X 3 test matrix for each scheduled hour long ses­sion. Varying the runways and direction of base to final turns resulted in a total matrix of 81 runs. Each of the 3 pilots com­pleted 63 of the 81 possible runs in the allotted time.[1159]

Commenting on the differences between the leader aircraft flight director and the more traditional HUD/Velocity Vector centered flight director, the author continued:

Some experimentation was performed to best adapt the amount of lead of the leader aircraft. It was initially agreed that a 25 to 15 sec lead worked best for the TSRV simulator.

The 5 sec lead led to a too high gain task for the lateral axis control system and resulted in chasing the leader continu­ously in a roll PIO state. Adjusting the amount of lead for the leader may need to be revisited in the airplane. A purely per­sonal opinion is that the leader aircraft concept is a higher workload arrangement than a HUD mounted velocity vector centered flight director properly tuned.[1160]

At the same time, a team led by Russ Parrish was developing its own fixed-based simulator intended to support HSR XVS research and devel­opment. Known as Virtual Imaging Simulator for Transport Aircraft Systems (VISTAS), this simulator allowed rapid plug-and-play evaluation of various XVS concepts and became a valuable tool for XVS research­ers and pilots. Over the next 5 years, this simulator evolved through a series of improvements, leading to the definitive VISTAS III configura­tion. Driven by personal computers rather than the Langley simulation facility mainframe computers, and not subject to as stringent review pro­cesses (because of its fixed-base, low-cost concept), this facility became extremely useful and highly productive for rapid prototyping.[1161]

From the ground-based TSRV. 1 test, XVS took to the air with the next experiment, HSR XVS FL.2, in 1996. Using Langley’s venerable ATOPS B-737, FL.2 built upon lessons learned from TSRV.1. FL.2 dem­onstrated for the first time that a pilot could land a transport aircraft using only XVS imagery, with the Langley research pilots flying the air­craft with no external windows in the Research Flight Deck. As well, they landed using only synthetically generated displays, foreshadow­ing future SVS work. Two candidate SVS display concepts were evalu­ated for the first time: elevation-based generic (EBG) and photorealistic. EBG relied on a detailed database to construct a synthetic image of the

terrain and obstacles. Photorealistic, on the other hand, relied on high – resolution aerial photographs and a detailed database to fuse an image with near-high-resolution photographic quality. These test points were in anticipation of achieving sensor fusion for the HSCT flight deck XVS displays, in which external sensor signals (television, FLIR, etc.) would be seamlessly blended in real time with synthetically derived displays to accommodate surmised varying lighting and visibility conditions. This sensor fusion technology was not achieved during the HSR program, but it would emerge from an unlikely source by the end of the decade.

The second flight test of XVS concepts, known as HSR XVS FL.3, was flown in Langley’s ATOPS B-737 in April 1997 and is illustrative of the challenges in perfecting a usable XVS. Several experiments were accomplished during this flight test, including investigating the effects of nonconformality of the artificial horizon portrayed on the XVS for­ward display and the real-world, out-the-side-window horizon as well as any effects of parallax when viewing the XVS display with a close design eye point rather than viewing the real-world forward scene focused at infinity. Both the Research and Forward Flight Decks (FFD) of the B-737 were highly modified for this test, which was conducted at NASA Wallops Flight Facility (WFF) on Virginia’s Eastern Shore, just south of the Maryland border. Located on the Atlantic coast of the Delmarva

Peninsula, Wallops was situated within restricted airspace and immedi­ately adjoining thousands of square miles of Eastern Seaboard warning areas. The airport was entirely a NASA test and rocket launch facility, complete with sophisticated radar – and laser-tracking capability, control rooms, and high-bandwidth telemetry receivers. Langley flight opera­tions conducted the majority of their test work at Wallops. Every XVS flight test would use WFF.

The modifications to the FFD were summarized in the author’s research notes as follows:

The aircraft was configured in one of the standard HSR XVS FL.3 configurations including a 2X2 tiled Elmo lipstick cam­era array, a Kodak (Megaplus) high resolution monochrome video camera (1028 x 1028 pixels) mounted below the nose with the tiled camera array, an ASK high resolution (1280 x 1024 pixels) color video projector mounted obliquely behind the co-pilot seat, a Silicon Graphics 4D-440VGXT Skywriter Graphics Workstation, and a custom Honeywell video mixer.

The projector image was focused on a 24 inch by 12 inch white screen mounted 17.5 inches forward of the right cockpit seat Design Eye Position (DEP). Ashtech Differential GPS receiv­ers were mounted on both the 737 and a Beechcraft B-200 tar­get aircraft producing real time differential GPS positioning information for precise inter-aircraft navigation.42

An interesting digression here involves the use of Differential GPS (DGPS) for this experiment. NASA Langley had been a leader in devel­oping Differential GPS technologies in the early 1990s, and the ATOPS B-737 had accomplished the first landing by a transport aircraft using Differential GPS guidance. Plane-plane Differential GPS had been per­fected by Langley researchers in prior years and was instrumental in this and subsequent XVS experiments involving traffic detection using video displays in the flight deck. DGPS could provide real-time relative positions of participating aircraft to centimeter accuracy.

With the conformality and parallax investigations as a background, Langley’s Beechcraft B-200 King Air research support aircraft was
employed for image object detection as a leader aircraft on multiple instrument approaches and as a random traffic target aircraft. FL.3 iden­tified the issue about which a number of XVS researchers and pilots had been concerned about at the XVS Workshop the previous fall: the challenges of seeing a target aircraft in a display. Issues such as pixel per degree resolution, clutter, brightness, sunlight readability, and con­trast were revealed in FL.3. From the flight-test report:

Unfortunately, the resolution and clarity of the video presen­tation did not allow the evaluation pilot to be able to see the leader aircraft for most of the time. Only if the 737 was flown above the B-200, and it was flying with a dark background behind it, was it readily visible in the display. We closed to 0.6 miles in trail and still had limited success. On final, for exam­ple, the B-200 was only rarely discernible against the runway environment background. The several times that I was able to acquire the target aircraft, the transition from forward display to the side window as I tracked the target was seamless. Most of the time the target was lost behind the horizon line or velocity vector of the display symbology or was not visible due to poor contrast against the horizon. Indeed, even with a bright back­ground with sharp cloud boundaries, the video presentation did not readily distinguish between cloud and sky. . . . Interestingly, the landings are easier this time due, in my opinion, to a per­ceived wider field of view due to the geometry of the arrange­ment in the Forward Flight Deck (FFD) and to the peripheral benefits of the side window. Also, center of percussion effects may have caused false motion cues in the RFD to the extent that it may have affected the landings. The fact that the pilot is quite comfortable in being very confident of his position due to the presence of the side window may have had an effect in reducing the overall mental workload. The conformality differ­ences were not noticeable at 4 degrees, and at 8 degrees, though noticeable and somewhat limiting, successful landings were possible. By adjusting eye height position the pilot could effec­tively null the 0 and 4 degree differences.[1162]

Another test pilot on this experiment, Dr. R. Michael Norman, dis­cussed the effects of rain and insects on the XVS sensors and displays. His words also illustrate the great risks taken by the modern test pilot in the pursuit of knowledge:

Aerodynamics of the flat, forward facing surface of the camera mount enclosure resulted in static positioning of water droplets which became deposited on the aperture face. The relative size of the individual droplets was large and obtrusive, and once they were visible, they generally stayed in place. Just prior to touch­down, a large droplet became visually superimposed with the velocity vector and runway position, which made lineup cor­rections and positional situational awareness extremely diffi­cult. Discussions of schemes to prevent aperture environmental contamination should continue, and consideration of incorpo­ration in future flight tests should be made. During one of the runs, a small flying insect appeared in the cockpit. The shadow of this insect amplified its apparent size on the screen, and was somewhat distracting. Shortly thereafter, it landed on the screen, and continued to be distracting. The presence of flying insects in the cockpit is an issue with front projected displays.44

Clearly, important strides toward a windowless flight deck had been achieved by FL.3, but new challenges had arisen as well. Recognizing the coupling between flight control law development and advanced flight displays, the GFC and Flight Deck ITD Teams planned a joint test in 1998 on a different platform, the Air Force Research Laboratory- Calspan NC-131H Total In-Flight Simulator aircraft.

TIFS, which was retired to the Air Force Museum a decade after­ward, was an exotic-looking, extensively modified Convair 580 twin – engine turboprop transport that Calspan had converted into an in-flight simulator, which it operated for the Air Force. Unique among such sim­ulators, the TIFS aircraft had a simulation flight deck extending in front of and below the normal flight deck. Additionally, it incorporated two large side force controllers on each wing for simulation fidelity, mod­ified flaps to permit direct lift control, and a main cabin with comput­ers and consoles to allow operators and researchers to program models of different existing or proposed aircraft for simulation. TIFS operated on the model following concept, in which the state vector of TIFS was sampled at a high rate and compared with a model of a simulated air­craft. If TIFS was at a different state than the model, the flight con­trol computers on TIFS corrected the TIFS state vector through thrust, flight controls, direct lift control, and side force control to null all the six degree-of-freedom errors. The Simulation Flight Deck (SFD) design was robust and allowed rapid modification to proposed design specifications.

Undertaken from November 1998 through February 1999, the FL.4 HSR experiment combined XVS and GFC experimental objectives. The SFD was configured with a large cathode ray tube mounted on top of the research pilot’s glare shield, simulating a notional HSR virtual forward window. Head-down PFD and NAV display completed the simulated HSR flight deck. XVS tests for FL.4 included image object detection and dis­play symbology evaluation. The generic HSR control law was used for the XVS evaluation. A generic XVS symbology suite was used for the GFC experiments flown out of Langley and Wallops. Langley research­ers Lou Glaab and Lynda Kramer led the effort, with assistance from the author (who served as Langley HSR project pilot), Calspan test pilot Paul Deppe (among others), and Boeing test pilot Dr. Michael Norman (who was assigned to NASA Langley as a Boeing interface for HSR).

The success of FL.4, combined with some important lessons learned, prepared the way for the final and most sophisticated of the HSR flight tests: FL.5, flown at Langley, Wallops, and Asheville, NC,

from September through November 1999. Reprising their FL.4 efforts, Langley’s Lou Glaab and Lynda Kramer led FL.5, with valuable assis­tance from Calspan’s Randall E. Bailey, who would soon join the Langley SVS team as a NASA researcher. Russell Parrish also was an indispens­able presence in this and subsequent SVS tests. His imprint was felt throughout the period of focused SVS research at NASA.

With the winding down of the HSR program in 1999, the phase of SVS research tied directly to the needs of a future High-Speed Civil Transport came to an end. But before the lights were turned out on HSR, FL.5 provided an apt denouement and fitting climax to a major program that had achieved much. FL.4 had again demonstrated the difficulty in image object detection using monitors or projected displays. Engineers surmised that a resolution of 60 pixels per degree would be necessary for acceptable performance. The requirement for XVS to be capable of providing traffic separation in VMC was proving onerous. For FL.5, a new screen was used in TIFS. This was another rear projection device, providing a 50-degree vertical by 40-degree horizontal field of view (FOV). Adequate FOV parameters had been and would continue to be a topic of study. A narrow FOV (30 degrees or less), while providing good
resolution, lacked accommodation for acceptable situation awareness. As FOVs became wider, however, distortion was inevitable, and resolu­tion became an issue. The FL.5 XVS display, in addition to its impres­sive FOV, incorporated a unique feature: a high-resolution (60 pixels per degree) inset in the center of the display, calibrated appropriately along an axis to provide the necessary resolution for the flare task and traffic detection. The XVS team pressed on with various preparatory check­outs and tests before finally moving on to a terrain avoidance and traf­fic detection test with TIFS at Asheville, NC.

Asheville was selected because of the terrain challenges it offered and the high-fidelity digital terrain database of the terminal area pro­vided by the United States Geological Survey. These high-resolution ter­minal area databases are more common now, but in 1999, they were just becoming available. This database allowed the TIFS XVS to pro­vide high-quality head-down PFD SVS information. This foreshadowed the direction Langley would take with FL.5, when XVS gave way to SVS displays incorporating the newer databases. In his FL.5 research notes of the time, the author reflected on the XVS installation, which was by then quite sophisticated:

The Primary XVS Display (PXD) consisted of three tiled pro­jections, an upper, a lower, and a high resolution inset display.

The seams between each projection were noticeable, but were not objectionable. The high resolution inset was designed to approach a resolution of about 60 pixels per degree in the ver­tical axis and somewhat less than that in the horizontal axis.

It is my understanding that this degree of resolution was not actually achieved. The difference in resolution between the high resolution inset and the surround views was not objec­tionable and did not detract from the utility of any of the dis­plays. Symbology was overwritten on all the PXD displays, but at times there was not a perfect match between the sur­rounds and the high resolution inset resulting in some dupli­cated symbology or some occulted symbology. An inboard field-of-view display (IFOV) was also available to the pilot with about the same resolution of the surround views. This also had symbology available.

The symbology consisted of the down selected HSR min­imal symbology set and target symbology for the PXD and a

horizon line, heading marks, and target symbology for the IFOV display. The target symbology consisted of a blue dia­mond with accompanying digital information (distance, alti­tude and altitude trend) placed in the relative position on the PXD or IFOV display that the target would actually be located. Unfortunately, due to several unavoidable transport delays, the target symbology lagged the actual target, especially in high track crossing angle situations. For steady relative bearing situ­ations, the symbology worked well in tracking the target accu­rately. Occasionally, the target symbology would obscure the target, but a well conceived PXD declutter feature corrected this.

The head down displays available to the pilot included a fairly standard electronic Primary Flight Display (PFD) and Navigation Display (ND). The ND was very useful to the pilot from a strategic perspective in providing situation awareness (SA) for target planning. The PXD provided more of a tactical SA for traffic avoidance. TCAS, Radar, Image Object Detection (IOD), and simulated ADSB targets were displayed and could be brought up to the PXD or IFOV display through a touch screen feature. This implementation was good, but at times was just a little difficult to use. Variable ranges from 4 to 80 miles were pilot selectable through the touch screen. In the past sunlight intrusion in the cockpit had adversely affected both the head up and head down displays. The addition of shaded window liners helped to correct this problem, and sun shafting occur­rences washing out the displays were not frequent.45

The accompanying figure shows the arrangement of XVS displays in the SFD of the TIFS aircraft for the FL.5 experiment.

The author’s flight-test report concluded:

Based on XVS experience to date, it is my opinion that the current state of the art for PXD technologies is insufficient to justify a "windowless” cockpit. Improvements in contrast, res­olution, and fields of view for the PXD are required before this concept can be implemented. . . . A visual means of verifying

the accuracy of the navigation and guidance information pre­sented to the pilot in an XVS configured cockpit seems man­datory. That being said, the use of symbology on the PXD and Nav Display for target acquisition provides the pilot with a sig­nificant increase in both tactical and strategic situation aware­ness. These technologies show huge potentials for use both in the current subsonic fleet as well as for a future HSCT.[1163]

Though falling short of fully achieving the "windowless cockpit” goal by program’s end, the progress made over the previous 4 years on HSR XVS anticipated the future direction of NASA’s SVS research. Much had been accomplished, and NASA had an experienced, motivated team of researchers ready to advance the state of the art as the 20th century closed, stimulated by visions of fleetwide application of Synthetic and Enhanced Vision Systems to subsonic commercial and general-aviation
aircraft and the need for database integrity monitoring. Meanwhile, a continent away, other NASA researchers, unaware of the achieve­ments of HSR XVS, struggled to develop their own XVS and solved the challenge of sensor fusion along the way.