Augmented Reality Use Cases in Aerospace

As the “Industrial Internet” and The Internet of Things (IOT) go beyond the buzz-words of the board room and technologies start being deployed within industry, aerospace manufacturers and industrial leaders have started to implement some of the latest technology to make their manufacturing processes more efficient, reduce errors on the shop floor, deliver products to market more rapidly and to finally leverage that big data within their own organization. Augmented Reality (AR) and Virtual Reality (VR) are both mediums at which the industries are looking at to make work instructions easier and training more adaptive, provide green talent with contemporary tools and improve the production process. Augmented and Virtual Reality experiences have been created for the entertainment and gaming industries and for various marketing and brand awareness campaigns, however both have still a way to go to influence the mass market. So, with early adoption and various use cases testing the productivity gains of industry, how has Augmented Reality impacted the aerospace shop floor and what technologies are necessary for its adoption to accelerate?

Industry giants like Lockheed Martin, Boeing and Applied Materials have invested in the development of both AR hardware and software solutions and have side projects and secretive labs dabbling with head-mounted wearables, light projections, telepresence or remote assist and SDKs to improve costs, time of task and error reductions on the shop-floor. The good news is that AR hardware and software are improving at a tremendous rate with the help of these use cases and investments. The capability of smart-glasses, smartphones and tablets are making astonishing strides in computational, graphical and sensory power with clearer, higher resolution displays and improvements to battery life enabling easier ways to consume and create AR and VR content. Some limiting factors that developers are dealing with is the amount of available memory on portable devices to store digital objects and content. Network latency also requires careful consideration when system architecture dictates whether this content should be stored on the device or a remote content server.

Aerospace manufacturers have adopted many form factors of AR on the factory floor. One of the first aerospace applications of augmented reality is the Heads-Up Display (HUD) used by fighter pilots; it provided a digital overlay on an artificial horizon with a variety of useful information like airspeed, engine gauges and power usage. Augmented Reality allows the user to see the real-world with virtual objects superimposed upon or composited with the real world. Today’s mobile devices have all the necessary components to enable location-based augmented reality packed into one device: a camera, a screen, GPS capabilities, accelerometers, gyroscopic sensors and even a compass.  However, let’s take a look at all the form factors in play throughout the aerospace community.

What are aerospace’s commercial intentions with Augmented Reality?
As mentioned earlier, some of the technical trends of IoT, the expanding video game market, as well as a variety of support technologies will improve and enhance AR systems and the overall experience. IoT will have a wide-ranging influence in the areas of information analysis, automation and control for all aerospace companies. It will be useful for information and analysis functions that:

1. Track Behavior & Productivity
2. Create Enhanced Situational Awareness
3. Provide Sensor Driven Information

And, IoT will allow for automation and control by allowing:

1. Process Optimization
2. Complete Component Health Monitoring
3. Proven Manufacturing Efficiencies

From these lists, Augmented Reality has, and will continue to play the largest role in further enabling situational awareness and productivity within the aerospace enterprise. Data from large number of sensors, deployed in heightened awareness of real-life events, particularly when the sensors are coupled with AR display technologies. As the Internet of Things and AR continue to develop, it’s becoming more probable that AR will become one of the predominant interfaces for IoT in much the same way that the graphical user interface (GUI) became the interface for the Internet.
 
Aerospace Use Cases
There are many use cases for Augmented Reality in the aerospace industry as it has had a long history with the technology:
 
Company: Boeing
Form Factor: iPad AR
 
In 2008, Paul Davies, a research and development engineer at Boeing began working with Boeing Technical Fellow, Anthony Majoros. Up until today, they’ve used commercially-available technologies such as Total immersion’s D’Fusion platform to demonstrate how technicians repairing and building wing assemblies could perform complex tasks with AR running on tablets. The AR group was 30% faster and 90% more accurate on their first try. By, their second try, errors had been reduced to virtually zero. Technicians who had never been on the wing before had become experts. Davies, as a result of his findings, explained that there are building blocks of AR, two of which still need some work:

• Tracking – understanding the 3D pose of a camera as well as the position of real objects in 3D space with a high degree of accuracy
• Core Function – a software component that ties in information and functions like CAS models, calibration data, user interface and rendering
• Visualization – How you present that augmented view to the user

Tracking and visualization are the two blocks that Davies feels haven’t been solved. For visualization, the issue is often that you have to hold something and that the ergonomics of making AR devices wearable and safe within the dangerous environments where we want to use them. As for Tracking, the issue is using external infrared systems to help locate AR devices in space. IMUs have drift issues, ultrasonic, and magnetic resonance, so which every technology you choose will require some kind of sensing infrastructure to be installed.
(Also in Nov 2014, APX Labs announced that its Skylight software platform for smart glasses had been selected by Boeing to evaluate wearable technologies to improve manufacturing efficiencies in manual assembly tasks.)
 
Company: Lockheed Martin
Form Factors: Smart Glasses (Epson Moverio)
Software: NGRAIN
 
Until recently, Lockheed Martin needed a team of technicians with years of training to wrench on sophisticated aircraft like the F-35. Now, on the advanced factory floor, engineers are using augmented reality glasses and educational software that provides real-time visuals during an aircraft’s complex assembly.  Lockheed Martin is collaborating with NGRAIN on a trail in which employees are wearing the $700 EPSON Moverio BT-transparent glasses with front-facing cameras using motion and depth sensors to assist while installing brake components for example. Divisions within Airbus have also been experimenting with AR for over a decade. They were studying how Augmented Reality could assist with assembly and service tasks but judged too immature to implement into production environments for reasons like:

1. Difficult Tracking Conditions Inside an Aircraft
2. Unsolved Cabling Issues
3. Maturity of Head-Mounted Displays
4. Different types of AR hardware usage (HMD, Mobile Devices, Projection)
5. Expensive Hardware, Software and Support

The AR glasses are still in trial mode on the Lockheed factory floor. Getting this technology onto military bases, where Lockheed technicians handle repairs to keep the planes battle-ready, will be slightly more difficult. The military doesn't allow cameras on those bases, so the front-facing optical sensor is currently verboten. However, the current project proved that their engineers could work 30% faster and with up to 96% accuracy. Those numbers are huge for a company like Lockheed Martin. If even one F-35 fighter jet needs additional repairs and can’t be sold or deployed, that costs the manufacturer big piles of money.
 
 
Company: Lufthansa Technik
Form Factors: Mobile Projection
 
Lufthansa Technik has now introduced a laser-based AR system for supporting installation in the VIP & Special Mission Aircraft Services business unit as part of the “Projection-Based Installation Support” project. The mobile projection system can be positioned and aligned flexibly in the aircraft fuselage in line with the respective installation situation. The installation template is projected onto the work environment for the relevant structural employee in line with the employee’s current job card. This template corresponds to the component contours selected in the virtual 3D model, which are projected onto the structure in the correct position and therefore act as a positioning and alignment aid for the component that is being installed. With no interruptions or other interactions, the employee can install the component with high precision and significantly more efficiently than using conventional alignment tools.
The technical implementation of the laser projection system was demonstrated in the first completed project phase both on a mock-up of the training workshop and in practice on three complete layovers – a result of the excellent inter-disciplinary cooperation between engineering, dock and IT. Not only was the precision significantly increased when installing primary structural connections (T-Rails), but the level of work required was reduced by almost 50 percent. Lufthansa Technik is currently testing additional applications of projection-based Augmented Reality, for example in the form of foiling and painting templates. The use of future digital assistance systems for other departments is also being examined at present for use in a larger-scale innovation project. The preparations for the project are already underway. 

---