Difference between revisions of "Adaptive Cockpit"

From MIT Technology Roadmapping
Jump to navigation Jump to search
(66 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<html>
    <script src="https://fred-wang.github.io/TeXZilla/TeXZilla-min.js"></script>
    <script src="https://fred-wang.github.io/TeXZilla/TeXZilla-parse.js"></script>
    <script src="https://fred-wang.github.io/TeXZilla/examples/customElement.js"></script>
</html>
Adaptive cockpit is a level 3 roadmap (3ADC) created by [mailto:bertucci@mit.edu Dave Bertucci] and [mailto:tcarey4@mit.edu Tyler Carey] with the support and mentorship of Embraer, specifically Clenilson Goncalves, Giuliano Mendonca, and Julio Bolzani.
= Roadmap Overview =
= Roadmap Overview =


FOCUS: Flight Operational Companion for Unexpected Situations
An adaptive cockpit is an advanced, AI-driven system designed to enhance aircraft safety and pilot performance during unexpected or high-stress situations. This technology integrates various sensors and intelligent algorithms to monitor both the aircraft’s status and the pilot’s physiological state in real time.
Key features of an adaptive cockpit include:
 
# Continuous monitoring of aircraft systems and flight parameters
# Real-time analysis of pilot biometrics and behavior
# Detection of sudden, potentially disorienting events or anomalies
# Intelligent assessment of the pilot’s cognitive and emotional state
# Provision of timely, context-aware assistance and guidance


FOCUS is an advanced, AI-driven assistant designed to help commercial pilots manage sudden and surprising events within the cockpit. In high-stress situations, pilots may experience natural “freeze” reactions, delayed responses, or unintended inputs, which can compromise safety and increase risk. FOCUS has been designed to address this challenge by monitoring for signs of such startling events in aircraft readings and pilot biometrics. After detection, FOCUS will provide real-time guidance to help pilots quickly regain situational control. Through intuitive, timely interventions, FOCUS enhances flight safety by enabling pilots to confidently navigate complex and unexpected scenarios. This technology roadmap outlines the developmental path for FOCUS, highlighting its role in advancing safety, reducing operational risks, and supporting pilots with reliable, real-time assistance.
When the system detects signs of pilot stress, confusion, or delayed responses, it intervenes with appropriate support. This may include a simplified display of critical information, biofeedback to reduce stress, or automated assistance with certain tasks.
The adaptive cockpit aims to mitigate the impact of human factors such as the startle effect and cognitive overload. By providing tailored, real-time support, pilots can more quickly regain situational awareness and make informed decisions during critical moments.
This technology represents a significant advancement in aviation safety, offering a proactive approach to managing complex scenarios and reducing operational risks. As it continues to evolve, the adaptive cockpit has the potential to revolutionize cockpit design and pilot training, ultimately leading to safer and more efficient flight operations.


[[File:AVIAIFOCUS2.png | 800px | Picture of FOCUS]]
[[File:AVIAIFOCUS2.png | 800px | Picture of FOCUS]]
Line 16: Line 33:


= Roadmap OPD =
= Roadmap OPD =
In the figure below, we provide an Object-Process-Diagram (OPD) of the 3ADC roadmap. This diagram captures the main object of the roadmap (FOCUS), its main process (Teaming), the various tools required, the sub-processes required for FOCUS, and the change in the status of the pilot as a result of Teaming. Additionally, we have included a breakdown of the Temaing process into its three sub-processes (Detecting, Supporting, and Augmenting).  
In the figure below, we provide an Object-Process-Diagram (OPD) of the 3ADC roadmap. This diagram captures the main object of the roadmap (Adaptive Cockpit), its main process (Teaming), the various tools required, the sub-processes required for Teaming, and the change in the status of the pilot as a result of Teaming. Additionally, we have included a zoomed-in look into the Temaing process and its four sub-processes (Detecting, Modeling, Supporting, and Augmenting).  
 
[[File:AVIAIOPD2.png | x800px | OPD]]


[[File:AVIAIOPD3.png | left | x800px | OPD]]


[[File:AVIAIOPL.png | left | x600px | OPD]]
The Object Process Language (OPL) below is the auto-generated grammatical version of the OPD above. The first OPL is for the Level 1 OPD, whereas the second OPL is for the decomposition of Teaming into its four sub-processes. 


[[File:AVIAIOPL2.png | x600px | Level 1 OPL]]


<div style="clear: both"></div>
 
 
[[File:AVIAIOPL3.png | x600px | Level 2 OPD]]


= Figures of Merit =
= Figures of Merit =
Many figures of merit were considered for this roadmap and are outlined below with more information and current trends on each. The first 3 were found to be especially important for the roadmap and are shown below in '''bold'''.


{| class="wikitable"
{| class="wikitable"
|+ Important FOMs for 3AISC
|+ Important FOMs for 3ADC
|-
|-
! Figure of Merit !! Description !! Trends !! Units  
! Figure of Merit !! Description !! Trends !! Units  
|-
|-
| Incident Rate  
| '''Pilot Efficiency'''
|| Measure of how many passengers each pilot is capable of transporting per year
|| increasing
||
<html>
  <math>
    <mfrac>
  <mrow> <mtext>passengers</mtext> </mrow>
  <mtext>pilot * year</mtext>
    </mfrac>
  </math>
</html>
|-
| '''Accident Rate'''
|| Rate of incidents and accidents normalized by flight time aircraft
|| Rate of incidents and accidents normalized by flight time aircraft
|| decreasing
|| decreasing
Line 39: Line 76:
   <math>
   <math>
     <mfrac>
     <mfrac>
  <mrow> <mtext>incidents</mtext> </mrow>
  <mrow> <mtext>accidents</mtext> </mrow>
  <mtext>passenger mile</mtext>
  <mtext>passenger mile</mtext>
     </mfrac>
     </mfrac>
  </math>
</html>
|-
| '''Biometric Activity Accuracy'''
||
Accuracy of best models of pilot pilot metrics to predict attention and startle response
|| increasing
||
<html>
  <math>
    <mtext>%</mtext>
   </math>
   </math>
</html>
</html>
Line 88: Line 136:
</html>
</html>
|-
|-
|}
== Pilot Efficiency ==
Pilots are currently a critical part of air transportation systems and making sure they are efficiently transporting passengers is an important figure of merit (FOM) of an airline. The definition of pilot efficiency is below.
::<html>
<la-tex>
{Efficiency}_{Pilot} = \frac{{Capacity}_{Pass} * {LF}_{Pass}}{{Pilot}_{Plane}} * {Flight}_{Year}
</la-tex>
</html>
<!--
[[File:3AISC-pilot-efficiency-calculation.png | 800px | Pilot Efficiency Equation]]
-->
[[File:3AISC-pilot-efficiency-detail.png | 800px | Pilot Efficiency Variable Detail]]
[[File:3AISC-pilot-efficiency-sensitivities.png | 800px | Pilot Efficiency Tornado Chart]]
== Accident Rate ==
=== Detail of Accident Rates (per million flight cycle), 1959 to 2022 ===
Large commercial aircraft have steadily been getting safer. The table below shows accident rates of aircraft organized by the date of their first flights (oldest first). This chart highlights that newer designs have proven to be safer than older designs. Additionally the chart categorizes aircraft by the generation of automation used in each.
; Generation 1 (Not shown) : Analog gauges and early auto-pilot systems
; Generation 2 : Integration of auto-pilot and auto-throttle systems
; Generation 3 : Glass cockpits, improved navigation systems, and terrain avoidance systems
; Generation 4 : Fly-by-wire and envelope protection systems
As these successive generations of technology have been integrated into aircraft the accident rates have dropped dramatically. Since safety record is a property of the system as a whole and the complexity of an aircraft is so high, generating an analytical model of the contributing factors is very complex, but by comparison to existing vehicles and considering the relative frequency of specific classes of accidents, it is possible to understand what technologies will be able to impact these metrics in the future.
[[File:3AISC-accidents-by-aircraft-type.png | 800px | Accidents per Aircraft type, Worldwide Commercial Fleet]]


|}
''Embraer aicraft highlighted in green.''
 
=== Detail of Accidents per Passenger Hour (US-based General Aviation fleet) ===


== Detail of Accidents per Passenger Hour ==
The general aviation (GA) fleet has steadily gotten safer over the last 20 years and we predict this trend will continue. This fleet is where most aviation accidents occur, due to older aircraft and less trained pilots than then more professional commercial fleet. Because of these features of the GA fleet, it has lots of room to improve with cockpit improvements and attention aids for pilots.


[[File:3AISC-accidents-per-flight-hour.png | 800px | Normalized Accidents per Year, United States General Aviation Fleet]]
[[File:3AISC-accidents-per-flight-hour.png | 800px | Normalized Accidents per Year, United States General Aviation Fleet]]


== Alignment with Company Strategic Drivers ==
== Biometric Activity Accuracy ==
 
Biometric Activity Accuracy is a measure of the performance of the system. In this case, biometric activity is used to determine if the pilot becomes startled or suffers from a lack of attention to the activity of flying or other critical actives of controlling the aircraft. The classifier is a binary classifier that either reports normal (attentive) or abnormal (startled or inattentive). This is calculated with the equation below.
 
<html>
<la-tex>
{Accuracy} = \frac{{correct}\;{classifications}}{{all}\;{classifications}} = \frac{TP + TN}{TP + TN + FP + FN}
</la-tex>
</html>
 
<!--
<html>
<la-tex>
{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}
</la-tex>
</html>
-->
 
Accuracy in general when applied to a binary classifier of this type is reported as a ratio of correct classifications to all classifications. This is calculated as the sum of true positives (TP) and true negatives (TN) to the total number of classifications, which is the sum of all responses to the classifier, true positive (TP), true negative (TN), false positive (FP) and false negative (FN).
 
= Alignment with Company Strategic Drivers =
 
{| class="wikitable" style="width: 800px;"
|+ Strategic drivers and targets
 
! ID
! Strategic Driver
! Alignment Targets
|- style="vertical-align:top;"
| 1
|
Provide framework that reduces the number of pilots required for flight to meet projected demand of commercial aviation.
 
“Demand for pilots is likely to soar. Projections indicate that, over the next 20 years, 602,000 new pilots will be needed to meet demand from commercial operators.” – Boeing Pilot and Technician Outlook 2022-2041
|
Our roadmap is focused on developing technology that identifies and adjusts pilot operation conditions to reduce perceived workload after adverse events. If this technology is successful, it will help to create a case for safe single pilot operations. Single pilot operations are not the primary purpose of the technology but is a possible outcome that assists in demand drivers. Therefore, this strategic driver is aligned with our technology roadmap, but isn’t the primary focus of our roadmap.
|- style="vertical-align:top;"
| 2
|
Reduce perceived stress/workload after adverse event by 35% in single pilot operations.
|
Our roadmap aims to develop a technology that will accurately measure pilot physiological parameters, model the status of pilot and aircraft, and augment the pilot's environment to reduce stress after adverse events. This strategic driver is strongly aligned with our roadmap.
|}
 


<!--
[[File:AVIAISD.png | 800px | Normalized Accidents per Year, United States General Aviation Fleet]]
[[File:AVIAISD.png | 800px | Normalized Accidents per Year, United States General Aviation Fleet]]
-->


== Positioning of Company vs. Competition ==
= Positioning of Company vs. Competition =


The chart below summarizes some key metrics for the major players in aircraft manufacturing. There is a clear difference in market share, with Boeing and Airbus demonstrating a duopoly over commercial aviation. Embraer and Bombardier have focused their efforts on commercial aviation's regional and business segments.  
The chart below summarizes some key metrics for the major players in aircraft manufacturing. There is a clear difference in market share, with Boeing and Airbus demonstrating a duopoly over commercial aviation. Embraer and Bombardier have focused their efforts on commercial aviation's regional and business segments.  


[[File:AVIAISum.png | 800px | Summary of Major Aircraft Manufacturers]]
[[File:AVIAICOMP.png | 800px | Summary of Major Aircraft Manufacturers]]




Line 113: Line 242:




The chart below shows that Airbus is the only major manufacturer actively investing in RDO / SPO. There is an opportunity for companies to differentiate their architectures and provide a solution that will assist the industry in meeting pilot demands in the future by providing single-pilot aircraft.   
The chart below shows that Airbus is the only major manufacturer that shows public-facing activity in RDO / SPO R&D. There is an opportunity for companies to differentiate their architectures and provide a solution that will assist the industry in meeting pilot demands in the future by providing single-pilot aircraft.   


[[File:AVIAISPO.png | 800px | Adaptive Cockpit Development]]
[[File:AVIAISPO.png | 800px | Adaptive Cockpit Development]]


== Technical Model ==
= Technical Model =
The development of an adaptive cockpit involves several key design considerations, particularly regarding bio-feedback devices. These devices play a crucial role in monitoring pilot physiological states, but their implementation requires careful balancing of accuracy and practicality.
 
 
'''Key Points'''
# Accuracy vs. Practicality: More accurate bio-feedback devices often come at the cost of being more cumbersome and potentially distracting for pilots.
# Cognitive Load Concerns: Highly accurate devices may inadvertently create unnecessary cognitive burden, which could counteract their intended benefits.
# R&D Focus: Future research and development efforts should prioritize creating bio-feedback devices that maintain high accuracy while minimizing invasiveness and pilot discomfort.
 
[[File:3AISC-morphological-matrix.png | 800px | Morphological Matrix ]]
 
= Financial Model =
 
Based on a reduction of accidents by 75% of the Embraer commercial fleet of 11% of the total, mean accident cost to industry of $170M, and discount rate of 12% the financial model of the value of this project is below. Additionally the project has an assumption of a rollout of the system to the global fleet over 7 years.
 
* Embraer commands 11% of the commercial fleet worldwide
* Improving pilot cognition can reduce accident rate by 75%
* The mean cost for a catastrophic accident to industry is $170M
* Discount rate of 12%
* R&D spend is allocated over the schedule in the R&D Projects section
* Retrofit into the fleet is over 7 years after conclusion of R&D effort
 
As shown below this shows the Net Present Value of the effort at ~$145M.
 
[[File:AVIA-financial-model.png | 800px | Financial Model]]
 
= List of R&D Projects =
 
The R&D project timeline has been developed based on limited information about competitors’ progress in Single Pilot and Reduced Crew Operations. Airbus’ Project Morgan and Project DISCO, which aim to implement single pilot operations in Fedex freight aircraft by 2030, provide the only publicly available insights. To maintain competitiveness with major manufacturers like Airbus, Embraer targets R&D completion by 2032, with the subsequent availability of aircraft and training for customers. The R&D trajectory begins by enhancing component-level technologies that best indicate pilot status, then advances to developing precise modeling algorithms and automation. The final phase of R&D projects focuses on human factors research, design, prototyping, and training development. This strategic approach ensures Embraer remains at the forefront of aviation technology while addressing the complex challenges of reduced crew operations.
 
[[File:AVIAIR&D3.png | 800px | R&D Projects]]
 
= Key Publications and Patents =
 
== Publications ==
 
# '''Masi, G.; Amprimo, G.; Ferraris, C.; Priano, L. Stress and Workload Assessment in Aviation—A Narrative Review.  Sensors 2023, 23, 3556. https://doi.org/10.3390/s23073556'''
#: This paper reviews various workload and stress assessment tools used within the military and civilian aviation sector. Stress and workload are key indicators of pilot performance, but measuring these conditions is not straightforward. Some of the highlighted indicators used for determining stress and workload are neurophysical (heart rate (HR) and heart rate variability (HRV)), respiratory rate, electrodermal activity (EDA), body temperature, eye movements and dilation, etc. Subjective measures such as the NASA-TLX can also be used to determine workload. One of the major conclusions was that there is no positive indicator for stress and workload in pilots. All the indicators mentioned above are only potential indicators for stress, so a comprehensive approach must be taken to model the status of the pilot. Most of the studies reviewed in this analysis used a combination of assessments to deduce stress and workload. The most common neurophysical used in the studies was HR and HRV. The authors comment on the feasibility of taking these measurements during flight, recommending that the devices should be light and not encumber the pilot's ability to function by using single-lead wearables or smart devices.
#: The authors suggest that more research should be done on eye tracking, near-infrared spectroscopy, grip strength during flight, and other parameters. Lastly, the author recommends further research into adaptive cockpits.
#: This review is an excellent source of background science regarding safety in aviation as it relates to pilot performance. The review's conclusion suggests that a combination of assessments is required, that there is still work to do to establish a reliable model for pilot stress and workload, and that adaptive cockpits would be the potential practical application of these findings.
 
[[File:A1.png | 400px | ST vs RF]]
 
# '''Li, Qinbiao & Chen, Chun-Hsien & Ng, Kam K.H. & Yuan, Xin & Yiu, Cho Yin. (2024). Single-pilot operations in commercial flight: Effects on neural activity and visual behaviour under abnormalities and emergencies. Chinese Journal of Aeronautics. 37. 10.1016/j.cja.2024.04.007.'''
#: This study explores the feasibility and safety of single-pilot operations (SPO) in commercial aviation. Using a flight simulator, 20 licensed pilots were tested in dual-pilot (DPO) and SPO scenarios. Findings reveal that neural activity, particularly in the frontal, parietal, and temporal brain regions, increased significantly during complex emergencies in SPO, indicating higher cognitive demands. Eye-tracking data showed that pilots in SPO focused less on the primary flight display (PFD) and more on secondary displays, reflecting dispersed attention and increased workload. Pilots made more operational errors in SPO, especially during complex scenarios like dual-engine failures and single-engine fires. Subjective feedback highlighted that pilots perceived SPO as having higher workload, reduced performance, and greater safety risks compared to DPO. The study identified specific physiological patterns, such as heightened cortical activity and altered visual scanning behaviors, which correlated with operational errors in SPO. To mitigate these risks, the study recommends integrating ground-based support systems and SPO-oriented intelligent flight systems to provide real-time assistance and maintain situational awareness. Additionally, improved cockpit design, such as head-up displays (HUD), is suggested to streamline information and reduce workload. Technological advancements, regulatory changes, and tailored training for single-pilot scenarios are critical to addressing these challenges. While SPO shows potential, its safe implementation depends on overcoming significant human performance and safety concerns.
#: This thesis directly supports the need for the technology we are roadmapping. The author provides the context surrounding the shift to SPO, and the risks associated with this transition.
 
[[File:A2.png | 400px | Eye Tracking]]
 
== Patents ==
 
# '''Eye Gaze Tracking – US Patent No. US11899837B2 – CPC G06F'''
#: The Eye Gaze patent shows a new way to detect and track eye gaze. The primary components of the patent are a visible wavelength camera, an infrared camera, and processors. It works by using the IR cameras to develop a 3D point cloud of a person’s face. It also creates a 2D image of the face with the visible wavelength camera. The processors take this data and create symmetry planes, which can be used to determine where the person is looking.
#: This technology is relevant because the publication review suggested that eye-tracking is a great indicator of pilot status. However, wearing devices that obstruct vision or could bounce around during flight is not ideal. This technology addresses both of those concerns.
 
[[File:1.1.png | 200px | Eye Gaze Tracking]]
 
 
# '''Smart Ring System for Monitoring Sleep Patterns And Using Machine Learning Techniques To Predict High-Risk Driving Behavior – US Patent No. US12077193B1– CPC B60W'''
#: The patent above is for a smart ring that monitors sleep patterns in users and predicts high-risk behaviors through machine learning. The device is a ring, similar to an Oura ring, that tracks HR/HRV, body temperature, motion, etc. This data can be used in conjunction with vehicle data to predict high-risk behavior.
#: While this patent is centered around sleep and performance of an automobile driver, the general concept can be applied in other domains. It shows that small, non-obstructive devices can be used to collect data, transmit data, and inform models that could improve people's performance in control of transportation systems.
 
[[File:2.1.png | 400px | Smart Ring]]


# '''Systems And Methods for Monitoring Pilot Health – US Patent No. US20190090800A1– CPC A61B'''
#: This patent outlines a system for monitoring the health of a pilot and status of the aircraft. It also contains a way to analyze the pilot status and interface with them. Lastly, this patent also includes an actuator to take control of the aircraft if necessary.
#: This patent is incredibly relevant as it is nearly identical to our technology with a few exceptions. The main difference is our technology aims to provide biofeedback to get the pilot back into normal operating conditions. In contrast, this patent utilizes a robot-pilot to manually control the aircraft in an emergency. The process up until that point is nearly identical.


== Key Publications and Patents ==
[[File:3.1.png | 300px | Pilot Health Monitoring]]
[[File:3.2.png | 300px | Pilot Health Monitoring]]
[[File:3.3.png | 300px | Pilot Health Monitoring]]


Publications
== Related Projects ==


1.) Masi, G.; Amprimo, G.; Ferraris, C.; Priano, L. Stress and Workload Assessment in Aviation—A Narrative Review.  Sensors 2023, 23, 3556. https://doi.org/10.3390/s23073556
* [https://haikuproject.eu/use-case-1/ Haiku Project - Use Case 1 - Intelligent Assistant]
a. Summary
i. This paper reviews various workload and stress assessment tools used within the military and civilian aviation sector. Stress and workload are key indicators of pilot performance, but measuring these conditions is not straightforward. Some of the highlighted indicators used for determining stress and workload are neurophysical (heart rate (HR) and heart rate variability (HRV)), respiratory rate, electrodermal activity (EDA), body temperature, eye movements and dilation, etc. Subjective measures such as the NASA-TLX can also be used to determine workload.
b. Methodology
i. This paper reviewed 48 studies from 2012-2022 that investigated workload and stress in pilots.
c. Findings
i. One of the major conclusions was that there is no positive indicator for stress and workload in pilots. All the indicators mentioned above are only potential indicators for stress, so a comprehensive approach must be taken to model the status of the pilot. Most of the studies reviewed in this analysis used a combination of assessments to deduce stress and workload. The most common neurophysical used in the studies was HR and HRV. The authors comment on the feasibility of taking these measurements during flight, recommending that the devices should be light and not encumber the pilot's ability to function by using single-lead wearables or smart devices.
ii. The authors suggest that more research should be done on eye tracking, near-infrared spectroscopy, grip strength during flight, and other parameters. Lastly, the author recommends further research into adaptive cockpits.
d. Relevancy
i. This review is an excellent source of background science regarding safety in aviation as it relates to pilot performance. The review's conclusion suggests that a combination of assessments is required, that there is still work to do to establish a reliable model for pilot stress and workload, and that adaptive cockpits would be the potential practical application of these findings.


2.) Li, Qinbiao & Chen, Chun-Hsien & Ng, Kam K.H. & Yuan, Xin & Yiu, Cho Yin. (2024). Single-pilot operations in commercial flight: Effects on neural activity and visual behaviour under abnormalities and emergencies. Chinese Journal of Aeronautics. 37. 10.1016/j.cja.2024.04.007.
* [https://executive.embraer.com/global/en/pulse Embraer Pulse Concept Jet]
a. Summary
i. This publication is a thesis that focuses on enhancing flight safety through neurophysical analysis and mental state predictions in single-pilot operations (SPO). As the airline industry seems primed to shift from dual-pilot operations (DPO) to SPO, a new method for sustaining safe flights must be adopted. This thesis aims to develop a system to monitor pilots and accurately identify off-nominal statuses in real time.
b. Methodology
i. The researcher performed various experiments on trained pilots in simulators. They collected neurophysiological data such as EEG and eye movements. In some experiments, the scenario was transitioned from DPO to SPO to investigate how the neurophysiological data changes. The data was then cleaned and analyzed
c. Findings
i. The primary finding reported here is that using, analyzing, and modeling this data type is a valid way to identify risky flight behaviors. The researcher claims an accuracy of up to 92.86%.
ii. A few other conclusions worth mentioning are that visual behaviors were pronounced during SPO, meaning the pilots were fixated on single points, especially during emergencies. Additionally, human error was more likely during SPO, and pilots reported feeling more stressed.
d. Relevancy
i. This thesis directly supports the need for the technology we are roadmapping. The author provides the context surrounding the shift to SPO, and the risks associated with this transition. Additionally, the research suggests it is feasible to implement the strategy our technology demonstrates.


* [https://www.aerospacetestinginternational.com/features/building-an-autonomous-business-jet.html Aerospace Testing International - Building an autonomous business jet]


Patents


1.) Eye Gaze Tracking – US Patent No. US11899837B2 – CPC G06F
= Technology Strategy Statement =
a. The Eye Gaze patent shows a new way to detect and track eye gaze. The primary components of the patent are a visible wavelength camera, an infrared camera, and processors. It works by using the IR cameras to develop a 3D point cloud of a person’s face. It also creates a 2D image of the face with the visible wavelength camera. The processors take this data and create symmetry planes, which can be used to determine where the person is looking.
b. This technology is relevant because the publication review suggested that eye-tracking is a great indicator of pilot status. However, wearing devices that obstruct vision or could bounce around during flight is not ideal. This technology addresses both of those concerns.


2.) Smart Ring System for Monitoring Sleep Patterns And Using Machine Learning Techniques To Predict High-Risk Driving Behavior – US Patent No. US12077193B1– CPC B60W
The aviation industry faces a significant challenge as the demand for pilots continues to outpace the available supply. By 2029, a global shortage of 50,000 pilots is anticipated, necessitating alternative solutions to address this growing gap. One promising approach to mitigate this shortage is to reduce the number of pilots required to operate aircraft. This strategy involves a phased implementation, starting with reduced crew operations, progressing to single-pilot operations for cargo aircraft (which pose no risk to passengers), and eventually extending to commercial aircraft.
a. The patent above is for a smart ring that monitors sleep patterns in users and predicts high-risk behaviors through machine learning. The device is a ring, similar to an Oura ring, that tracks HR/HRV, body temperature, motion, etc. This data can be used in conjunction with vehicle data to predict high-risk behavior.  
To achieve this goal, a comprehensive plan for research and development is proposed, focusing initially on improving component-level systems and enhancing accuracy-related figures of merit. These foundational projects are slated for completion by 2025. Following this, the development of an adaptive cockpit will progress through stages of modeling, human factors analysis, design, and prototyping. A fully functional prototype is expected to be ready by 2030, with pilot training commencing immediately thereafter. The ultimate objective of this roadmap is to have single-pilot capable aircraft available for sale to freight companies by 2032, marking a significant milestone in aviation technology and addressing the critical pilot shortage.  
b. While this patent is centered around sleep and performance of an automobile driver, the general concept can be applied in other domains. It shows that small, non-obstructive devices can be used to collect data, transmit data, and inform models that could improve people's performance in control of transportation systems.  


3.) Systems And Methods for Monitoring Pilot Health – US Patent No. US20190090800A1– CPC A61B
[[File:AVIAISWOOSH.png | 800px | Swoosh Chart]]
a. This patent outlines a system for monitoring the health of a pilot and status of the aircraft. It also contains a way to analyze the pilot status and interface with them. Lastly, this patent also includes an actuator to take control of the aircraft if necessary.
b. This patent is incredibly relevant as it is nearly identical to our technology with a few exceptions. The main difference is our technology aims to provide biofeedback to get the pilot back into normal operating conditions. In contrast, this patent utilizes a robot-pilot to manually control the aircraft in an emergency. The process up until that point is nearly identical.

Revision as of 00:11, 4 December 2024

Adaptive cockpit is a level 3 roadmap (3ADC) created by Dave Bertucci and Tyler Carey with the support and mentorship of Embraer, specifically Clenilson Goncalves, Giuliano Mendonca, and Julio Bolzani.

Roadmap Overview

An adaptive cockpit is an advanced, AI-driven system designed to enhance aircraft safety and pilot performance during unexpected or high-stress situations. This technology integrates various sensors and intelligent algorithms to monitor both the aircraft’s status and the pilot’s physiological state in real time. Key features of an adaptive cockpit include:

  1. Continuous monitoring of aircraft systems and flight parameters
  2. Real-time analysis of pilot biometrics and behavior
  3. Detection of sudden, potentially disorienting events or anomalies
  4. Intelligent assessment of the pilot’s cognitive and emotional state
  5. Provision of timely, context-aware assistance and guidance

When the system detects signs of pilot stress, confusion, or delayed responses, it intervenes with appropriate support. This may include a simplified display of critical information, biofeedback to reduce stress, or automated assistance with certain tasks. The adaptive cockpit aims to mitigate the impact of human factors such as the startle effect and cognitive overload. By providing tailored, real-time support, pilots can more quickly regain situational awareness and make informed decisions during critical moments. This technology represents a significant advancement in aviation safety, offering a proactive approach to managing complex scenarios and reducing operational risks. As it continues to evolve, the adaptive cockpit has the potential to revolutionize cockpit design and pilot training, ultimately leading to safer and more efficient flight operations.

Picture of FOCUS

Design Structure Matrix (DSM) Allocation

Our technology roadmap focuses on an Adaptive Cockpit (3ADC) at the subsystem level (level 3). Components of the adaptive cockpit are AI for interpreting and modeling data to determine aircraft and pilot status (4AI), Biometric Sensors for measuring pilot status indicators (4BIO), aircraft sensors for detecting aircraft status (4ACS), adaptive displays that can highlight critical information for a pilot (4ADD), a lighting system that assists pilots in maintaining respiratory rates and creating a calming atmosphere (4LIS), and the pilot responsible for flying the aircraft (4PIL). Other subsystems in the cockpit include navigation and communication systems (3NAC), and flight controls (3FLC). We have also listed other level 2 products that support our level 1 market of safe commercial aviation (1SCA).

Decomposition of 3ADC

DSM of 3ADC

Roadmap OPD

In the figure below, we provide an Object-Process-Diagram (OPD) of the 3ADC roadmap. This diagram captures the main object of the roadmap (Adaptive Cockpit), its main process (Teaming), the various tools required, the sub-processes required for Teaming, and the change in the status of the pilot as a result of Teaming. Additionally, we have included a zoomed-in look into the Temaing process and its four sub-processes (Detecting, Modeling, Supporting, and Augmenting).

OPD


The Object Process Language (OPL) below is the auto-generated grammatical version of the OPD above. The first OPL is for the Level 1 OPD, whereas the second OPL is for the decomposition of Teaming into its four sub-processes.


Level 1 OPL


Level 2 OPD

Figures of Merit

Many figures of merit were considered for this roadmap and are outlined below with more information and current trends on each. The first 3 were found to be especially important for the roadmap and are shown below in bold.

Important FOMs for 3ADC
Figure of Merit Description Trends Units
Pilot Efficiency Measure of how many passengers each pilot is capable of transporting per year increasing

passengers pilot * year

Accident Rate Rate of incidents and accidents normalized by flight time aircraft decreasing

accidents passenger mile

Biometric Activity Accuracy

Accuracy of best models of pilot pilot metrics to predict attention and startle response

increasing

%

System Maintainability How much time is spend maintaining aircraft systems, measured by comparing maintenance time per flight time decreasing

maintenance hours passenger mile

Response Time Time required to identify and mitigate safety issues flat

minutes

System Uptime Amount of time aircraft is available to be dispatched on a mission as a percentage of wall time increasing

%

Cognitive Load on Crew As measured by industry standard NASA TLX score flat

unitless

Pilot Efficiency

Pilots are currently a critical part of air transportation systems and making sure they are efficiently transporting passengers is an important figure of merit (FOM) of an airline. The definition of pilot efficiency is below.

{Efficiency}_{Pilot} = \frac{{Capacity}_{Pass} * {LF}_{Pass}}{{Pilot}_{Plane}} * {Flight}_{Year}


Pilot Efficiency Variable Detail

Pilot Efficiency Tornado Chart

Accident Rate

Detail of Accident Rates (per million flight cycle), 1959 to 2022

Large commercial aircraft have steadily been getting safer. The table below shows accident rates of aircraft organized by the date of their first flights (oldest first). This chart highlights that newer designs have proven to be safer than older designs. Additionally the chart categorizes aircraft by the generation of automation used in each.

Generation 1 (Not shown)
Analog gauges and early auto-pilot systems
Generation 2
Integration of auto-pilot and auto-throttle systems
Generation 3
Glass cockpits, improved navigation systems, and terrain avoidance systems
Generation 4
Fly-by-wire and envelope protection systems

As these successive generations of technology have been integrated into aircraft the accident rates have dropped dramatically. Since safety record is a property of the system as a whole and the complexity of an aircraft is so high, generating an analytical model of the contributing factors is very complex, but by comparison to existing vehicles and considering the relative frequency of specific classes of accidents, it is possible to understand what technologies will be able to impact these metrics in the future.

Accidents per Aircraft type, Worldwide Commercial Fleet

Embraer aicraft highlighted in green.

Detail of Accidents per Passenger Hour (US-based General Aviation fleet)

The general aviation (GA) fleet has steadily gotten safer over the last 20 years and we predict this trend will continue. This fleet is where most aviation accidents occur, due to older aircraft and less trained pilots than then more professional commercial fleet. Because of these features of the GA fleet, it has lots of room to improve with cockpit improvements and attention aids for pilots.

Normalized Accidents per Year, United States General Aviation Fleet

Biometric Activity Accuracy

Biometric Activity Accuracy is a measure of the performance of the system. In this case, biometric activity is used to determine if the pilot becomes startled or suffers from a lack of attention to the activity of flying or other critical actives of controlling the aircraft. The classifier is a binary classifier that either reports normal (attentive) or abnormal (startled or inattentive). This is calculated with the equation below.

{Accuracy} = \frac{{correct}\;{classifications}}{{all}\;{classifications}} = \frac{TP + TN}{TP + TN + FP + FN}


Accuracy in general when applied to a binary classifier of this type is reported as a ratio of correct classifications to all classifications. This is calculated as the sum of true positives (TP) and true negatives (TN) to the total number of classifications, which is the sum of all responses to the classifier, true positive (TP), true negative (TN), false positive (FP) and false negative (FN).

Alignment with Company Strategic Drivers

Strategic drivers and targets
ID Strategic Driver Alignment Targets
1

Provide framework that reduces the number of pilots required for flight to meet projected demand of commercial aviation.

“Demand for pilots is likely to soar. Projections indicate that, over the next 20 years, 602,000 new pilots will be needed to meet demand from commercial operators.” – Boeing Pilot and Technician Outlook 2022-2041

Our roadmap is focused on developing technology that identifies and adjusts pilot operation conditions to reduce perceived workload after adverse events. If this technology is successful, it will help to create a case for safe single pilot operations. Single pilot operations are not the primary purpose of the technology but is a possible outcome that assists in demand drivers. Therefore, this strategic driver is aligned with our technology roadmap, but isn’t the primary focus of our roadmap.

2

Reduce perceived stress/workload after adverse event by 35% in single pilot operations.

Our roadmap aims to develop a technology that will accurately measure pilot physiological parameters, model the status of pilot and aircraft, and augment the pilot's environment to reduce stress after adverse events. This strategic driver is strongly aligned with our roadmap.


Positioning of Company vs. Competition

The chart below summarizes some key metrics for the major players in aircraft manufacturing. There is a clear difference in market share, with Boeing and Airbus demonstrating a duopoly over commercial aviation. Embraer and Bombardier have focused their efforts on commercial aviation's regional and business segments.

Summary of Major Aircraft Manufacturers


Aircraft are incredibly complex systems that impart a tremendous demand on pilots. We will look at some characteristics and traits of various aircraft and use them as a proxy for complexity. The chart below highlights the various levels of complexity in commercial aviation across segments and manufacturers.

Complexity of Competitors Aircraft


The chart below shows that Airbus is the only major manufacturer that shows public-facing activity in RDO / SPO R&D. There is an opportunity for companies to differentiate their architectures and provide a solution that will assist the industry in meeting pilot demands in the future by providing single-pilot aircraft.

Adaptive Cockpit Development

Technical Model

The development of an adaptive cockpit involves several key design considerations, particularly regarding bio-feedback devices. These devices play a crucial role in monitoring pilot physiological states, but their implementation requires careful balancing of accuracy and practicality.


Key Points

  1. Accuracy vs. Practicality: More accurate bio-feedback devices often come at the cost of being more cumbersome and potentially distracting for pilots.
  2. Cognitive Load Concerns: Highly accurate devices may inadvertently create unnecessary cognitive burden, which could counteract their intended benefits.
  3. R&D Focus: Future research and development efforts should prioritize creating bio-feedback devices that maintain high accuracy while minimizing invasiveness and pilot discomfort.

Morphological Matrix

Financial Model

Based on a reduction of accidents by 75% of the Embraer commercial fleet of 11% of the total, mean accident cost to industry of $170M, and discount rate of 12% the financial model of the value of this project is below. Additionally the project has an assumption of a rollout of the system to the global fleet over 7 years.

  • Embraer commands 11% of the commercial fleet worldwide
  • Improving pilot cognition can reduce accident rate by 75%
  • The mean cost for a catastrophic accident to industry is $170M
  • Discount rate of 12%
  • R&D spend is allocated over the schedule in the R&D Projects section
  • Retrofit into the fleet is over 7 years after conclusion of R&D effort

As shown below this shows the Net Present Value of the effort at ~$145M.

Financial Model

List of R&D Projects

The R&D project timeline has been developed based on limited information about competitors’ progress in Single Pilot and Reduced Crew Operations. Airbus’ Project Morgan and Project DISCO, which aim to implement single pilot operations in Fedex freight aircraft by 2030, provide the only publicly available insights. To maintain competitiveness with major manufacturers like Airbus, Embraer targets R&D completion by 2032, with the subsequent availability of aircraft and training for customers. The R&D trajectory begins by enhancing component-level technologies that best indicate pilot status, then advances to developing precise modeling algorithms and automation. The final phase of R&D projects focuses on human factors research, design, prototyping, and training development. This strategic approach ensures Embraer remains at the forefront of aviation technology while addressing the complex challenges of reduced crew operations.

R&D Projects

Key Publications and Patents

Publications

  1. Masi, G.; Amprimo, G.; Ferraris, C.; Priano, L. Stress and Workload Assessment in Aviation—A Narrative Review. Sensors 2023, 23, 3556. https://doi.org/10.3390/s23073556
    This paper reviews various workload and stress assessment tools used within the military and civilian aviation sector. Stress and workload are key indicators of pilot performance, but measuring these conditions is not straightforward. Some of the highlighted indicators used for determining stress and workload are neurophysical (heart rate (HR) and heart rate variability (HRV)), respiratory rate, electrodermal activity (EDA), body temperature, eye movements and dilation, etc. Subjective measures such as the NASA-TLX can also be used to determine workload. One of the major conclusions was that there is no positive indicator for stress and workload in pilots. All the indicators mentioned above are only potential indicators for stress, so a comprehensive approach must be taken to model the status of the pilot. Most of the studies reviewed in this analysis used a combination of assessments to deduce stress and workload. The most common neurophysical used in the studies was HR and HRV. The authors comment on the feasibility of taking these measurements during flight, recommending that the devices should be light and not encumber the pilot's ability to function by using single-lead wearables or smart devices.
    The authors suggest that more research should be done on eye tracking, near-infrared spectroscopy, grip strength during flight, and other parameters. Lastly, the author recommends further research into adaptive cockpits.
    This review is an excellent source of background science regarding safety in aviation as it relates to pilot performance. The review's conclusion suggests that a combination of assessments is required, that there is still work to do to establish a reliable model for pilot stress and workload, and that adaptive cockpits would be the potential practical application of these findings.

ST vs RF

  1. Li, Qinbiao & Chen, Chun-Hsien & Ng, Kam K.H. & Yuan, Xin & Yiu, Cho Yin. (2024). Single-pilot operations in commercial flight: Effects on neural activity and visual behaviour under abnormalities and emergencies. Chinese Journal of Aeronautics. 37. 10.1016/j.cja.2024.04.007.
    This study explores the feasibility and safety of single-pilot operations (SPO) in commercial aviation. Using a flight simulator, 20 licensed pilots were tested in dual-pilot (DPO) and SPO scenarios. Findings reveal that neural activity, particularly in the frontal, parietal, and temporal brain regions, increased significantly during complex emergencies in SPO, indicating higher cognitive demands. Eye-tracking data showed that pilots in SPO focused less on the primary flight display (PFD) and more on secondary displays, reflecting dispersed attention and increased workload. Pilots made more operational errors in SPO, especially during complex scenarios like dual-engine failures and single-engine fires. Subjective feedback highlighted that pilots perceived SPO as having higher workload, reduced performance, and greater safety risks compared to DPO. The study identified specific physiological patterns, such as heightened cortical activity and altered visual scanning behaviors, which correlated with operational errors in SPO. To mitigate these risks, the study recommends integrating ground-based support systems and SPO-oriented intelligent flight systems to provide real-time assistance and maintain situational awareness. Additionally, improved cockpit design, such as head-up displays (HUD), is suggested to streamline information and reduce workload. Technological advancements, regulatory changes, and tailored training for single-pilot scenarios are critical to addressing these challenges. While SPO shows potential, its safe implementation depends on overcoming significant human performance and safety concerns.
    This thesis directly supports the need for the technology we are roadmapping. The author provides the context surrounding the shift to SPO, and the risks associated with this transition.

Eye Tracking

Patents

  1. Eye Gaze Tracking – US Patent No. US11899837B2 – CPC G06F
    The Eye Gaze patent shows a new way to detect and track eye gaze. The primary components of the patent are a visible wavelength camera, an infrared camera, and processors. It works by using the IR cameras to develop a 3D point cloud of a person’s face. It also creates a 2D image of the face with the visible wavelength camera. The processors take this data and create symmetry planes, which can be used to determine where the person is looking.
    This technology is relevant because the publication review suggested that eye-tracking is a great indicator of pilot status. However, wearing devices that obstruct vision or could bounce around during flight is not ideal. This technology addresses both of those concerns.

Eye Gaze Tracking


  1. Smart Ring System for Monitoring Sleep Patterns And Using Machine Learning Techniques To Predict High-Risk Driving Behavior – US Patent No. US12077193B1– CPC B60W
    The patent above is for a smart ring that monitors sleep patterns in users and predicts high-risk behaviors through machine learning. The device is a ring, similar to an Oura ring, that tracks HR/HRV, body temperature, motion, etc. This data can be used in conjunction with vehicle data to predict high-risk behavior.
    While this patent is centered around sleep and performance of an automobile driver, the general concept can be applied in other domains. It shows that small, non-obstructive devices can be used to collect data, transmit data, and inform models that could improve people's performance in control of transportation systems.

Smart Ring

  1. Systems And Methods for Monitoring Pilot Health – US Patent No. US20190090800A1– CPC A61B
    This patent outlines a system for monitoring the health of a pilot and status of the aircraft. It also contains a way to analyze the pilot status and interface with them. Lastly, this patent also includes an actuator to take control of the aircraft if necessary.
    This patent is incredibly relevant as it is nearly identical to our technology with a few exceptions. The main difference is our technology aims to provide biofeedback to get the pilot back into normal operating conditions. In contrast, this patent utilizes a robot-pilot to manually control the aircraft in an emergency. The process up until that point is nearly identical.

Pilot Health Monitoring Pilot Health Monitoring Pilot Health Monitoring

Related Projects


Technology Strategy Statement

The aviation industry faces a significant challenge as the demand for pilots continues to outpace the available supply. By 2029, a global shortage of 50,000 pilots is anticipated, necessitating alternative solutions to address this growing gap. One promising approach to mitigate this shortage is to reduce the number of pilots required to operate aircraft. This strategy involves a phased implementation, starting with reduced crew operations, progressing to single-pilot operations for cargo aircraft (which pose no risk to passengers), and eventually extending to commercial aircraft. To achieve this goal, a comprehensive plan for research and development is proposed, focusing initially on improving component-level systems and enhancing accuracy-related figures of merit. These foundational projects are slated for completion by 2025. Following this, the development of an adaptive cockpit will progress through stages of modeling, human factors analysis, design, and prototyping. A fully functional prototype is expected to be ready by 2030, with pilot training commencing immediately thereafter. The ultimate objective of this roadmap is to have single-pilot capable aircraft available for sale to freight companies by 2032, marking a significant milestone in aviation technology and addressing the critical pilot shortage.

Swoosh Chart