HIPNOSIS: Smart Sensing Systems to Monitor Pilots

The HIPNOSIS approach is based on two sensing systems:

• A vision-based system integrated in the cockpit dashboard to monitor the pilots in the cockpit and detect behavior linked to drowsiness in real-time.
• A smart wristband to sense several bio signals through advanced optical sensing technologies, to detect physiological measures linked to drowsiness.

For such systems to be successfully adopted, a high level of accuracy and robustness is required, considering aviation-specific environmental factors such as extreme lighting conditions and vibrations.

HIPNOSIS vision system

The developed algorithmic pipeline for feature extraction, which is a combination of data-driven and analytics approaches, includes face detection and facial landmark localization, as well as the iris, glints, and pupil detection modules.

Highlights

• Maximized performance for a large field of view needed in avionics. The vision system is robust to pilots’ varying head orientation during the flight operations.
• Optimized architecture for real-time edge processing (> 40fps) with low latency. The acquisition and processing of high-resolution images runs at < 25ms with a latency of approximately 50ms.
• Technological transparency. Whilst matching high-end commercial systems, the HIPNOSIS system is transparent and adaptable to custom solutions based on the use case requirements.

HIPNOSIS illuminators

The tailored illuminators meet a need not covered by current COTs offering, i.e., strong Near InfraRed spectrum (850nm) illumination. This is essential for surveillance applications with NIR cameras which don’t inconvenience the user. Active illumination with tailored pulse duration and illumination intensity, allow image capture with a short exposure time giving sharper images when the subject is in motion. It also automatically regulates the orders received so as not to exceed the safety and health limits.

The Illuminator characteristics are:

• Powered by avionics 28V
• Communication with the control system via CAN BUS
• Adjustable on two axes of rotation
• Panel mounting showing only the light head possible
• Pulse lighting, with 60 pulses per second, for durations ranging from 1ms to continuous lighting
• Synchronization by an external signal allowing lighting and simultaneous image capture
• Compliance with NF EN 62471 standard: The radiation accessible at the output of the illuminators is in the “risk 0” group (without risk)

HIPNOSIS wristband

The HIPNOSIS wristband based on optical sensing technologies allows direct on- body analysis of pilot’s bio signals. The wristband includes monitoring of heart rate, heart rate variability, breathing rate, skin temperature, activity, and sleep. 

Optical heart rate variability monitoring

CSEM optical heart rate variability (HRV) monitoring relies on photoplethysmography (PPG) signals of high temporal resolution. This technology allows the detection of single cardiac beat events and the measurement of HRV in the time and frequency domains (as defined in the European HRV guidelines). This is computed via a CSEM proprietary analysis of arterial pulsatility patterns at any location on the body with sufficient perfusion.

Features

• Measures the arterial pulsatility, via photoplethysmography (PPG) with high temporal resolution in transmission or reflectance
• Estimates beat-to-beat time intervals
• Gives HRV measurement (time and frequency domains)

About HIPNOSIS

HIPNOSIS forms part of the European Clean Sky initiative, funded by the EU’s Horizon 2020 program. This initiative contributes to strengthening European leadership in aviation, focusing on the reduction of aircraft noise and CO2 and NOX emissions while promoting collaboration, global leadership, and competitiveness.  The project consists of the following partners: CSEM SA (CH) (the Project Coordinator), SERMA Ingénierie (F), and Innov+ (F).

The consortium partners worked in close collaboration with the topic leader Honeywell Aerospace, a leader in aerospace development and the manufacturer of avionics solutions. CSEM developed the vision system and machine learning algorithms for detecting behavioral features linked to drowsiness, addressing the unique challenges and requirements in the aeronautic context. In addition, CSEM has developed a smart wristband, based on optical sensing technologies, which allows direct on-body analysis of a pilot’s bio signals. SERMA Inge¿nierie was responsible for developing the HIPNOSIS vision system hardware, particularly the near-infrared illuminators synchronized with image capture, as well as integrating the system into the cockpit simulator. French startup Innov+, which already commercializes similar solutions for the automotive industry, contributed to the development of the HIPNOSIS software API.

The HIPNOSIS project has received funding from the Clean Sky 2 Joint Undertaking under the European Union’s Horizon 2020 research and innovation program under grant agreement No. 821466.