April 24, 2019

Ensuring harmonious human–machine collaboration in the cockpit

Led by CSEM, the PEGGASUS consortium is set to enable new types of human–machine interface (HMI) across cockpit avionics, pushing the boundaries of augmentation in the cockpit. Using the latest in artificial intelligence (AI) and computer vision technologies, this European-funded project will integrate—for the first time—remote eye-gaze tracking and gesture recognition for pilots in a single framework. Its purpose is to enhance human-machine interaction in the complex flight operations of today’s cockpits through pilot monitoring, for applications aiming to improve crew efficiency and pilot training, towards the development of new generation cockpits.

Two pilots in the cockpit

Since autopilot was first demonstrated in 1914, pilots have found themselves increasingly “connected” to their aircrafts through numerous displays, knobs and instruments in the cockpit. This increasing degree of pilot assistance has been developed to help reduce pilot workload, always with aviation safety as a driver.

Paradoxically however there is the risk that when faced with an atypical event pilots encounter difficulties coping with the vast amounts of information generated by their instruments. It is therefore crucial to improve and optimize their situational awareness and the relationship between the flight crew and the aircraft controls.

PEGGASUS—optimizing human–machine interactions

Endorsed by the Clean Sky 2 initiative, the European project PEGGASUS aims to counter the “instruments paradox” by understanding crew members’ actions and behaviour and by moving towards a multimodal cockpit interactivity, thus allowing pilots greater levels of control. “We need to recognize when limitations such as confusion or drowsiness impact attention, mental workload, and decision-making on the flight deck,” explains Andrea Dunbar, Head of Embedded Vision Systems at CSEM. “Additionally, the new HMI we are developing will eventually enable a more intuitive and natural interaction so they can make quick, informed decisions across any situation, even when stressed.”

To design a powerful and tailored solution, CSEM and three partners will provide the essential complementary skills, ensuring the project’s success. “Our company is pleased to support the consortium partners with the expertise of a professional airline,” says Christoph Ammann, Vice-President, Head of Crew Training at Swiss International Air Lines, a member of the Lufthansa Group. “Mutual exchange with research and industry partners enables us to reflect on our training standards and on potential future applications."

The airline has previously collaborated with consortium member ETH Zurich to develop novel gaze-based techniques to monitor pilots’ cognitive states and situational awareness. In particular, these new methods were designed to allow more efficient and effective interaction between the pilot and the aircraft, while also expanding pilot training techniques. The consortium will build upon this excellent experience base and previously collected data when developing PEGGASUS.

Integrating HMI systems into the cockpit

“The aeronautic context poses unique challenges for our team,” comments Andrea Dunbar. “The vision systems and machine learning algorithms CSEM will develop must take into account both the pilots in the cockpit. The technology will be developed to remain accurate and robust during the course of a flight, considering aviation-specific environmental factors such as changing lighting conditions and vibrations.” Consortium partner SERMA Ingénierie will be responsible for integrating the PEGGASUS output into a cockpit prototype for testing.

The consortium’s work will also be supported by Thales, Clean Sky 2 leader. Thales’ Thierry Maret, Program Manager, is proud that PEGGASUS will mean another step forward in “providing pilots with new ways of interacting with the aircraft system so pilots can easily and efficiently adapt to the changing and complex needs of 21st century avionics.”

Europe flag and Clean Sky logo

The PEGGASUS project has received funding from the Clean Sky 2 Joint Undertaking under the European Union’s Horizon 2020 research and innovation program under grant agreement No. 821461.

Download press release
Download picture