3.6

High-priority research and development and innovation (R&D&I) areas:

• Digital inclusion: tools, infrastructure, training, connectivity.
• Online education and examination: VR/AR training and support.
• Improved human–machine interaction (HCI) solutions.
• Support devices: wearables, robots, cobots, etc.
• Nudging and serious gaming: for personal development and healthier lifestyles.
• The trusted element: how to be sure that not more is done (under the surface)

To improve the awareness of our body’s condition to external or internal stimuli, smart systems can provide support for disabilities, or a personal coach and trainer to identify behaviour to be avoided (wrong body position and other unhealthy habits), as well as possible future injuries or disorders. Smart systems can also offer an immersive experience through vision, gaming, and sensory interaction by way of VR or AR. Consumers can be offered the immediacy, individualisation, interactivity, and immersion they expect from media content consumption (“even better than being there”).

A healthier and more comfortable environment can be offered based on personal preferences (control of temperature, humidity, air flux, etc.), in the context of running activities and clothing, and by adapting lighting and acoustic quality to one’s own sense of well-being. It also provides the capability to comfortably communicate and interact remotely with people, institutions, and sellers, possibly without leaving home, saving time for self- development and leisure.

Selective automation, AR at work and a range of feedback tools can help boost satisfaction and give more meaning to work. This is a particularly important element for the millennial generation, which -according to surveys- tends to place more emphasis on work satisfaction than on income (above a certain income level). Technological advances have made it possible to place audiences in the middle of the action and to offer them immediacy, individualisation, interaction, and immersion without the need for them to actually be there in person. This will further change consumption patterns and create new business opportunities.

Furthermore, we have to mitigate the risks of AI and other functionality that is based on massive data storage in the cloud. This can be done in different ways, such as:

• Introduce fake news detection by analysing AI-generated content on social media
• Develop technologies to prove the source of content creation, for instance by showing that a video was created with an actual camera. If part of the processing is done externally and/or in the cloud, add traceability of what was done where.
• Add watermarking to created content to proof that it is not artificial.
• Secure control of systems used for content creation, for instance by prevention of production equipment to be taken over by external aggressors in big events like Olympic Games, Eurovision song festival, European and World championships, etc.

3.6.3.1.2 Microelectronics and integrated photonics for XR applications

Currently, commercial head-mounted display solutions rely upon high-end microdisplays deploying various technologies such as organic light-emitting diodes (OLEDs), micro-LEDs, lasers, and LEDs with liquid crystal on silicon (LCoS), or microelectromechanical systems (MEMS). Nonetheless, these microdisplays will still need to undergo further improvements regarding size, power consumption (<300 mW; full display system), device efficiency, resolution (8K and beyond, pixel densities > 10kppi), high dynamic range (HDR), colour gamut, contrast and refresh rate. Besides utilizing classical visualization modes, plenoptic modes such as light field, multifocal and holography are gaining ground, holding the promise for truly immersive experiences and potentially improving vergence-accommodation behavior for (Head Mounted Device) HMDs and autostereoscopic displays. Both the classical approach and the plenoptic solutions require not only a significant evolution regarding the capabilities and features of the opto-nanoelectronic display devices – such as multi-ridge lasers and associated waveguide combiners, light-field (micro)displays, multifocal light engines, spatial light modulators (SLMs) based on LCOS, and MEMS – but also state-of-the-art drivers and optical systems and architectures.

These complex optical systems must support the required field of view (FoV), be lightweight and safe for the user, and provide excellent color uniformity and high brightness efficiency. Furthermore, they must satisfy the use case requirements for both professional use and mass adoption, i.e., be scalable to large volumes (cost-effective). The optical system will ultimately determine the image quality and, therefore, the end-user experience and market adoption of XR devices. 

Within this optical system, the design and fabrication of waveguide optics, holographic elements, diffractive optics, reflective light guides, freeform optics, and meta-surfaces with optimal optical properties and behaviour are crucial. They will be deterministic in obtaining the required form factors and user experience for the envisioned display devices. Furthermore, photonic integrated circuits and light-shaping optics based on liquid crystal devices, enabling the miniaturization of, e.g., waveguides, couplers, and lasers, combined with more macroscale components such as lenses, will also be essential to reach these goals.

Other key parts of XR solutions are the variety of integrated sensors to create an immersive and interactive experience. Vital signs sensors additionally contribute to personal health monitoring and wellness.

Eye tracking can serve as a hand-free communication between the user and the system; it allows for adaptive image color correction and rendering at the point where the eye points at; can provide gaze point prediction and user sensing functionality (user emotions for avatar, tiredness detection while driving etc.). However, they face challenges such as accuracy in diverse lighting conditions and with different eye physiologies. The precision in detecting subtle eye movements or distinguishing between intentional gaze and casual looking remains a limitation.

Other sensors such as ambient light sensors, spectral sensors or depth sensors must cope with the high dynamic range of ambient light (1 lux … 100 klux).

Intelligent LED/Laser Driver do control respective emitting devices but are also a crucial part of the optical eye tracking system to monitor correct operation of the LEDs/Lasers. They are constantly supervising illumination time and power therefore ensuring that the light output does not exceed safe limits even under single fault conditions.

In general, integrating these systems seamlessly into mixed-reality devices without increasing bulk or reducing comfort is challenging.

Required R&D&I developments within ECS

Taking the above into account, specific R&D developments are necessary within ECS technology, as shown in Figure 3.6.

Figure 3.6.3 - Required R&D&I developments within ECS –Major Challenge 1

High-priority R&D&I areas:

• Reliable and ubiquitous digital infrastructures.
• Access control/intrusion detection/surveillance.
• Provide protective environment and tools against virus infections.
• Protect individual citizens against cyber-fraud (scams) and identity theft.
• Off-grid living and emergency survival.

Since the Covid pandemic, working from home has become an integral part of how knowledge workers do their work. To further enable working from home (or wherever and whenever one wants), wireless and wired infrastructures will have to be further improved (through increased reliable bandwidth, lower cost, better geographical coverage and finer granularity), security of connections will have to improve to protect the worker at home (as will the company using a distributed workforce with many internet connections) against cyber-attacks, and the theft of personal and/or company information. New functionality running in the private/public cloud will be needed to support real-time actions that may suffer from latency issues over the internet, as well as to support the worker in decision-making. Examples here are control of robotic surgical devices, remote control of robots in industrial processes, remote control of cameras in security applications and live television productions. Other professions, such as translation services, voice recognition and all kind of analytical algorithms for data analysis, also come to mind.

To create equal opportunities, innovative research should include: speech-generating devices (SGD) to help people with speech disorders; exoskeletons that empower disabled people in their everyday life; semi-autonomous vehicles that increase mobility for people with deafness and blindness; smart objects linked to geospatial information to improve women’s security (e.g. invisible SOS buttons); augmentative and alternative communication tablets that help paralysed patients; VR solutions that provide realistic experiences for people with physical disabilities; and smart glasses that can be used to help people with autism on cognitive, social and emotional skills.

Given the experience of the past pandemic, we learned that Europe needs better technologies: (i) to fight and contain the rapid spread of highly contiguous diseases (such as Covid-19); and (ii) to ensure that public health institutions can maintain their capacity to meet the ever-increasing needs caused by such a pandemic⁷. The in-depth analysis provided by the European Parliamentary Research Service’s “Ten Technologies to Fight Coronavirus” identifies the importance of AI, blockchain, open-source-, telehealth- and gene-editing technologies, 3D printing, nanotechnology, synthetic biology, and drones and robots for fighting pandemics.

Intrinsically, technology is neither good nor bad – it is the use to which it is put that makes the difference. Malicious uses of technology include mass disinformation campaigns and cyber-attacks that seek to jeopardise national security, and cyber-fraud that targets consumers. This duality has always existed. Over the coming years, technologies such as the IoT, smart robotics, automation and AI are likely to follow the same pattern. It is up to European technology specialists to ensure that the technologies developed not only support diversity and inclusion, but also protect both the individual and groups against cyber-attacks, theft of personal information and unwanted intrusion into the personal environment.

Required R&D&I developments within ECS

To facilitate empowerment and resilience, specific R&D developments are necessary within ECS technology, as shown in Figure 3.6.4.

Figure 3.6.4 - Required R&D&I developments within ECS – Major Challenge 2

High-priority R&D&I areas:

• Digital inclusion: tools, infrastructure, training, connectivity.
• Collective safety: secure access control, surveillance, pandemic control, prevention of misinformation without limiting freedom of expression.
• Safe environment for living, working and transport: buildings and bridges resilient against earthquakes through continuous monitoring (e.g. fibre-based stress sensors).
• Emergency/crisis response solutions and services.
• Dynamics of society: systemic change.
• Affordable trustworthy ECS tools, miniaturization and energy management of ECS enabled services for society

As Europe wants to play a key role in digital inclusiveness, it is important to ensure availability and accessibility of solutions to enable remote education, learning, training and assessment of professionals, students, and consumers in all regions (both cities and rural areas). Also, solutions to support social inclusiveness for people of all age should become available.

The EU has stated, in their document on orientations towards the first strategic plan for implementing Horizon Europe⁸, that the interaction of science, technology, social sciences and humanities will be crucial in this respect, as will be the input of the creative sector and artists to sustainable inclusive innovation and human- oriented technologies.

To facilitate inclusion, more research will be needed on education, simple human–machine interfaces and digital technology interfaces that avoid the digital split between high- and low-educated citizens. In addition, remote presence, and remote connectivity to keep people connected even if they are not in the same location, trustworthy social media, serious gaming, media consumption and AR/VR will be key.

To safeguard digital inclusion, education is one of the most important research areas. Examples here are the use of AI to build personalised journeys and enhance learning outcomes, to adapt curriculum to individual student needs, digital support and nudging systems to reduce the administrative burden on teachers, tablet-based learning to improve results and decrease distress for students with dyslexia, automation of administrative tasks to free up time and resources for educational professionals, wearable devices that provide real-time support to pupils, eye-tracking solutions to adapt students’ learning experiences, and use of AR/VR to provide immersive experiences to civilians in less well-served areas.

AR may improve connectedness for remote places, reducing the need for commuting or business travel. It could also enable consumers to enjoy an event together even if they are not physically at the event.

There are still several challenges to effectively take full advantage of AI in video creation and consumption. One is the size of video data. Results are only accurate when algorithms are fed with millions of observations. Technologies therefore have to be deployed, and strategies have to be implemented to gather data at scale to harness the full power of AI techniques. However, size creates another challenge: datasets need to be manually labelled by humans to train the model, making the process expensive and cumbersome. Many new techniques are becoming available to overcome the challenge of data categorisation, such as reinforcement learning, generative adversarial networks, transfer learning and “one- shot learning” and large-language-models. In consumer-facing applications, such as marketing and recommendation algorithms, AI models may need to be refreshed continuously due to changes in the environment that drives them. Continuous updates to AI models are expensive. Other challenges relate to data management and data gathering: to create accurate results with AI, and thus value, diverse types of data have to be managed in a unified manner. This includes audience data, operational data, and content data (metadata). Also, “selection bias” (i.e. the data gathered is not representative of the population studied) has to be prevented to exclude wrong conclusions in a perfectly working model.

To facilitate collective safety, further research is required on secure access control, intrusion detection, (video) surveillance of security sensitive areas, and individual and collective activity tracking.

Secure access control as a service (ACaaS) is growing in relevance. This combines biometric readers and identity access management and can be integrated with other physical security systems (e.g. video surveillance) and building automation systems. Combined with building occupancy management systems, it can deliver valuable information on the location of staff and visitors, and in the event of an emergency to rapidly clear the building.

The recent and current military conflicts on our continent has brought new physical security requirements. In addition to regular cameras, thermal cameras as well as different sensing devices could be added at the entrance of buildings and venues to measure people’s state (e.g. temperature) and investigate if they do not carry dangerous materials as they enter premises. Physical access control, enriched with video security evidence, can provide important insights on where an infected individual has been, which doors they have used and who else may have come into contact with those doors and that individual. It can also provide these insights for more general security purposes.

More research on AI security solutions will ease the work of security operators. AI software can analyse images and audio from video surveillance live streams and recordings, and use image recognition algorithms to recognise faces, objects, and events, more than a hundred times faster than human operators. AI algorithms can also be used to carry out event detection, scene reconstruction, video tracking, object recognition, and (re)-identification, 3D pose estimation, motion estimation and image restoration. Video surveillance may be extended with freely moving cameras mounted under drones to recognise unusual behaviour in crowds from a high altitude, to monitor hazards such as fires, floods, or erupting volcanoes, and to recognise criminal faces and follow targets. Since drones are airborne, they need fast mobile and wireless communications. Low-latency broadband technologies such as 5G/6G can improve the precision and speed of their response times and enable high-speed communication to a nearby edge computing device.

Video quality should be further improved to support deep-learning algorithms, and to improve the video experience in media consumption: the spectral range and colour gamut can be extended, sensitivity must increase for low light use and especially dynamic range for better performance under all (and changing) lighting conditions as this has the greatest impact on the perception.

AI video and audio algorithms will have to be transparent and explainable. Dedicated video and audio technologies will be required to prevent and trace fake video and audio used to create misinformation in (social) media. Audio and video equipment used to create content must be able to watermark content streams to prove authenticity, and metadata should be added to the streams to prove what processing activities have taken place from image capture until display at home. Next to that equipment used for content creation based on open public infrastructures (such as internet and 5G) and with processing or control activities in the cloud must be better secured to prevent unwanted take-over by aggressors.

Required R&D&I developments within ECS

To facilitate inclusion and collective safety, specific R&D developments within ECS technology are necessary, as shown in Figure 3.6.5.

Figure 3.6.5 - Required R&D&I developments within ECS – Major Challenge 3

High-priority R&D&I areas:

• Physical infrastructure management/physical resilience and security.
• Intelligent infrastructure management (intelligent buildings, city-owned infrastructure, synergies with industry, etc.).
• Digital solutions and infrastructure for resources management, digital resilience, sustainability, e-government and citizen support.
• Resource and environment monitoring (air, water, etc.) and feedback to enable more effective management.

To further improve digital transformation towards more secure, effective, and sustainable environments, new digital tools employing IoT, unmanned vehicles/robots and AI/ML-enhanced algorithms have to be infrastructures, investments should be aimed at enhancing infrastructure coverage and quality . Also, outcomes have to be influenced through legal frameworks and by setting standards.

Intelligent buildings will require security, eco-friendship and building management. Security systems such as access control and cybersecurity were covered under Major Challenge 3, but the further development of smart lighting, air quality monitoring and control, and IoT-based real-time monitoring of electric, water and gas meters to increase the energy efficiency of buildings with the help of distributed energy systems will improve the well-being of occupants and reduce the carbon footprint of buildings. Smart technology (e.g. sensors placed around radiators, boilers, pumps, and other machinery to detect critical levels of noise, vibration, or heat) will enable facility managers to save maintenance costs by switching from a reactive to a predictive maintenance model.

Cities are very complex organisms. They combine a variety of means allowing for mobility, city infrastructure providing diverse types of media (gas, water, energy, etc.), and citizen-oriented services that increase their quality of life. It is predicted that by 2050 between 68% and 90% of the global population might live in cities, from small municipalities right up to megacities⁹. This means that, in the near future, technical means will be required to enable digital solutions for more sustainable development in cities of all size and wealth. Available technologies from tech giants such as IBM, Microsoft, Amazon, Google, and Cisco raise concerns from city managers about data privacy policies, and the very high maintenance costs caused by licence fees and the potential for vendor lock-ins¹⁰. Available open-source solutions – such as the Red Hat integration platform, which could be used in smart city applications – can also easily be acquired by large companies such as IBM¹¹ to be integrated with their company product portfolio offered commercially. This means that, in such a dynamically changing world, open-source solutions that are widely available, promoted and deployed within EU (such as FiWARE¹²) have to be developed to protect European sovereignty and values. Additionally, due to the rich industrial heritage in many EU countries, opportunities for re-using or integrating available well- developed open-source industry platforms, such as the Eclipse Arrowhead Framework¹³, can be adapted to smart city needs based on requirements gathered in EU-funded projects. This is especially the case since industry sites are often integrated within city areas, and therefore naturally create synergies that can influence each other. These smart city applications create natural synergies with the System of Systems, Mobility and Digital Industry sections.

The impact of technology on environmental sustainability is likely to be highly significant. In retail, where shifting customer habits will be key (for example, for new products such as plant- or insect-based food), IoT sensors and devices will also yield a positive impact – for example, by reducing waste through improved food temperature or expiry date management. In the manufacturing sector, smart building applications related to energy and wastewater management, as well as applications such as carbon capture and biofuel generation on industrial sites, will have a significant impact.

Required R&D&I developments within ECS

Development of supportive infrastructure and a sustainable environment within EU needs the following specific R&D developments within ECS technology:

Figure 3.6.7 - Required R&D&I developments within ECS – Major Challenge 4

• KEY ENABLING METHODOLOGIES

Key Enabling Methodologies¹⁴ support in bridging the opportunities of the Key Enabling Technologies in ECS (electronic components and systems) with the four Major Challenges for the Digital Society. Key Enabling Methodologies (KEMs) are the ‘instruments’ that direct and structure the way of working in multi-collaborative settings, give direction and realise impact with useful applications and meaningful interventions.

KEMs contribute to the integration of the technological opportunities of ECS with the knowledge from design, social sciences, and the humanities about the Digital Society.

Several categories cover the main areas of the KEMs: Vision & Imagination, Participation & Co-creation, Behaviour & Empowerment, Value Creation & Upscaling, Institutional & System Change. KEMs can support in answering questions such as how technological tools, infrastructure and training can empower citizens towards digital inclusion. How to imagine and anticipate for setbacks and build collective resilience? How to include the relevant societal stakeholders into the R&D&I developments within ECS?

• TIMELINE

The following table illustrates the roadmaps for Digital Society.