Enhanced Reality Manifesto

Enhanced Reality Manifesto Joe Zott 13 Aug 2022 - Joe Zott

The immersive experience market continues to evolve with the search for revenue and as the market grows. As the different participating companies search for differentiation and sales, they have come up with a variety of terms to describe the market that they are addressing

VR seeks to create an entirely immersive experience with everything visualized is synthetic.

Augmented reality

AR is the oldest term for integrating reality and augmentation. This term is still being used by Apple and has become the common term to refer to presentation of augmented information on a phone or pad display panel.

eXtended Reality

XR is an emerging umbrella term for all the immersive technologies.

Mixed reality

MR has been popularized by Microsoft for their augmented headsets. Their justification for this is that MR not just overlays but anchors virtual objects to the real world. As AR has evolved it is also similarly providing more realistic integration of the virtual objects and the real-world.

Spatial computing

Many of the applications rely on a spatial or geographic context. The term spatial computing is being used for the describe the presentation of information in real time using an immersive experience device as opposed to the traditional presentation of information such as provided by a GIS system (Geographic Information System). This term is being popularized as Magic Leap.

Enhanced Reality - ER

All the existing market definitions don’t adequately address where the virtual objects come from or address objects and information extracted from the real-world.

Enhance Reality seeks to provide improved information about the world around you. This is distinguished from:

ER combines enhanced sensing, processing and immersive visualization.

Enhanced sensing

  1. Optical sensing
  2. Non-visible wavelengths
  3. Low light
  4. Distance - LIDAR
  5. RF based sensing
  6. Radar
  7. Synthetic aperture radar (SAR)
  8. Ground-penetrating radar (GPR)
  9. Spectral mapping

Processing

Processing latency

The main characteristic of a real-time imaging system is the need for deadline satisfaction.

For a realistic immersive experience processing and end-to-end latency is critical. While less is better, a good rule of thumb is that the latency needs to be less than 15msec to achieve cognitive alignment between the virtual and real worlds.

ISO/IEC 62366-2 includes the notion of a perception-cognitive-action (PCA) error assessment as shown in the following figure. The medical device field has found that individual performance action is influenced by the impact that a product makes on the perception and cognition that the product experience creates.

Processing patterns

  1. Enhance the sensor dataz
  2. Data fusion of multiple sensors
  3. Combing sensor data with known data

Cloud vs. local processing

The opportunity and benefits that cloud computing (processing as well as visualization rendering) provides needs to be balance by the latency impact. The network delay contribution may for certain applications dominate the cloud processing delay.

Visualization

All 3 immersive visualization modalities (screen, headset, and projection) can be used for ER.

Field of view

Like the VR/AR situations the ER FOV needs to match the dimensions of the enhanced sensed object in the reality scene.

Operating range

One important attribute of an ER product is that the sensed object needs to be visualized at the appropriate distance. If the object is physically 18” away from the observer, then the ER version of the object needs to be at that same distance.

Physical/virtual position accuracy

For many ER applications one of the most important performance attributes is alignment between the physical and virtual world.

Deliberate change of size

For some ER applications it might be optimal to change the visualized size of the sensed object. This could be a situation where an object is far away, such as a plane, and making the object appear larger may be best or the object could be very small, such as a case of something being viewed by a microscope.

Application Areas

Medical

References

A Neurocognitive Approach to Developing Safer Medical Devices, AAMI BIT January / February 2020

ANSI/AAMI/IEC 62366-1:2015; Medical devices – Part 1: Application of usability engineering to medical devices

AAMI/IEC TIR 62366-2:2016; Medical devices – Part 2: Guidance on the application of usability engineering to medical devices.

Optical Architectures for Augmented-, Virtual-, and Mixed-Reality Headsets, Bernard Kress