Low Latency Displays for Augmented Reality

Опубликовано 6 сентября 2017, 21:36

The primary, long-standing goal for Augmented Reality (AR) is bringing the real and virtual together into a common space. To maintain the illusion that these two worlds coexist, however, requires that they maintain pose registration among related objects spatially and temporally consistent (i.e., objects at the same location should stay connected despite changes in user or object pose). The greatest source of registration error in this regard is from latency—the delay between when a pose changes and the display changes in response—which breaks temporal consistency. Furthermore, the real world varies greatly in brightness; ranging from bright sunlight to deep shadows. Thus, a compelling AR system must also support High-Dynamic Range (HDR) to maintain its virtual objects’ appearance both spatially and temporally consistent with the real world. This presentation demonstrates new methods for low latency displays, primarily in the context of Optical See-through, AR, Head-Mounted Displays, which has the least tolerance for latency in displays, focusing on temporal consistency in registration, HDR color support, and spatial and temporal consistency in brightness: 1. For registration temporal consistency, the primary insight is breaking the conventional display paradigm: computers render imagery, frame by frame, and then transmit it to the display for emission. Instead, the display must also contribute towards rendering by performing a post-rendering, post-transmission warp of the computer-supplied imagery in the display hardware. By compensating for latency in the processing and transmission of the imagery using the latest tracking information in the display, much of the system latency can be short-circuited. Furthermore, the low latency display must support ultra-high frequency (multiple kHz) updates in order to take full advantage of the imagery updates. 2. For HDR color support in low latency displays, the primary insight is the development of new display modulation techniques. Ultra-high frequency displays, like DMDs, generally emit binary output, which require modulation to produce analog values. Conventional modulation would break the low latency guarantees, and modulation of bright LEDs illuminators at frequencies to support HDR at ultra-high display frequencies exceeds their capabilities; thus one must directly synthesize the necessary variation in brightness. 3. For brightness spatial and temporal consistency, the primary insight is integrating HDR light sensors into the display hardware: the same processes which both compensate for latency induced errors and generate the HDR output can also modify it in response to the spatially sensed brightness of the real world. This presentation contains implementations, results (both visual and performance), and future steps towards improving the methods above. 

See more on this video at microsoft.com/en-us/research/v...