While reading and watching Apple Vision Pro reviews to prepare for a previous blog post, I was reminded of a technology term that I had heard for the first time back in June 2023 at WWDC (Apple’s Worldwide Developer Conference): Dynamically Foveated Rendering (Apple Developer, 2024). Fast-forward 8 months later, and I now realize that I am experiencing this technology everyday. Best of all, I had no idea that this technology was happening—except for one giveaway that I will discuss later—a fact that means the tech is functioning perfectly.
In his Apple Vision Pro review video, "Apple Vision Pro Review: Tomorrow's Ideas... Today's Tech!," Marques Brownlee gave an excellent explanation of foveated rendering. In part, he said, “It combines the insanely fast eye tracking with what's called foveated rendering, meaning, it's only actually rendering in high resolution exactly what you're looking at when you're looking at it.” He then pointed out that this kind of rendering depends upon the ultrafast processing power of the Apple Vision Pro to be able to track your eye movement and nearly instantly deliver a perfectly crisp focus for your field of vision exactly where you are looking, while the area in the periphery remains less focused.
The idea is that foveated rendering can greatly improve power efficiency of the device because the Apple Vision Pro is “aware” of where you are looking, and does not needlessly use resources to render the entire 8K screen all the time.
Brownlee also noted that this effect is very apparent when taking screen captures with the Apple Vision Pro. Until he mentioned this, I had been somewhat frustrated by the highly variable quality of my screen caps. However, I now understand why screenshots are only focused in one place—where I was looking at the time. The following low-light screen capture illustrates this effect: I was watching Brownlee’s YouTube video and looked to the lower-right corner to resize the window when I made the capture. The window-resize corner tool is in perfect focus, but the focus radiates outward from that point and gets worse. To my eyes, everything I was viewing at the time appeared perfectly in focus, but seeing my full field of view in this screen capture reveals otherwise.
The concept of this technology is based upon human vision. A research paper from 2016 succinctly describes human vision as it relates to this technology. “Humans have two distinct vision systems: foveal and peripheral vision. Foveal vision is sharp and detailed, while peripheral vision lacks fidelity. The difference in characteristics of the two systems enable recently popular foveated rendering systems, which seek to increase rendering performance by lowering image quality in the periphery” (Patney et al, 2016).
Since these kinds of topics endlessly fascinate me, I started to delve into the technology. My first stop was a short article from 2016 about graphics-leader Nvidia. Paul Miller of The Verge wrote, “[this technology] combines the insanely fast eye tracking with what's called foveated rendering, meaning, it's only actually rendering in high resolution exactly what you're looking at when you're looking at it.” He goes on to say (in 2016), “Of course, this only works if you know where someone is looking, and none of the major VR headsets currently available do this.” The article also references a couple of then-new eye-tracking sensors along with Nvidia’s “foveation technique that can blur the periphery of an image while still maintaining the parts that humans perceive at the edges of their vision — color, contrast, edges and motion.”
Another source from 2016, the website Digital Trends, reported that, “according to Nvidia, foveated rendering systems have been in use for around 20 years” (Parrish, 2016). This got me wondering, how far back does this technology go? So I dug deeper and found studies related to this technology that are now nearly 30 years old.
I first found a wave a published papers from around 2003. The first, Foveated 3D model simplification (Cheng, I., 2003), was presented at the IEEE Xplore conference related to Signal Processing and Its Applications. The author explained a technique for “enhancements to 3D model simplification based on interactive level-of-detail update with foveation.” However, the earliest description of this technology I could find was from 1996, by Ohshima, Yamamoto, & Tamura. The Abstract of their paper does not specifically use the term “foveated,” but the description of the technique is nearly identical: “This...new method of rendering for interaction with 3D virtual space [uses] geometric models of graphic objects...constructed prior to the rendering process. The rendering process first calculates the visual acuity... Second, the process selects a level from the set of hierarchical geometric models depending on the value of visual acuity...a simpler level of detail is selected where the visual acuity is lower, and a more complicated level is used where it is higher.”
I also found an article about Apple’s patent on this technology. Patently Apple (2023) reported, “During Apple's WWDC23 introduction to their revolutionary Vision Pro Spatial Computing headset, Mike Rockwell, VP, Technology Development Group, came on stage to describe the key components behind the new device.” Rockwell described: “The rendering process can provide foveated rendering by drawing regions based on gaze direction (e.g., drawing only some regions, drawing some regions at higher resolution/higher frame rate, etc.). In some implementations, the rendering process provides an increase in quality in a focus region and/or reduces the amount of computation and memory used in providing perspective correct rendering” (Patently Apple, 2023).
To me, this is an excellent example of an innovative idea being conceived as a theory, moving into an early research and development about 10 years later, advancing again about 10 years after that, and finally reaching a stage that the original 30-year-old idea can be realized with a combination of current technology in a shipped product at a high implementation level. Granted, this technology is not fully “there” yet—it takes a $3,500 ski-goggle-sized headset to deliver—but the processing power, imaging, power consumption, and display technology is perfectly demonstrated on the Apple Vision Pro right now in 2024—by the imperfect screen capture of the technology in practice.
References
Apple Developer. (2024). Discover visionOS: Unity. Retrieved from https://developer.apple.com/visionos/
Brownlee, M. (February 3, 2024). Apple Vision Pro Review: Tomorrow's Ideas... Today's Tech! [Video]. Retrieved from www.youtube.com/watch?v=86Gy035z_KA
Cheng, I. (August 2003). Foveated 3D model simplification. Conference: Signal Processing and Its Applications, 2003. Proceedings. Seventh International Symposium. Retrieved from www.researchgate.net/publication/4030859_Foveated_3D_model_simplification
Miller, P. (July 22, 2016). Nvidia's foveated rendering tricks for VR could improve graphics and immersion. The Verge. Retrieved from www.theverge.com/2016/7/22/12260430/nvidia-foveated-rendering-vr-graphics-smi-eye-tracking-siggraph
Ohshima, T., Yamamoto, H., and Tamura, H. (April 1996). Gaze-directed adaptive rendering for interacting with virtual space. 1996 Virtual Reality Annual International Symposium (VRAIS 96), IEEE Computer Society. Retrieved from https://ieeexplore.ieee.org/document/490517
Parrish, K. (July 22, 2016). Nvidia plans to prove that new method improves image quality in virtual reality. Retrieved from www.digitaltrends.com/computing/nvidia-research-foveated-rendering-vr-smi/
Patently Apple. (2023). A new Apple patent describes Perspective Correct Vector Graphics with Foveated Rendering for Vision Pro, iPhone & more. Retrieved from www.patentlyapple.com/2023/06/a-new-apple-patent-describes-perspective-correct-vector-graphics-with-foveated-rendering-for-vision-pro-iphone-more.html
Patney, A., Kim, J., Salvi, M., Kaplanyan, A., Wyman, C., Benty, N., Lefohn, A., & Luebke, D. (2016). Perceptually-Based Foveated Virtual Reality. NVIDIA, SIGGRAPH 2016 Emerging Technologies. Retrieved from https://research.nvidia.com/sites/default/files/pubs/2016-07_Perceptually-Based-Foveated-Virtual/foveated-sig16-etech.pdf
No comments:
Post a Comment