GlintMarkers: Spatial Perception on XR Eyewear using Corneal Reflections
Seungjoo Lee, Vimal Mollyn, Chris Harrison, Justin Chan, Mayank Goel
TLDR
GlintMarkers enables gaze-driven spatial perception on XR eyewear by using corneal reflections and retroreflective markers for object tracking.
Key contributions
- First system for gaze-driven spatial perception using inward-facing cameras on XR eyewear.
- Utilizes the cornea as a mirror to encode gaze and environmental information from reflections.
- Introduces passive retroreflective markers that create bright glint patterns on the cornea.
- Develops a custom PnP framework for corneal imaging to estimate object orientation and distance.
Why it matters
This paper introduces a novel method for XR devices to understand their environment using existing inward-facing cameras. By leveraging corneal reflections and custom markers, it enables precise object tracking and identification without adding external sensors. This could significantly enhance XR experiences by allowing more intuitive interaction with the real world.
Original Abstract
We present GlintMarkers, the first system to perform gaze-driven spatial perception using the inward-facing cameras on XR eyewear. Our key observation is that the cornea acts as a mirror that encodes both gaze direction and visual information about the environment in a small, low-contrast reflection. To extract spatial and semantic information from this reflection despite the camera's limited pixel budget, we present a passive retroreflective marker design that concentrates reflected near-infrared light onto the cornea, producing bright glint patterns. We develop a custom Perspective-n-Point (PnP) estimation framework adapted to corneal imaging and perform orientation and distance estimation of tagged objects, as well as unique object identification.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.