One of the biggest shortcomings of traditional video conferencing isn’t resolution, bandwidth, or even latency—it’s eye contact. Anyone who has spent time on a video call knows the subtle disconnect that happens when gaze direction is slightly off. Even high-end 2D telepresence systems struggle to convey who is actually looking at whom, weakening social cues like attention, engagement, and trust.
In a landmark SIGGRAPH 2009 paper, researchers from USC’s Institute for Creative Technologies and collaborators presented a bold solution: a one-to-many, real-time 3D teleconferencing system capable of producing accurate eye contact between a remote participant and a live audience. The result is a system that moves telepresence closer to true face-to-face communication.
Why Eye Contact Matters
Human communication relies heavily on gaze. Subtle changes in eye direction signal turn-taking, attention, and intent. Studies have shown that even a few degrees of error in gaze alignment can break the illusion of eye contact. While some 2D systems try to “fake” eye contact using camera placement or software correction, these approaches fundamentally break down in group settings—when one person looks into a camera, everyone feels looked at.
The goal of this research was to solve that problem by transmitting not just video, but true 3D facial geometry, rendered correctly for multiple viewers standing around a display.
A Life-Sized, 3D Remote Participant
At the heart of the system is a real-time 3D face scanner. Using structured light projection and a high-speed camera running at 120 Hz, the system captures the remote participant’s facial geometry at 30 frames per second. This produces a continuously updated 3D mesh and texture of the face, accurate enough to reproduce subtle expressions and gaze direction.
Rather than transmitting dozens or hundreds of camera views (as required by light-field approaches), the system sends a compact depth map and texture stream—keeping bandwidth requirements practical while still allowing the face to be rendered from many viewpoints.
The Autostereoscopic 3D Display
The scanned face is displayed on a custom autostereoscopic 3D display that provides horizontal parallax over more than 180 degrees. This allows multiple audience members to walk around the display and see the remote participant from their own correct perspective—without wearing glasses.
The display uses a rapidly spinning, anisotropic reflective surface and a high-speed projector capable of over 4,000 binary frames per second. Each audience member effectively sees a unique view of the face, creating a convincing sense of three-dimensional presence.
Solving the Projection Problem
Rendering a 3D face accurately onto a spinning, curved surface is not trivial. The researchers developed a generalized projection technique that maps 3D vertices to projector pixels while accounting for:
- The rotation angle of the display surface
- Viewer height and distance
- The anisotropic reflection properties of the mirror
- Arbitrarily curved display geometries
To achieve real-time performance, they precomputed a six-dimensional lookup table (LUT) that allows the GPU to quickly determine where each 3D point should be drawn for any given viewing configuration.
This approach is powerful because it works not just for flat mirrors, but also for concave and convex display surfaces, enabling optical designs tailored to human faces and viewing behavior.
Concave Mirrors and Focused Viewing
One of the most important design insights in the paper is the use of a concave display surface. Unlike flat or convex mirrors, a concave mirror can focus reflected light toward individual viewers. This makes it possible to render imagery optimized for one audience member at a time as the mirror spins.
Why does this matter? Because it simplifies another major challenge: vertical parallax.
Tracking the Audience for Vertical Parallax
While the display naturally provides horizontal parallax, vertical perspective is harder—especially for multiple viewers of different heights. The system solves this by tracking audience members’ faces using the same 2D video feed shown to the remote participant.
As the display rotates, the system identifies which audience member is closest to the current reflected ray and renders the face geometry using that viewer’s height and distance. The result is:
- Instant horizontal parallax (no tracking required)
- Tracked vertical parallax (updated per frame)
- Gaze angles accurate enough to fall within human perceptual thresholds
This hybrid approach takes advantage of the fact that people are more sensitive to horizontal gaze errors than vertical ones.
Does It Actually Work?
Quantitative tests showed gaze errors of approximately 3–5 degrees on the 3D display, largely limited by view sampling resolution. Informal user feedback was even more compelling: participants consistently reported the sensation that the remote person was genuinely making eye contact—especially when the remote participant shifted their gaze to address specific individuals.
That moment—when a remote face turns and looks directly at you—is something no conventional video conferencing system can truly deliver.
Looking Ahead
The authors outline several promising directions for future work, including:
- Full-color and higher grayscale fidelity displays
- Improved antialiasing for smoother 3D imagery
- Autostereoscopic displays for the remote participant’s view of the audience
- Scaling from one-to-many to many-to-many 3D teleconferencing
- Formal user studies on trust, engagement, and communication effectiveness
Used with attribution for educational and editorial purposes
Courtesy of the University of Southern California (USC) Institute for Creative Technologies (ICT) Vision & Graphics Laboratory
A Step Toward True Telepresence
This work represents a major step toward teleconferencing systems that preserve the subtle, nonverbal cues of human interaction. By combining real-time 3D scanning, advanced projection mathematics, viewer tracking, and custom display hardware, the researchers demonstrate that accurate eye contact in group telepresence is not only possible—it’s practical.
More than a technical achievement, this system points toward a future where distance no longer strips away the most human aspects of communication.
For readers who want to explore the technical details, algorithms, and experimental results in depth, you can download the full academic paper as a PDF here.