During I/O 2023 last week, Google announced a new version of Project Starline and allowed more people to try it. We got a chance to experience the immersive video conferencing technology, while MKBHD was able to record a good approximation of what it’s like to use Project Starline.
Original vs. new Project Starline
The first Project Starline prototypes were housed in what was essentially a booth with two main components. The “Display Unit” included a 65-inch 8K (60Hz) panel, speakers, microphones, illuminators, and computer, as well as several sets of cameras:
- Four high frame-rate (120FPS) face tracking cameras to “estimate the 3D location of the eyes, ears, and mouth within about five millimeters of precision.”
- Three groups of camera pods for 3D video. Each contains two infrared and one color camera, while the bottom pod contains another color camera zoomed in on the face.
The “Backlight Unit” consisted of the bench and infrared backlighting. Four NVIDIA GPUs (two Quadro RTX 6000 and two Titan RTX) were used to compress four color and three depth streams. Everything was transmitted over WebRTC with Project Starline combining Google’s work in 3D imaging, real-time compression, spatial audio, and light field displays.
That setup is obviously expensive and technically intensive, with Google noting last week how the “size and complexity of the system made it challenging to bring to many of today’s offices.”
The new Project Starline prototype is more like a digital whiteboard in size with a depth guard (of sorts) in front of the screen, as well as a set table. Notably, only a “few standard cameras” arranged at the top, left, and right of the display and AI are used to create a depth map.
Project Starline hands-on
Since Project Starline was announced at I/O 2021, people have raved about how life-like and three-dimensional the people in front of them look. Google has slowly expanded access to third-party companies since then.
In using Starline last week, there’s definitely a sense of depth that’s unexpected. The person in front of you is not flat and I immediately peered to the left and right of them to check.
However, I wouldn’t say they are fully 3D either. There’s a limit to their depth that, to me, was obvious after the initial surprise and wonder. It felt like the person I was looking at was a 1- to 2-inch thick cardboard cutout rather than a full head and body. It was same when my demo partner held out an apple in front of me.
For whatever reason, the 3D effect was much better when I removed my glasses. With them on, the 3D effect felt localized to my frames and somewhat akin to watching a 3D movie in a fancy theater setup.
Meanwhile, I wish that the 3D person was much sharer. I could see pixels in the same way I find the 1800 × 1920 per-eye resolution of the Meta Quest Pro to not be sharp enough.
Given my brief usage, I see Starline as a plus/add-on to the video calling experience. There’s a novelty to adding some amount of real depth that I could see wanting to always have over time.
While the person in front of me had depth, one of the things I found most surprising was watching the transition between the edge of the Starline screen and the wall behind the display. The Starline window genuinely felt like a portal to another world that I literally wanted to step through. I found that tremendously amusing.
MKBHD’s Starline demo
Given the tracking involved, Starline is very hard to demo virtually. You need to experience it in-person to get the full effect. However, Marques Brownlee and team had the idea to replicate the experience in a video:
You see, basically once you first sit down in the Project Starline booth, this system has to identify where your face is and then you can move around and look at stuff with head and body tracking. And of course that’s not gonna work if you just put a camera in there. But if you do show it a face and a camera at the same time, like say maybe a cardboard face where you can stick a camera lens somehow through that cardboard face, then it would track and you could see what I’m talking about.
FTC: We use income earning auto affiliate links. More.
Comments