Google Daydream testing AR, 6DoF controllers w/ ML, and mobile Android apps in VR

Abner Li | Sep 21 2018 - 11:11 am PT

Looking towards the future, augmented reality will be a very important technology once it gets miniaturized into wearable glasses. Till then, apps and best practices are being developed through phone-based AR and VR headsets. Google today announced experimental features for standalone Daydream including a “See-Through Mode” and 6DoF Controllers.

This experimental functionality is aimed at standalone Daydream, and specifically the Lenovo Mirage Solo. “See-Through Mode” allows you to “see what’s right in front of you in the physical world while you’re wearing your VR headset.”

Pass-through modes exist on other platforms, but Google’s takes advantage of its WorldSense technology for low-latency tracking and depth perception. Google touts the quality as being “good enough to let you play ping-pong with your headset on.”

However, the real promise of this technology is augmented reality and overlaying digital objects into the physical world.

The combination of see-through mode and the Mirage Solo’s tracking technology also opens up the door for developers to blend the digital and physical worlds in new ways by building Augmented Reality (AR) prototypes.

One experimental app allows users to overlay furniture into the world by using 6DoF controllers that allow for more natural and intuitive interaction with virtual objects.

With 6DoF tracking, you can move your hands more naturally in VR, just like you would in the physical world. To date, this type of experience has been limited to expensive PC-based VR with external tracking.

This six degrees of freedom on Google-created controllers use a “unique optical tracking system” that heavily relies on machine learning rather than hardware. It allows for a self-contained package, fitting with standalone Daydream. Google has a sign-up page for developers that want the 6DoF controllers, which includes an attachable faceplate for Mirage Solo.

Instead of using expensive external cameras and sensors that have to be carefully calibrated, our system uses machine learning and off-the-shelf parts to accurately estimate the 3D position and orientation of the controllers. We’re excited about this approach because it can reduce the need for expensive hardware and make 6DoF experiences more accessible to more people.”

Lastly, Google is letting developers open any Android app in Daydream VR. Users will have an artificial background and can open an app as a floating window.

With support for Android Apps in VR, developers will be able to add Daydream VR support to their existing 2D applications without having to start from scratch. The Chrome team re-used the existing 2D interfaces for Chrome Browser Sync, settings and more to provide a feature-rich browsing experience in Daydream.