5 years of secret research and development! Google's 3D 'Magic Mirror' test, no need for 'facehuggers' when chatting with real people

Can I only rely on a head-mounted display to enter the metaverse?

No. Let our imaginations run wild a little more.

The giant Google has given another solution: instead of using a head-mounted display, use a 3D “phone booth”!

#If you are used to skype or zoom, you will definitely be surprised when you see starline for the first time.

Google officially announced the Project Starline project last year. It is a video calling booth that uses 3D imagery, high-resolution cameras, a custom depth sensor and a breakthrough light field display.

#This creates an extremely realistic experience for callers on both sides of the screen.

#After a year, Google’s holographic video chat project Project Starline will install prototypes in partner offices later this year to complete initial testing.

Naked-eye 3D

Starline is one of the two chess pieces laid out by Google in the metaverse (the other is the AR glasses Iris).

# Google has been brewing this project for more than five years.

#At the 2021 I/O conference, Google officially announced Project Starline, a super-high-resolution video chat room.

In the past, traditional video chat software such as skype or zoom could not overcome the sensory distance. The person you are chatting with appears opposite you in a 2D plane. And his voice was obviously coming from a mobile phone, tablet or computer.

#This kind of "separation of sound and picture" experience is not real.

Is it possible to completely eliminate this "sense of distance"?

Google asked the question, "Can Zoom be turned into a giant sit-down arcade?" They succeeded.

# After five years, Google developed a black technology - Starline 3D video calling technology.

#This technology has two major highlights: one is real, and the other is that there is no need to wear any equipment.

A 65-inch light field display, plus a dozen pre-arranged sensors and cameras, constitute such a chat room.

Looks like the effect is quite real.

Such a real appearance of a character is captured by the camera and sensor and displayed directly on the other party's display ?

Not really. The role of each camera and sensor is to capture portraits from different angles, use deep learning to perform real-time compression, transmit these data to the opposite side, and reconstruct a 3D avatar on the display screen.

In other words, in this phone booth, what you see is "Reconstruction" of portraits.

#This kind of real-time reconstruction of character images makes each other’s feelings unparalleledly real. It's as if the person you're chatting with is actually sitting across from you.

The transmission of these collected data, like ordinary video conferencing, is based on WebRTC, and Google The unique compression algorithm makes it possible to transmit 3D images in real time in both directions.

Transmitting 3D avatars over a video chat connection allows Google to solve the eye contact issue. In regular video chats, this has always been a problem. In the past, webcams on the monitor made it impossible to make eye contact when looking at the monitor, but 3D avatars can bridge the disconnect between the center of the camera and the center of the monitor, allowing eye contact between the two parties.

Google is processing this data in a powerful dual Xeon workstation with four Nvidia GPUs (two Quadro RTX 6000 and two Titan RTX).

Another great feature of Starline is that it does not require any equipment to be worn.

Yes, you heard it right, VR headsets and smart glasses are not required!

#Even, in the concept map previously released by Google, the camera and sensor are placed in a table.

#After a little decoration, you will even have the illusion that your desk is in front of you, and the person you are chatting with is sitting opposite the desk.

According to the technical diagram provided by Google, this set of equipment includes: a display, camera, speaker, microphone , illuminator and display unit of the computer, and a backlight unit containing an infrared backlight and used as a bench seat. Both units contain white LED strips angled toward the walls and ceiling to create soft, reflected lighting.

The display is a 65-inch, 8 K, 60 Hz autostereoscopic lens panel on which a life-size naked-eye 3D view avatar is generated.

The capture subsystem consists of three synchronized three-dimensional RGBD capture pods: two above the monitor and one on the "middle wall" below the monitor. wall).

#The pod further down includes an additional color camera for zooming in on the subject's face. Additionally, two of the four monochrome tracking cameras are above the display and the remaining two, one on each side, are used to capture high-speed, wide-angle images of the eyes, ears, and mouth.

The rendering method is as shown in the figure below.

The pod is the RGBD capture device mentioned above.

#The effects of characters rendered in this way are more realistic and lifelike.

Starline is also equipped with an advanced sound system.

This system uses talker-tracked beamforming, reverberation reduction, WebRTC transmission, talker/listener-tracked Technologies such as virtual audio synthesis, binaural crosstalk elimination frequency division combination, amplitude translation, spatialized audio and dynamic beamforming.

#With the blessing of these technologies, it is as if the two people on the call are talking face to face.

#Last year, Google engineers said the device would work on a standard office network. It seems that Starline technology is expected to be widely promoted.

#At the time, Google did not disclose further details about the price.

#It’s hard to imagine the market for a six-digit phone booth that measures 7 by 7 feet and is as big as a bathroom, but Google is pushing ahead with more tests.

Don’t be a “facehugger”

How is the progress of this project so far?

At Google Cloud Next 2022, Google CEO Sundar Pichai introduced some progress made in the holographic video chat project (Project Starline).

Google said it will conduct regular testing at select partner offices, including Salesforce, WeWork, T-Mobile and Hackensack Meridian Health.

WeWork and Salesforce said they were interested in trying the technology, arguing it could make connections between people more meaningful. .

To date, Project Starline has undergone thousands of hours of testing within Google and is used daily in its U.S. offices.

Google said in a statement: "Today, across the United States, Prototypes of the Starline project can be found in all Google offices, and employees are using the technology for meetings every day."

When this project was pushed to actual testing, You can understand how this technology will be used in the real world and what challenges it may face.

Many people think it is very cool, but some people question whether the Starline project can eventually become a part of the daily life of working people.

Facing the vast enterprise conference equipment market, Starline does have some space. But some limitations make Starline difficult to use for serious meetings. Enterprise conferencing equipment is often suitable for large groups of people sitting around a table, and broad compatibility allows anyone to start a meeting using almost any hardware.

## However, Starline only supports one-on-one chat with one person. Can it be used to make presidential calls?

#Recently, Google invited a reporter to conduct an experience test on Starline. The reporter described it as follows:

"The whole process felt much more natural than a Zoom call. There was no obvious delay in the other party's movements and our conversation, so It just feels like talking. No weird audio or visual delays."

"I will believe that I'm really with someone because I'm Make real eye contact with a full-size virtual human instead of squinting into a tiny Zoom window."

Part of what makes Starline so convincing is that you're not just looking at the screen, but you're looking at a series of lenses, or lens arrays, in front of the screen. The principle is similar to holographic cards, which can display different images or 3D effects when you shake them back and forth. Starline tracks your eyes to know where to point the images it shows you.

Finally, the reporter concluded: "It's useful for one-on-one conversations, especially in a doctor's office or where you need to serve customers."

#The long-term status of the Starline project at Google has always been unknown.

A year ago, the project was reorganized, with Google integrating its AR/VR technology and internal R&D teams Area 120 is reorganized into the "Labs" team.

#Last month, Google also cut the number of Area 120 personnel in half.

#This suggests that Google may not consider these types of experiments a priority in the current economic climate. Even some internal Google employees aren't sure Project Starline still exists.

# Even so, Starline is still exciting and a bet on a different kind of virtual reality.

People do not need Avatar, they can connect in the virtual world only through their real selves.

#A few days ago, Meta also launched its most expensive high-end headset, Quest Pro.

And Google has not developed technology that uses cameras to track eye and facial movements, as Meta does now.

# also did not study how to "add legs" to your body in virtual reality, but focused on presenting a real person.

The most important thing is that you don’t have to wear it on your face like a “facehugger”, adding to the burden.

#At the same time, as more and more businesses try to figure out a hybrid future of work, Starline can bridge the gap in today’s 2D video chat. .

2024, bringing the first AR head-mounted display to the market

In fact, Google is not far behind in terms of AR head-mounted display layout.

At the beginning of this year, two people familiar with the matter revealed that Google has begun to step up the development of AR headsets, which are internally code-named Project Iris and hope to be launched in 2024.

#And Google hopes to release Starline with Iris by 2024.

The latest Iris head display will use Google’s latest processor, a fully self-developed Tensor chip, and also supports running on Android systems.

#As for a new operating system, it is also in Google’s future research and development plans.

This AR device uses outward-facing cameras to blend computational graphics with the real world, creating a more immersive experience than existing AR glasses like Snap and Magic Leap mixed reality experience.

Due to power constraints, Google will use its data center to remotely render some graphics and transmit them to the AR glasses through the Internet connection.

Project Iris can be said to be a turning point in the development of Google AR glasses.

As early as 2012, Google released a prototype of AR glasses. Similar to a pair of ski goggles, but does not require connection to an external power source.

#With a wave of hardware and software upgrades, it successfully landed in the US market in 2014, priced at US$1,500. However, in 2015, it was forced to be removed from the shelves due to various negative news.

Then, in 2015, Google released a “VR glasses box” at its I/O conference. The device is very simple and mainly consists of a piece of cardboard and two special plastic lenses.

#In 2017, Google decided to shift the application of AR glasses to industry.

#Despite its rush to catch up, Google appears to be lagging behind its peers in building new forms of AR hardware.

Since then, Google has been noticeably silent on the hardware development of AR glasses, and has chosen to focus on the development of AR in Google Lens, a visual search engine, and Google Maps. Related software functions.

With the explosion of the Metaverse concept in 2021, many major manufacturers have launched their layout on this track.

#In fact, Google is also quietly playing its own game. Whether it’s the AR headset Iris or the holographic video calling project Starline.

Xiao Zha once said in an interview: "The magic of VR is that it can immediately make you believe that you are in another place and are with the people there. Together." His idea was a hologram, and Starline happened to have the same idea as the hero.

Now, two of the world’s largest tech companies are working hard to improve avatars, albeit in different ways: Meta is going all-in on the idea of ​​a metaverse, And Google is trying to take over the business.

Chat rooms and expensive mixed reality displays are neither practical nor ideal for most people, but something that really takes off, could happen in Somewhere in between.

​参考资料：

https://techcrunch.com/2022/10/13/googles-3d-video-calling-booths-project-starline-will-now-be-tested-in-the-real-world/

https://www.theverge.com/2022/10/13/23400870/google-project-starline-3d-chat-booth-impressions

The above is the detailed content of 5 years of secret research and development! Google's 3D 'Magic Mirror' test, no need for 'facehuggers' when chatting with real people. For more information, please follow other related articles on the PHP Chinese website!