Beyond Zoom: Meetings with Holograms

Forget meeting via Zoom, Teams or Google Meet. In the future, we will speak online via 3D images, so that you can almost see your colleagues standing in front of you in real life. Pablo Cesar (TU Delft/CWI) is working on this form of communication. NEMO Kennislink visits him in his startling lab.

With the push of a button, your mother, who is on the other side of the world at that moment, appears in your own living room. No, this is not Star Trek and she was not teleported to your house. It is a new technology that scientists in Delft and Amsterdam are currently working on, so that you no longer have to make video calls, but someone appears in front of you as a hologram. Then it is almost as if your mother is really standing in front of you.

“That is useful now that we are increasingly communicating remotely during the corona crisis,” says Pablo Cesar. He is professor of human centered multimedia systems at TU Delft and heads the distributed and interactive systems group at the Center for Mathematics and Computer Science (CWI). “We communicate more and more via the computer, both with family and for work. But we all notice that looking at a screen makes us tired quickly, is complicated and does not work well. We are investigating the communication of the future, in which, for example, you can actually see each other in front of you using virtual reality glasses and thus talk to each other or play a game.”

By recording someone with multiple cameras, you can create a 3D image of that person and then send it to other people participating in a meeting.

Long corridors with suspended ceilings

Cesar is working on these new forms of communication in what appears to be a dull, gray office building. Here you don’t expect innovative scientific research that wants to shape the future. It doesn’t look very special from the inside either: long corridors with suspended ceilings and many closed doors. But exciting research is taking place behind those doors.

Cesar opens the door of one of the rooms. Here you will not find formally dressed employees in suits staring at screens. No, there are mostly boxes and multiple cameras on tripods. The boxes are made pink on one side. A colleague of Cesar walks past it and waves his arms vigorously. An image of the man appears on the computer screen. “In this way we check whether we can make good 3D images of a person,” says Cesar. He points to the screen. “We then check whether the cameras show everything correctly.” The boxes help to check whether the technology represents objects correctly from different sides.

It is one of the experiments that should enable the rendering of a person in 3D. Cesar and colleagues investigate virtually all of this communication in their lab. How users apply and experience it, but also how you send all this data in the best possible way. “The large 3D image that we make of someone produces a huge file. We still have to send that data. For example, from this room to the other side of the world. We are therefore also investigating how you can send all data smartly and then rebuild it at the recipient.”

Pack files at lightning speed

That works as follows. When we send large files to each other over the internet, they get chopped up. You can compare this with small packages that quickly travel to the other side of the globe via the digital highway. When sending, the first step is to determine exactly what you put in the packages. “If I want to talk to my family in Spain and a projection of me is shown there, they don’t need to see my back. So there’s no need to send those images.” If you do that, you send so many packets that there are congestion and therefore delays. “The most important thing is the face, so we send that in high resolution. But we don’t have to display everything so sharply. That’s why we show images that are not needed and noise is eliminated, so the files become smaller even before we send them.”

Algorithms, smart calculation programs, determine what exactly is relevant, which Cesar and his colleagues develop. Those algorithms do even more. They make the packages they send very small, this is called compression. It works much like creating a zip file on your computer, packing a large file in such a way that it takes up less space and sending it faster. The packets that matter are sent to the recipient. There they have to be unpacked and reassembled. You can somewhat compare this with a bookcase that you saw and bought in the store, which is shipped packed and then unpacked at your home. “Our algorithms also play a key role in unpacking, for example the way we compress determines the image quality and how much delay there is. For us, it is about the highest possible quality for the users.”

This should make it possible for us to communicate with each other like this in about five to ten years’ time. It is difficult to indicate what it will look like in your work or living room. In the lab it is still quite large cameras on tripods that create a 3D image of you and you need virtual reality glasses to see each other. But soon in your house it may be that you need far fewer cameras and that they are as small as a pinhead, and that a hologram appears in front of you, making glasses unnecessary.

A good example of an application of Cesar’s research is cutting a birthday cake together. The CWI, where Cesar partly works, recently celebrated its seventy-fifth birthday. There was a cake, but it was appropriately placed in a virtual space. You could enter that place via computers, cameras and virtual reality glasses. “And then enjoy the cake together. This way you meet each other in a better environment than when you just look at a screen.” There was both a real and a virtual cake.

Grandma room chock full of technology

According to Cesar, this technology is not only useful for work or to maintain contact with family and friends. It can also be used to communicate remotely with doctors. With this technology you can easily do a check-up or ask a question to the general practitioner or a medical specialist. “And it can be extended even further. For the BBC and British Telecom, we offered a second screen to supporters for the FA Cup final in England years ago. They could determine the camera angle of the images they were looking at themselves. It was also possible to watch replays. The viewer therefore decided for himself what he saw. In a pub full of supporters of the club that is playing, you can also choose to only play the sound of the away section. Or to show on an extra screen what the trainer or star player is doing. This also involves sending a lot of data, which has to be packaged and unpacked smartly. You have to display this to the recipients in near real-time. In the FA Cup experiment, we showed that this is already possible.”

How will we communicate with each other in the future? That’s what Pablo Cesar is investigating. He looks at how people use the technology, but also at the best way to send and extract files at their destination.

It is not surprising that Cesar participated in this test. He prefers to work in settings that are as realistic as possible. You notice that when you wander through his own lab. When he closes the door of the room with the boxes, he walks across. A new door opens and we are in the middle of a room that could have belonged to our grandmother. There is a Persian rug on the floor. On one wall hangs a flat screen and on the other a portrait of the king and queen. Not the space you expect in high-tech research. “Yet this is also one of our research rooms,” says Cesar.

He points to an old lamp and tells that it has a microphone in it to record what is said in the room. Cameras hang on the walls. In this way, the researchers want to map out as accurately as possible how people use new technology. “We determine, among other things, what people see on the television screen. For example family members, with whom you play a game from a distance and project it onto it. And we’re investigating how people watch television programs here. We look at how people apply the technology, because it’s not just about cleverness in how you package data, but also about how it is used.” It is a nice contrast: a room for the elderly in which tomorrow’s research takes place.

Cesar shapes the world of tomorrow with his research. He is now working on technology, which we will use in about ten years. “Over the past year and a half, we have started communicating much more online. That won’t go away. If we are allowed to go to the office more often, some of the people will continue to work from home and they will use remote communication. This also makes it easier to maintain contact with people on the other side of the world. When we communicate online in a 3D world, things go much better. It is necessary to make it better and better.”

For the seventy-fifth anniversary of the Center for Mathematics and Computer Science, there was of course cake to celebrate. But this one was cut virtually, completely in style. In this video you can see the images and a small demonstration of the technology that Cesar and colleagues are developing. The accompanying commentary is in English.

Source: Kennislink by www.nemokennislink.nl.

*The article has been translated based on the content of Kennislink by www.nemokennislink.nl. If there is any problem regarding the content, copyright, please leave a report below the article. We will try to process as quickly as possible to protect the rights of the author. Thank you very much!

*We just want readers to access information more quickly and easily with other multilingual content, instead of information only available in a certain language.

*We always respect the copyright of the content of the author and always include the original link of the source article.If the author disagrees, just leave the report below the article, the article will be edited or deleted at the request of the author. Thanks very much! Best regards!