Creating a reality beyond the real

girl video conference with family

This article was originally published to the Nokia Bell Labs blog by Markus Hofmann on March 25, 2019.

There is something missing in the way that many are looking at what’s next for human communication.

One may argue that there has always been something missing.

Humans, as inherently social beings, have always cherished interacting with each other and with our communities to learn, exchange information and share experiences. We have traditionally done this by gathering around campfires, over dinner, at local events, in joint activities and in cheering for our favorite teams together. But our world expanded and dispersed and we have started to miss these essential social interactions among friends and families.

Audio and visual communication

Advances in computing and networking came into play. Technology allowed us to socialize over great distances. I still remember moving to the U.S. about 20 years ago, leaving my family in Germany behind, and the telephone was our primary means of staying connected.  But a one-minute phone call cost $1.25, meaning a simple audio-only interaction cost a fortune.

Humans, as inherently social beings, have always cherished interacting with each other and with our communities to learn, exchange information and share experiences.

What about today? Want to interact with your friend or a relative on the other side of the world? Just start up FaceTime or Skype, pick up the phone, send an email, Snapchat away, Instagram your latest photo, or – for the ‘older’ set – post an album on Facebook.  Now my family puts an iPad on the breakfast table, we open our favorite video conferencing application, and Grandma and Grandpa join us from Germany for breakfast in New Jersey.

Or do they really join us? Yes, we see their faces on the iPad that prominently fills an empty chair at the breakfast table, but the display shows Grandma and Grandpa in their environment. In their home, not in ours. They do not join us for breakfast in our home. They are not part of our immediate surroundings. We are not sharing a common experience.

We “intrude” into their home and we stare into their living space to interact with them. They cannot feel the heat wave that makes us sweat on a hot summer day. Or the humidity of the Jersey Shore in summer. They do not feel the truck passing by that makes our house vibrate. Or smell the beautiful blossoms on the plum tree in our backyard. Instead, they are stuck in their own environment, merely peeking into our world through a tiny window that defines the current small, flatscreen world, providing them with a very limited visual and audio snapshot.

We are clearly missing something

Sure, we are connecting…but we are not experiencing together. Today’s communications and interactive technology is like a narrow tube, almost like blinders on a horse, that allows us a limited view into each other’s lives. We cannot truly share the multifaceted experiences that drive our feelings and emotions until we embed in each other’s full virtual environments.

What if there were technologies that could break through the limitations of these purely visual and audible communications that have defined us in the last few years?

At Nokia Bell Labs, we believe that the future of communications is not about going from HDTV to 4K to 16K to 360 videos. It is also not about replicating the real world in ever greater quality and detail. It is about adding the missing modalities to our remote, disjointed, digital interactions.

Think about haptic feedback as one example. Simple ‘touch’ can convey deep emotions and feelings. The future has to be about breaking through the barrier of realities, not just replacing them. It has to be about going beyond reality to augment our lives and our abilities for the better.

Think about being able to really feel how your aging parents are doing from a distance? How about being able to see, hear and experience the emotions of an autistic child? Or feeling the slap of a high-five with your friend who lives on the other side of the world, through a hologram?

And this is not just for play. Humans like to experience the same in their workplace. How about seeing and feeling the vibrations of the engine that you are trying to repair remotely? Or a doctor sensing a twisted arm of an injured athlete? We could augment reality to gain a much better understanding of each other and of our environment.

We need to recover these experiences by capturing, transmitting and recovering physical, physiological and psychological human signals over digital distances.

These digital representations will then need a super high-performance, low latency network to transmit and share such rich datasets instantaneously. The good news is, that is what end-to-end 5G networks are being built and defined to do.

At Nokia Bell Labs, teams of researchers and engineers are working hard to get us closer to such reality. To a reality that goes beyond what is considered real, or possible, today. A reality that blurs the lines between physical and digital worlds, making them indistinguishable. Breakthroughs in capture technology will allow us to create life-like digital representations of objects and of people at the snap of our fingers, in easy and convenient ways.

These digital representations will then need a super high-performance, low latency network to transmit and share such rich datasets instantaneously. The good news is, that is what end-to-end 5G networks are being built and defined to do.

New playback methods and devices then embed these digital representations into our physical world in a way that makes them nearly indistinguishable from real objects. Today’s HoloLens and Magic Leap glasses are just the beginning, paving the way to new projection technologies that will provide ubiquitous mixed realities. These mixed realities will be assembled with more than visual and audible elements. Millions of sensors in our environment, together with information obtained from wearables and even in-body biosensors, will provide much deeper context and situational awareness to create the most helpful, most sensitive, and most privacy-preserving mixed realities.

A child or an elderly parent living far away, or in a remote location not feeling well today? Their life-like holographic representation on the sofa in your living room will make this obvious through appropriate facial expressions and a certain sadness in their tone of voice.

The operator of a vehicle on a road, or of a crane moving heavy materials at the construction zone getting tired? The system will recognize this instantaneously and prevent accidents by alerting the tired operator and taking appropriate actions.

So, what’s missing today?

Getting close to creating this digital utopia

New capture and playback devices are being worked on with novel devices and sensors.

Low-latency, high bandwidth connectivity to millions of mobile devices with limited battery life are getting there with the global rollout of 5G and IoT connectivity solutions. Edge cloud is beginning to be deployed to make “better than being there” contextualized, personalized services a real possibility.

It is obvious we need AI/ ML to infer never-known-before situational awareness. This is where advanced analytics platforms and stream processing solutions become essential, built on new approaches to software creation, execution, and management. Software will no longer run on a limited set of well-defined hardware platforms, under well-defined scenarios. Instead, software will instantaneously adapt to and execute on never-seen-before devices and sensors that come and go at any point in time. This requires a fundamental re-design of how software is built with new AI/ML platforms and methods.

We are creating a reality that goes far beyond what is considered real in today’s “small flat-screen world”, to create a world as big as our human imaginations and expectations.

About Fast Future

It is our mission to explore the implications of emerging technologies, seeking answers to next-level questions about how they will affect society, business, politics and the environment of tomorrow.

We aim to inform and inspire through thoughtful research, responsible reporting, and clear, unbiased writing, and to create a platform for a diverse group of innovators to bring multiple perspectives.

Fast Future is building the media that connects the conversation.

More
family checkup doctor
AI could diagnose your illnesses