Experience the digital world with all the senses

June 19th, 2019 | Send a digital lollipop or delicate fragrance notes via email or let the wind virtually blow against your face - research makes it possible.

Thomas Heinrich Musiolik is ambassador of Brain City Berlin and doctoral student at Universität der Künste Berlin, where he researches the field "Experience in Digital Media". He is also Visiting Doctoral Researcher at the Imagineering Institute (Malaysia), focussing on topics such as "Artificial Intelligence“, "Robotic“ and"Multisensory Experience“. He also works as Co Editor-in-Chief at the journal “Transactions on Leading-edge Synergistic Robots“ and writes books and articles in the field of marketing & communications.

Imagine if you could set your newspaper down now and travel around the world. Taste the freshly brewed coffee in Italy, shop in every store in the world, enjoy the sunset in Venice with your sweetheart, and finally jump naked into the sea in the Maldives. And all this from the comfort of your own home. That is now possible!

Together with my colleagues, Prof. Dr. Adrian David Cheok, Director of the Imagineering Institute (Malaysia), I’m exploring future technologies for a multisensory experience in digital media (Musiolik/Karunanayaka/Cheok 2019).

The scent of roses via SMS

Some time ago we were able to develop Scentee, a small device that you can attach to your smartphone. For example, when you receive an email, you will be notified by scent, or you can send scent messages directly to your friends. Scentee consists of a base unit that connects to the smartphone or tablet via the headphone jack. This has a built-in LED ring that lights up with notifications in different colors. The scent is provided by a fragrance cartridge, which is inserted into the base unit.

At its launch in 2011, Scentee already had five different types of scents: Rose, strawberry, coffee, lavender, and rosemary. Now, there are different versions of such “smell-o-phones” on the market that build on our ideas. The goal of our research is to create scents without the use of chemicals, something that has never been done before. In the longer term, Scentee opens up completely new possibilities for creating experiences in digital media and experiencing them with the sense of smell. For example, we could add a fragrance to the photo of a tasty meal on Instagram.

Smelling via VR mask

We’re already one step ahead in Virtual Reality (VR): Driven by the gaming industry and the high expectations of gaming fans for new gadgets and future technologies, manufacturers are constantly working on new prototypes. Ideas and innovations are quickly adopted and brought to market at a breathtaking pace. Based on our research, in 2015 Feelreal developed the first prototype of a VR face mask that adds new dimensions to virtual reality. In its current form, the mask allows the gamer to perceive up to ten different scents and touches when playing online. In the future, the offer will be extended to 255 scents.

The method just described for the chemical activation of the smell sensation, as well as the VR face mask, has several disadvantages. For example, low controllability, high complexity, and high costs. In addition, the scent cartridges have to be refilled by the consumer frequently. For this reason, we have been researching the development of a new interface for quite some time that can induce weak electrical impulses at the scent receptors and generate scent sensations. Our goal is to activate the scent receptors with weak electrical signals and thus to generate scent perceptions.

The sensitivity of human olfactory receptors to electrical stimulation and their effectiveness is tested using a current control device. This is equipped with adjustable parameters: Frequency as well as electricity to create the necessary electrical impulses. To activate the stimulation process, two silver electrodes are placed inside the nose, where the olfactory nerves are then touched.

The odor-sensitive receptors are located near the olfactory bulb and the turbinate (nasal concha): Within the nostrils there are three areas called the upper turbinate, middle turbinate, and lower turbinate. These are closest to the important olfactory receptors. The stimulating electrodes can be arranged here. As part of our research, we mainly stimulate the receptor cells in this area.

Once the electrodes have been placed with the help of a medical expert, we control them with a dedicated interface. The odor-sensitive cells are stimulated with a few milliamperes of current. To date, very few attempts have been made to electrically stimulate the olfactory system. In most system-related studies, air injection was used as an olfactory stimulus to activate the olfactory bulb.

A digital lollipop simulates taste

Another example of our work is the digital lollipop which can simulate taste and be sent over the Internet. Just imagine a digital lollipop that virtually synthesizes flavors by stimulating the human tongue with electrical currents. Taste usually results from the reaction between ionized materials and the taste receptors on the taste buds of our tongue. The taste synthesizer consists of two thin metal plates that are placed on and under the tongue. These manipulate the taste sensors via alternating current and small temperature changes.

The device is thus already able to produce the five main tastes: Sweet, sour, salty, bitter, and umami (savory/spicy). Meanwhile, we have also started to stimulate different types of taste receptors on the tongue. But it will certainly take some time until we are able to create and modify complete virtual taste experiences.

The sensation of wind and sun

For the last example, I’ll take you on an adventure: Imagine that you put on your VR glasses and suddenly find yourself in the Salzburg countryside on a hill again. You look down the slope, tighten the straps on your snowboard, and off you go. Of course you chose the black diamond slope. You thunder down the slope, the glare of the sun hits you from above, and the headwind is getting stronger…

What certainly sounds like it’s still a long way off is already a reality in research: Ambiotherm is a gadget that can be linked to VR glasses. The device simulates sun and wind in virtual space. Ambiotherm consists of two modules: a double fan and a module responsible for the temperature. The two fans are mounted on the underside of the VR glasses and are aimed at the mouth and nose. Thanks to the constant wind on the throat and the area around it, for example, the aforementioned headwind on the slope can be credibly simulated. The second module simulates the ambient temperature in virtual space. It is worn on the neck and nestles around the throat like a warm pillow. When needed, it radiates heat from the neck area throughout the body. Researchers at the National University of Singapore have already shown in several experiments that thermal simulation can influence the human body and alter body temperature.

Ambiotherm is still a scientific experiment, but it is expected to be ready for the market in the coming year. Enjoy your descent down the slopes!

Feeling objects with a smartphone

Only the digital sense of touch is missing. Even if it can already be “activated” by computers and devices – for example by swiping, turning, pulling, and pressing on smartphones – it will take some time until the technology is ready for the market. Both at the University of Pennsylvania and at IBM Research in the United States, researchers are working on innovative technologies to digitally touch things in real time. For example, it will soon be possible to feel objects via smartphones – through the use of “microscopically” low vibrations.

And in the near future, we will be able to feel a hologram through electrical impulses. In addition, thanks to the latest research on virtual reality, some input devices can already provide people with force feedback to their hands or other body parts. As a result, we are able to orient ourselves through haptic and sensor technology as an additional sensory perception in the three-dimensional world and obtain realistic experiences.

The Kissinger for instance, a gadget which we developed at Imagineering Insitute,“feels“ the pressure of lips with its sensors and transfers it in real time. The gadget stands for amazing multisensoric experiences we can expect from digital media in the near future.

In the coming years, we will certainly be able to extend this technique to the senses of taste and smell. We will be able to psychologically and technically introduce digital multisensory experiences under the skin – more intensively and effectively than in the real world. In the future, we will have completely new shopping experiences, rediscover traveling, organize our work very differently, and revolutionize shopping. In short: Multisensory technology in the digital space will change every aspect of our lives.