Beyond bulky VR headsets: Voice recognition, eye tracking, and natural gestures in the era of the metaverse and the “Medi-verse”

Imag­ine you have moved to a new city and bought a con­do adver­tised as “meta­verse enabled.” Upon clos­ing, along with the phys­i­cal keys to your con­do, you receive a unique cryp­to­graph­ic key to the com­mu­ni­ty. You move into your neigh­bor­hood both phys­i­cal­ly and dig­i­tal­ly. With the cryp­to key, you link your own per­son­al meta­verse pro­file to the con­do. This merges your per­son­al meta­verse, includ­ing a dig­i­tal cat­a­log of all the objects in your home, to a dig­i­tal map of the new space inte­grat­ing all the sen­sors and devices that con­trol the objects through the metaverse.

As you move from your bed­room to the kitchen, the lights switch on, and the room tem­per­a­ture adjusts. In the kitchen, you tell the stove to turn on and a graph­i­cal dis­play appears in front of you as you turn your gaze toward the refrig­er­a­tor. You reach out your hand and select the egg icon, which reveals that you have three eggs left. You swipe up with your hand and the order goes out for more eggs.

As you pre­pare break­fast, you ges­ture again, and in anoth­er moment your mother’s face appears in front of you. “Mom, do you want to join me for break­fast?” “Sure,” she responds as she turns down your stove so your eggs don’t burn! You gath­er your eggs and bacon and sit down at the table, which has moved out from a recess in the wall in antic­i­pa­tion of your approach. You sit down, your mother’s holo­gram appears across from you with her break­fast, and the two of you begin to dis­cuss your day.

Lat­er that evening as you walk around the neigh­bor­hood guid­ed by your AR dis­play, you’re prompt­ed with infor­ma­tion tagged to hous­es through­out the neigh­bor­hood. One that catch­es your eye says, “Book club meet­ing at 6:30 p.m. tomor­row at my house—digital only this week, meet in the com­mu­ni­ty meta­verse lounge. New mem­bers wel­come!” You’ll fit right in!

Ready to move into this con­do? What I’m real­ly describ­ing is a form of spa­tial computing—technology that seam­less­ly inte­grates the phys­i­cal and vir­tu­al world. I wrote about spa­tial com­put­ing in a Sci­en­tif­ic Amer­i­can arti­cle in 2020, and it’s one of the build­ing blocks of the meta­verse, the vast net­work of phys­i­cal and vir­tu­al net­works primed to rede­fine the way we live and work in the not-so-dis­tant future.

You may have glimpsed the meta­verse already if you’ve watched the movie Ready Play­er One or played Minecraft or Fort­nite. Mas­sive­ly mul­ti­play­er online games like Fort­nite have been around for over two decades, as have online worlds like Sec­ond Life. Around the time I start­ed Anthro­Tron­ix in 1999, Jac­ki Morie, a vir­tu­al real­i­ty pio­neer, helped estab­lish the Uni­ver­si­ty of South­ern California’s Insti­tute for Cre­ative Tech­nolo­gies, an army-fund­ed research lab find­ing con­nec­tions between enter­tain­ment and mil­i­tary needs.

She start­ed the Com­ing Home project in 2009, cre­at­ing a cen­ter in Sec­ond Life where vet­er­ans and sol­diers could find stress relief and reha­bil­i­ta­tion activ­i­ties. Kin­dred spir­its in the quest to use tech­nol­o­gy to cre­ate mean­ing­ful expe­ri­ences that enable abil­i­ty and enrich people’s lives, we began a col­lab­o­ra­tion. We con­nect­ed our dig­i­tal glove tech­nol­o­gy to Sec­ond Life, enabling vet­er­ans using wheel­chairs to rock-climb, and our teams joint­ly invent­ed a “scent neck­lace” that released com­put­er-con­trolled scents to make vir­tu­al real­i­ty worlds more immersive.

We don’t real­ly know the poten­tial of the meta­verse, or whether it will be com­plete­ly vir­tu­al or a mix of phys­i­cal and dig­i­tal. As we lever­age the inter­net, advances in vir­tu­al and aug­ment­ed real­i­ty tech­nolo­gies, and spa­tial com­put­ing, there will like­ly be mul­ti­ple meta­vers­es, includ­ing a poten­tial “Medi-verse” that focus­es on health­care deliv­ery. But whether you enter a con­sumer-ori­ent­ed gam­ing meta­verse or an immer­sive work one, one thing that seems unlike­ly is us con­tin­u­ing to inter­act with the dig­i­tal world through desk­top com­put­ers, smart­phones, and bulky head­sets. More like­ly, we will inter­act with the dig­i­tal world through voice recog­ni­tion, eye track­ing, and nat­ur­al ges­tures like in the sce­nario above.

My vision of the meta­verse is that we will be able to phys­i­cal­ly go about our dai­ly activ­i­ties while inter­act­ing in real time with friends and col­leagues around the world. I want to go into a busi­ness meet­ing and sit around a table with col­leagues from dif­fer­ent coun­tries with­out fight­ing with video con­fer­enc­ing dis­plays. I want to go to the gym and work out with my friends in Paris and San Francisco.

There is a much-used quote attrib­uted to many, includ­ing Abra­ham Lin­coln: “The best way to pre­dict the future is to cre­ate it.” We have the tools to cre­ate the meta­verse. What we do with them will be up to us.

– Dr. Corin­na (Cori) Lath­an is a tech­nol­o­gy entre­pre­neur who has devel­oped robots for kids with dis­abil­i­ties, vir­tu­al real­i­ty tech­nol­o­gy for the space sta­tion, and wear­able sen­sors for train­ing sur­geons and sol­diers. Above is an adapt­ed excerpt from her new book, Invent­ing the Future: Sto­ries from a Tech­no-Opti­mist (Lion­crest Pub­lish­ing; 2022), which explores the many pos­si­bil­i­ties of tomor­row through Cori’s twen­ty-year jour­ney invent­ing at the edge of tech­nol­o­gy and human performance.

Related reading:

• Tak­ing your brain vitals: Sto­ries from a tech­no-opti­mist invent­ing the future of human performance