The Augmented Reality Gap

I spoke at the first Dryburgh Emerging Communications Conference (eComm) of 2010 about the application of Augmented Reality to face-to-face conversation. Video will come online later this year. These are my notes from that six-minute presentation.

 

I. The reversal

Interpersonal interaction started in the real world. The software community started by modeling observed behavior. Postal service became email, writing became word processing, meetings became conferencing, bulletin boards, instant messaging, and Skype calls. Services experimented with a mix of media choices; time/space/channel shifting; participant discovery, invitation, and scheduling; defining participant roles and moderating behavior; backchannels and voyeur streams; and the archiving and distribution of meeting work product.

The first generation of AR for conversation will reverse the flow, bringing the benefits of newly defined online social experiences to face-to-face real life encounters.

 

II. A dream platform

Let’s take a walk with a conversation charged with by augmented reality. Dream with me that we reach technical and economic feasibility in a few years.

In this world, eyeware becomes AR’s delivery platform, freeing your hands. Gestures and speech are our mouse and keyboard. I’ll assume a relatively thin client that senses locally and works with information via computing clouds.

 

III. Aug’d Talk

Before our meeting, gatekeeper bots help you screen, find, schedule, negotiate people to visit. Think services like Tungle.

Flickr recognizes their faces at a distance, tells you to turn around to greet them, and makes them glow as they approach.

Fashionista scores the other person’s wardrobe and accessories. “Gucci knockoff”, “UC Berkeley Store: $34.95”, “Amazon Wishlist: Pearl Earrings, learn more.”

Equifax shows a hovering frame with their latest credit score, criminal background check, and public updates. "Experian: 4 of 5 stars."

Seesmic dampens your heads-up social peripheral vision so you can pay attention to each other.

Plaxo reminds you of who your closest mutual friends are with a two-second montage of faces.

Systran interpreter bots overlay speakers with live subtitles in your first language.

Nuance and Bing listen in as you talk, whispering private tips and reminders in your ear. "Amy’s mother’s name is Gail." The social secretary serving up social objects and avoiding faux pas.

SecondLife adds a third participant’s avatar as we walk and talk down the street. Giving a virtual participant a medium for to participate in a realworld conversation.

Zoho Planner tells us we have three more minutes on this topic.

I see my RSAbot flash a caution halo over your head when it thinks your voice, facial expressions, and body language show deceit or that you are very likely to become violent.

Yelp Monocle suggests a nearby lunch place we might like and books a table for us, including a chair for the avatar and a blank wall for projections.

Google AdWalls places poster ads on blank walls as we walk by, some optimized by our mutual Buzz history, others unique to each participant.

UStream.ar makes a composite video stream from our respective Logitech PoVcams. Hundreds of voyeurs join our conversation, chatting in their own instant messaging backchannel.

Zemanta links to related conversations you might like. “Carol and David – same spot, same day, last year.” “Ed and Faith – similar topic (live).” “Gary, Harry, Ike – same café (live)."

Meetup reminds us there’s a flashshmooze in five minutes. "Our World’s A Conversation®."

Basecamp helps us recap our action items and commitments for our next meeting.

 

IV. Augmenting conversation requires technical architecture

We don’t have what it takes to deliver this experience in 2010. Let’s start with the technology.

A. The AR community opens its architecture.

Most parts of augmented reality software are bundled, intertwined, and often proprietary. I heard more than one AR researcher say at eComm they build AR browsers from scratch for each project.

Developers need a stack architecture that isolates components from one another, that defines how the parts talk to one another. A well-crafted stack means technologists and businesses can compete to become best-of-class within their part of the stack without breaking dependencies with other parts. For example, improvements in how bits are routed over the Internet don’t require an update to every web page; the two components are isolated, each doing their part.

This is how our telephones, local networks, and the Internet work. A stack speeds engineering through focus, reduces risk through compartmentalization, and allocates resources well as each layer of the stack becomes sends signals about its technical and business opportunity.

For a stack architecture to work, it must become a de facto standard, used by all.

B. The AR community separates browsers from content.

Most AR systems bundle content and features with a browser. This lets programmers tinker with the nuance of user experience, to exploit hardware, to code within an agile and iterative design process.

AR must split this atom to lower the bar for those who want to augment reality with information, behavior, communication and interaction.

Some companies, like Layar, aspire to be AR’s Netscape. That’s the right direction. Layar’s architecture is closed (you need their permission to use it), hosted (a single point of failure), and proprietary (only Layar defines how to use it). AR investors and consumers need a less risky architecture, so content and services survive when Layar closes (remember Netscape?) or acts capriciously (remember Apple?).

As with a protocol stack architecture, delaminating content creation and service from content players lets both evolve more quickly and with greater competition. At the moment, Layar offers a point-of-interest annotation browser; remember the web before forms, CGI scripts, Javascript and AJAX. AR has the potential for much more. That only happens when public and open protocols for serving and playing content emerge from the AR community.

Perhaps this is a call for Layar to follow Netscape’s shoes and further open its protocols and code, so its browsers can serve augments from any server, not just Layar’s. And so others can compete to serve AR content better, just as Firefox, Chrome, Opera, and Internet Explorer vie for web browser share.

C. AR browsers must offer concurrent experiences

I can only experience one reality at a time. I can’t see Wikipedia entries or the morning local news stories while looking for coffee. I want to. People need the power to blend content in our reality browsers. We need to be able to see content from multiple, independent sources at the same time.

Let’s assume your RayBan RealWare(TM) browser is letting you run a hundred views at once. Browser plug-ins and other services will help manage that experience to avoid overloading the user and to increase relevance. To do this, we need a few technology standards comparable to what’s evolved on the web. A secure way to deliver a service layer. A standard model of a layer as a container. A standard model of things within a layer and a way for your software to discover its properties and behaviors. Universal baseline user interaction methods so users can navigate and manipulate and control their environment. Agreement, on a limited basis, of when and how objects in one layer may interact with objects from another layer and how to give power over that choice to each user.

E. Layer Discovery and Stores

Which brings me to distribution. Build interoperability so anyone and everyone can publish with minimal opportunity for governmental or corporate restriction. Decentralization is a defense against consolidated power. More on this later. 

 

V. What we don’t know

Despite more than twenty years of augmented reality research, AR is still early and immature. There’s a lot we don’t know.

A. Managing field of view

I was astonished by the 4K video displays at CES. So many pixels that you keep discovering new levels of detail as you walk toward the 152 inch monitor with four times 1080p HDTV resolution. At some point you can’t see the whole screen. You focus your attention, and your eyeballs, on a small portion of the whole. Just like real life.

When there’s a hard limit to what’s visible, you have scarcity. The economics of a zero sum game come to play. Those who value your attention (including you) will fight for prominent positioning within your field of view.

It’s happened before.

Billboards filled United States highways, advertisers bidding for the best locations to reach the most drivers. The public fought back after fifty years with the Highway Beautification Act promoted by Lady Bird Johnson.

Desktop operating systems and browsers also fight for scarce default pixels. Governments accused Microsoft of anticompetitive behavior for favoring its own Internet Explorer browser with an icon on the default Windows desktop. Google’s position as Firefox’s default search engine is worth more than $50 million annually. Skype agreed to exclusivity and to give lower its privacy expectations in exchange for scarce Verizon mobile desktop "on deck" placement.

For everyone else who wants your attention, there’s the $500 billion web browsing economy. Like America’s byways, the web brings a torrent of intrusive and distracting ads. Again, the public fights back with ad-blocking software, users seizing control of their pixels.

So the technology reveals a political question: who has power over what you see?

B. Interruption and Alerting

If you thought distracted driving was a problem… Interruption overload of your reality will be huge. AR interfaces can take your eyes from where they are needed, your mind off what you are doing, and pull your hands from what they are controlling.

I predict the first death from an AR interruption will be reported in a top news service by 1 January 2017.

Rapid contextual filtering will save lives.

Workshop to explore human cognitive limits and approaches to filtering well? 

C. Gestural Language Tower of Babel

All of today’s gestural interfaces are limited in their scope and vocabulary. And no two of them are the same. When you are living with multiple layers, you need universal universal primitive gestures. With the web we have "click this link", "go back", "change context", "type something here", "press this button to make something happen" and "close the browser." Let’s avoid having the "get me out of here" gesture from one service to be interpreted as the "place my order for Viagra" gesture by another.

Workshop to start harmonizing gesture languages, anyone?

D. Layer Discovery Protocols

Many web pages hide little notes that point to alternative versions of that page. That’s how feed readers discover RSS, Atom, and ActivityStream feeds.

We’ll want our layer discovery protocols to be useful for people, helping them choose what layer services and objects to consume, and for systems, describing layers in ways that let software understand how to interact with that layer’s capabilities.

This technical disclosure powers the realtime web and enables our mashup economy. Disclosure protocols can fuel a world of AR mashups and interplay.

Workshop to start prototyping and testing discovery mechanisms, anyone?

E. Designing for variable infrastructure capacity

Cyberspace isn’t flat. It’s lumpy, twisted, sporadic, and changing. AR will challenge the ability of each part of the Internet’s infrastructure. 

• Mobile CPU/Storage/Battery
• Bandwidth bottlenecks.
• Wireless coverage
• PAN bandwidth
• Relevant content density

F. Social conventions

What is the proper way to meet an augment? To agree on mutual augments? To include a non-augment in your conversation? To behave in the presence of offline people?

What is forbidden? Is it OK for me to alter how I see you without telling you?

OK to share our conversations and augments with others without disclosure? Would it be OK to do that if I felt you are dangerous to me? If I was paid to broadcast? Could you claim an equity stake in my broadcast revenue?

Should I automatically disclose to you the analysis and metadata I collect about you, so you understand how I perceive you?

We have an entirely new body of social behavior to evolve, adapt and codify. An augment etiquette that defines new conventions to support our new virtual realities.

G. Public Policy

We need to start a conversation about society’s interest in encouraging good behavior and discouraging the bad.

• Privacy for Public Conversations. I live in California where phone conversations may only be recorded if all the parties consent (California Penal Code 632). Should we extend this to conversations held in semi-public places now we wear recording media and as the streets and architecture become sensor-rich? 
• Data Portability Mandates. Companies hold our data hostage, the way landlords might hold your deposit and furniture hostage. This is a strong imbalance of power. As we come to rely on our augments for basic services all day long, their power will grow. Communities passed laws to protect renters from abusive landlords. Will we pass laws to protect people from sites that won’t let our data go?
• AR-Free Zones. There may be a public good in defining some areas free of some or all augmentation. Courtrooms, perhaps. Some public parks. Would it be OK for a restaurant to define itself as an AR-Free Zone and require you to turn yours off before coming in? We have precedent in smoke-free, phone-free, and pet-free restrictions. Is access to your view of the world a fundamental right, to be protected by law?
• Net Neutrality for AR era. Reality is too precious to let someone distort it. You don’t want any of the companies between you and your experience to have a say in what you see/hear/feel and what you don’t. In what you say and do, and what you can’t. To pick for you what is important and what isn’t. Their interests may not be yours. Should we legislate that ISPs serving AR experiences be forbidden from treating some bits differently from others based on their content or source? Should we apply net neutrality principles to companies like Layar who have the power to ban a publisher? To the companies that make your AR devices?
• AR Carterfone. The Carterfone ruling said you may connect any device to the telephone network so long as it doesn’t harm the network. This led to a world of fax machines, private telephones, mobile phones, telephone switches, and eventually Skype. We need an AR Carterfone. So industry innovates and consumers choose and power doesn’t lie with those who connect us to each other.
• Disclosure. "This is not real. Push here for details." "This is an old version of this place." "I am wearing an avatar."
• Equity and Property. Does your employer get a copy of your AR life when you leave the company? Does your employer get to cut you off from the people and memories you created through AR? Who gets the layers you created with the kids once you divorce? 
• Freedoms. Politically, AR is speech; everyone will be a publisher. We want our technology to promote free speech. Socially, AR is conversation, so we want technology to help people organize and assemble themselves. Economically, AR is land; everyone will homestead and grow and build on it. We want everyone to build well without stifling the AR economy. We need to map AR against our ideas of civil liberty, human rights, jurisdiction, citizenship, protections for the weak and disenfranchised.

 

VI. In short…

We will augment walking and talking. Enhancing our face-to-face conversations will make money. Industries that now help us organize our time and relationships will compete to be part of our world view.

The technology comes quickly but huge obstacles remain. The biggest is a lack of an IT architecture.

The future depends on research and conversations. Design research into vision, attention, interruption, and gestural language. Consensus on how to behave. Debate on how to set things up so we build an augmented society we want to live in.

For a start, let’s talk about an Augmented Reality Stack. Get started. Let’s target have a workshop early this fall.