Mark Patrick, Mouser Electronics
The world is beginning to realize the potential of Augmented Reality (AR). With its ability to join reality to virtual worlds, user interfaces will be de-cluttered in a way we could not imagine previously. Eventually HMI devices such as keyboards, mice and even displays will disappear, being replaced with a more natural, gesture based, interface.
Unlike the newer Virtual Reality (VR) technology, AR does not create the entire world, it simply overlays useful information and / or graphics on what is already there. In a simple form, AR has been with us for a while in the Head-Up Displays (HUDs) that have made military vehicles (and some high-end mass-market cars) easier to drive. Early incarnations projected images onto the windshield, giving information while allowing the driver to see the real world beyond – reality was augmented in this way.
The ergonomics of AR have been considered for some time – one key issue is considering what types of (and how much) information can be displayed without distracting the user. Also, designers need to consider the amount of information in the context of the processing power of the system – too much information could cause unacceptable latency, diminishing the value of the AR.
Primary types of current AR
Like many early-phase technologies, AR has several forms and no one in particular has come to the fore – yet. While there are many sub-categories (and many more in development) current AR can be grouped into these three primary categories: –
• Handheld AR
• Wearable/Immersive AR
• Projected Area AR
AR is normally defined by the way it appears to the user. Both handheld and wearable are considered ‘first person’ approaches where the user is fully immersed whereas projected is a ‘third person’ technology where the user is looking into the world from above or outside – almost an ‘out-of-body’ experience.
The ubiquity of smartphones and, to an extent, tablets has made handheld AR the most popular as these portable devices can detect motion and their orientation – the essential ingredients of handheld AR. Smartphones and tablets provide a movable window into the AR world and this has been used to great effect in applications such as the Pokémon Go game as well as apps that are able to superimpose reviews and ratings on nearby businesses such as hotels and the like.
To be fully immersed in the AR world a headset or eyeglasses is often required although adapters to hold a smartphone securely in front of the eyes are becoming popular as many people have a smartphone and the adapters are relatively low-tech and low cost. Immersive AR has many commonalities with VR and, in fact, the edges are slightly blurred giving opportunities for products and applications to effectively bridge the gap.
The world is noticing
Head-mounted displays (HMDs) in their various forms are what are getting noticed by consumers and journalists, despite mobile device incarnations of AR being, by far, the most common. In general, the AR imagery is delivered by reflected micro-displays using one of a number of technologies including liquid crystal on silicon (LCOS), MEMS-based digital light processing (DLP) or organic light emitting diode (OLED) arrays. OLEDs self-illuminate meaning that the device size can be smaller (and consume less power) as they do not require a light path as LCOS and DLP do.
The key difference between AR and VR is that AR requires a ‘passthrough’ whereby the user is able to see the real world as well as the augmented display. Currently direct passthrough is the only approach in actual use although research is under way to provide the view of ‘reality’ via a small camera. In reality this may be a way off as issues such as latency and synchronisation need to be addressed – and adding a camera and more information to the display will require enhanced processor systems.
These AR headsets, wearables and glasses are similar to VR devices, but with the important addition of a ‘passthrough’. This refers to a view of the outside world, upon which virtual elements are overlaid – either via direct view, or possibly using a low-latency camera feed. To summarise the current state of the art, direct view passthrough AR devices are seeing some real-world use, whereas camera-based AR technology is in truth more of a work in progress. While all forms of AR must overcome challenges of resolution, motion tracking and synchronisation of real and virtual elements, camera-based AR is more prone to these problems and also more demanding on hardware, because it has to transmit the entire real viewing field to the user in real time.
Real world examples
The AR device that most people have heard of is Google Glass. Based on a monochrome LCOS array that uses fast cycling of colour LEDs to create a full colour display, Google Glass reflects this to superimpose the AR visuals onto what the user is actually seeing through the glasses. Microsoft’s HoloLens is larger and more sophisticated, but does follow some of the same principles. With HoloLens, two LCOS arrays generate separate images for each eye and they also use the field sequential approach to convert the monochrome display into colour. One major benefit of the two LCOS arrays is – with some clever optics and image processing – HoloLens has the ability to deliver 3D imagery.
AR: Enterprise or consumer?
While Google Glass caught the public’s imagination for a number of good reasons, it also raised a number of privacy concerns, which has slowed adoption of the technology as a whole. This has turned the industry to look to the commercial / enterprise sector as the next big opportunity – certainly for the short term.
The financial dynamics in commercial applications are somewhat different. In the consumer world, perceived value is a significant factor in the decision whether to buy, or not. In commercial use, if a profit can be driven (or a cost saving achieved) then the purchase can be justified, whatever the price tag. Provided the value proposition can be constructed and delivered to enterprises, such as cost savings and / or productivity improvements by equipping production-line workers with AR headsets then sales will almost certainly follow. Even the re-branded Google Glass ‘Enterprise’ version is seeing a much better response from commercial customers than it did from the general public.
Unlimited AR possibilities in commercial usage
The primary benefit cited for AR in commercial applications is making staff more efficient and productive – it is almost as if they carry their desk with them wherever they walk in the building. In the retail sector, or anywhere face-to-face customer contact takes place, other possibilities spring to mind. A vastly enhanced customer relationship management (CRM) system that incorporates facial recognition would allow customer-facing staff to greet repeat customer by name, and even ask about a recent purchase – or to know their preferred brand without asking.
Healthcare professionals the world over are generally resource-poor, so the ability to pull up medical history, medication plans or test results in real time, there and then not only makes the process more efficient but also safer and more accurate as well. Compared to the potential cost of getting things wrong, AR represents a relatively small investment.
In the future there will be two types of AR usage, those where AR is used to extend existing systems and processes and new applications written specifically with AR in mind. Enhanced CRM is one example of an ‘extension’ app where details already in a database are presented to staff at the point where they need them.
Another example would be to walk production line staff through complex assembly processes on the production line, or even to confirm if a part was fitted correctly. Test results could be shown to quality staff, allowing them to understand and resolve the issue without needing to move from their workstation.
Surgical use has already been tried with both Google Glass and Microsoft HoloLens – in both cases they can show surgeons positions of vital things such as arteries or organs, making the surgery safer and perhaps more efficient too.
In security applications staff could identify individuals at a major event and be informed if they had been previously banned. Law enforcement professionals could look at a face in the crowd and know if there were outstanding warrants or any other issues to address.
Summary
As AR hardware is reaching the early stages of maturity it is now becoming a viable technology, albeit one that will continue to develop over time. The current challenge is increasing the sophistication of the software to move from the basis alphanumeric text of today to ‘all singing, all dancing’ systems of the future that will deliver the virtually limitless potential that AR promises.
