In-Air Device Formations for Cross-Device Interaction via Multiple Spatially-Aware Armatures

Опубликовано 11 октября 2021, 19:00

The Ultimate Flexible Workstation? In-Air Device Formations for Cross-Device Interaction via Multiple Spatially-Aware Armatures ("Air Constellations")

"The Ultimate Flexible Workspace" is a new device concept that uses spatially aware armatures, similar to articulated lamps, to make multiple devices work better together. The user can freely adjust, orient, and juxtapose multiple devices in-air, and the applications on each device react accordingly as displays approach, re-orient, or split apart.

Also known as "Air Constellations," our prototype system supports a unique semi-fixed style of cross-device interactions via multiple self-spatially-aware armatures to which users can easily attach (or detach) tablets and other devices. In particular, AirConstellations affords highly flexible and dynamic device formations where the users can bring multiple devices together in-air – with 2-5 armatures poseable in 7DoF within the same workspace – to suit the demands of their current task, social situation, app scenario, or mobility needs.

This affords an interaction metaphor where relative orientation, proximity, attaching (or detaching) devices, and continuous movement into and out of ad-hoc ensembles can drive context-sensitive interactions. Yet all devices remain self-stable in useful configurations even when released in mid-air.

We explore flexible physical arrangement, feedforward of transition options, and layering of devices in-air across a variety of multi-device app scenarios. These include video conferencing with flexible arrangement of the person-space of multiple remote participants around a shared task-space, layered and tiled device formations with overview+detail and shared-to-personal transitions, and flexible composition of UI panels and tool palettes across devices for productivity applications.

A preliminary interview study highlights user reactions to AirConstellations, such as for minimally disruptive device formations, easier physical transitions, and balancing “seeing and being seen” in remote work.

These concepts address the very real user needs of information workers to combine resources from other devices nearby, to collaborate in nuanced and socially fine-grained ways with colleagues whether local or remote, and to create new, more useful experiences that compose multiple device capabilities such as pen, touch, camera, and expanded screen real estate.

This work is part of a larger Microsoft Research project known as SurfaceFleet that explores the distributed systems and user experience implications of a “Society of Devices” in the New Future of Work.