Current Research

Aware Projector
A Steerable Gesture and Speech Driven Projector and Camera Unit

The topic of my dissertation is the realization of an Aware Projector - a steerable gesture and/or speech driven camera-projector unit, which will be used to create interfaces for human computer interaction. Thus the work will be based in the research area of Dialogue and Generation.

Previous Work

The topic of my diploma thesis is steerable projectors. Such devices like the Everywhere Displays (ED) Projector are used to create virtual displays on arbitrary surfaces in instrumented environments.
In contrast to the ED-Projector which uses a steerable mirror placed in front of a stationary projector, the Fluid Beam introduced in my diploma thesis is a combined projector-camera unit placed into a moving yoke. This solution is very efficient in minimizing blind spots, i.e. it can project in almost any direction. Using a 3D-model of the environment, the physical world can be superimposed by a virtual layer on which text, images, videos, live streams, etc., can be displayed. In this environment the steerable projector can be used as a virtual torchlight, making the virtual information visible in the region it is projecting onto. This approach makes it possible to supply additional information about physical objects in the environment or to place desktops on different surfaces.
The Fluid Beam application developed in my diploma thesis has been integrated into the SearchLight project which implements a physical search function for an instrumented environment. It uses the movable unit of the Fluid Beam to scan the environment for optical markers and to highlight searched objects within a projected spot. This functionality of locating physical objects in an instrumented environment is analogous to file search on a common PC, and helps blur the distinction of physical world and virtual data.
Currently I cooperate with Michael Kruppa in using the Fluid Beam to display a virtual room inhabitant (VRI) driven by the presentation planner he is developing in his Ph.D. work. Combining both projects enables the virtual character to "move" freely across the whole instrumented environment. Additionally a spatial audio, developed by Michael Schmitz, is generated in a way that it appears to come from the position of the character.

Ongoing and Future Work

As nowadays the technological progress enables displays growing larger and flatter one can imagine that in the future the whole walls of a room will consist of interactive displays. There exists already a vision of a Living Wallpaper that turns the room into a large display continuum. By using steerable projectors we try to make this vision real.
The central goal of my Ph.D. work is to find and explore novel interaction options for the Fluid Beam, thus upgrading it to an Aware Projector, e.g.:


• A system for automatic 3D model acquisition and self-calibration of the projector and camera unit
  
  In order to use the virtual camera method to correct for the projection distortion we need a 3D model of the instrumented environment. Currently this model is created by hand, but the Fluid Beam application would be much more comfortable if it can be acquired automatically using the movable unit's digital camera. There are two possible methods to do this:
  
  • By projecting a certain (rectangular) pattern on a surface, probably not perpendicular to the projection beam, one receives a distorted image on the surface. This distortion can be observed by the camera, and analyzing it the Aware Projector will not only be able to locate projection surfaces but also to detect his own position in the environment.
  • A simpler approach may be the use of some Photogrammetric Modeling software (e.g. FAÇADE) with which the interior of a room can be modeled from some photographs. In this case the user might have to define some key points in the pictures.
  Thus the user will no longer need to specify a virtual model of the environment but this will be done fully or half automatically by the Aware Projector itself.


• Different interaction options like gesture or speech recognition
  
  As in the area of ubiquitous computing the use of keyboard and mouse will play a less prominent role, the user should be able to steer the Aware Projector by gestures or speech. One interesting research topic regards multimodal interaction, e.g. one can imagine a situation in which the user is pointing to a certain wall in the room saying "Create a display!", and after analyzing his gesture and the sentence the Aware Projector will display the user's desktop at the specified surface.


• Combining several steerable projectors
  
  If the instrumented environment is very large (e.g. warehouses, supermarkets, etc.) a combination of several projectors will enable the creation of larger display surfaces and the compensation of shadows on them. It will also allow passing on virtual objects from one projector to another in order to move them over large distances. In this way - if the user is tracked - he can be guided through the environment.


• Interaction between stationary displays and steerable projection
  
  In our Saarland University Pervasive Instrumented Environment (SUPIE) there is a large plasma touch screen and several smaller screens. The interaction between these sceens, providing islands of higher resolution, and a projected image is also an important research topic.


• Performing user studies
  
  Users will be invited to work with the Aware Projector in order to evaluate its usability. Together with Michael Kruppa I am planning to implement a scenario in which a projected character appears as an assistant moving through the environment helping the user by giving him advise, showing him useful items, answering questions, etc.