|

Section Navigation

• Phase 4 (2005-2008):
• Phase 3 (2002-2005):
• Phase 2 (1999-2002):
• Phase 1 (1996-1999):

Throughout the previous funding periods of the CRC 378 in the projects EM 5 READY and EM 4 REAL, methods and techniques have been investigated and developed to adapt dialogues and presentations to the limited cognitive and technical resources of users and systems. Both projects have started their research at opposite ends of the common problem definition. While EM 5 READY has investigated how to model and how to recognize the limited cognitive resource of users, EM 4 REAL has been concerned with the adaptation of presentations to the detected resource limitations of users and to the technical limitations of the target devices.
As shown in figure 1, there has been a trend of convergence of the project questions and results over the last years. Already in the second funding period collaborative activities have led to several joint publications. In the 2002-2005 funding period results from EM 5 READY and EM 4 REAL were integrated into a joint system prototype. Consequently, we now plan to combine both projects for the last funding period of the CRC 378.

This newly formed project EM 4 BAIR (User Adaptation in Instrumented Rooms) focuses on foundational research on the role of the cognitive and affective state of the user for the adaptation of information presentation in instrumented environments. For this purpose we will extend the infrastructure of the instrumented environment that has been set up during the last funding period in EM 4 REAL. Figure 2 shows the additional horizontal affective layer which is placed between the services and the physical environment, that is used on the one hand to take into account the user's affective state and on the other hand to generate affective responses from the instrumented environment. For this purpose we plan to investigate two different anthropomorphic user interface methodologies. Firstly, the use of a virtual inhabitant of the instrumented environment (i.e. a life-like character) will be explored to guide and inform users, and secondly our new idea of talking objects will be explored in more depth. In this context, we will extend the airport shopping scenario to include smart objects that can respond to users spoken requests directly in natural language. Furthermore, users will be enabled to delegate information requests to the virtual inhabitant, which can in turn assist users pro-actively in accomplishing their tasks.

The introduction of the affective layer also has a vertical influence on the other layers that have been established so far. The physical layer has to be augmented by bio-sensors that are either worn by the user or installed in the instrumented environment. The data obtained from these sensors has to be integrated with other modalities and the modality fusion component developed during the last funding period has to be extended respectively. Hybrid classifiers, that include embedded Bayesian networks and Hidden-Markov-Decision-Processes, which were under investigation in EM 5 READY so far, will be applied to correctly classify and react to the user's affective states. The service layer will be extended to support multiple users in the same instrumented environment. This again will raise new technological issues regarding the tracking and identification of users. Finally, the knowledge layer will be extended by an explicit affective model, which contains both knowledge needed for the identification of emotions and for the correct use of affective responses by the instrumented environment.

There has been a close cooperation with the project EM 1 VEVIAG (Zimmer/Mecklinger) and EM 2 ARC during the last periods, which will be continued as described in this proposal for the last funding period.

In the third funding period, the project REAL is concerned with the main question: How can a system assist its user in solving different tasks in an instrumented environment?  Such environments consist of distributed computational power, presentation media and sensors, and also entail the observation and recognition of implicit user interactions in the environment. This offers the possibility to infer about a user's plans and intentions, and to proactively assist in solving their task.
We focus our interest on two particular tasks in an airport scenario, shopping and navigation.  In the shopping scenario, we explore how to assist the user in achieving their goal of the best possible buying decision within a given limited time.  We employ an RFID-technology based infrastructure of readers and labeled products to sense implicit user interaction, such as picking up a product from the shelf or putting it into the shopping cart.
We can imagine a variety of mobile and stationary devices to be used for information presentation and dialogues with the user. Some user might prefer their own PDA or smartphone, others might use a shopping cart equipped with a touchscreen. It is also desirable to involve large public displays for the presentation of rich media content, which otherwise are used to display advertisements.
In order to enter the shop and to pick up a certain product, the user pursues certain navigational goals. Therefore we are integrating our previously developed stand-alone pedestrian navigation system into the instrumented environment, which provides for routing and positioning services. Besides the navigational aid, the system also offers an exploration mode, which allows the user to query information on points of interest within a three-dimensional navigational map. The user may formulate their request using combined speech and stylus gestures.
More information is to be found on the M3I pedestrian navigation system and the Smart Shopping Assistant homepages.

In the second phase of the project REAL we are investigating the resource-adaptive graphical presentation of spatial information for way descriptions. The techniques we came up with are currently being implemented within two scenarios, a building navigation system called IRREAL and an outdoor pedestrian navigation system called ARREAL. In IRREAL we are designing a building navigation system based on handheld PDAs, such as the 3COM Palm Pilots and a number of strong infrared transmitters. The off-the-shelf PDAs are used as display units and receive up-to-date information and directions for the user through their built-in IrDA interface. The transmitters are custom hardware designed to cover a range of up to 30 meters. These transmitters, placed at strategically important points throughout a building, broadcast text and graphics for way directions or other localized dynamic information. The advantage over a self-contained information database on the PDA is that information can be localized with a granularity of a few meters and depending on the user's viewing or walking direction. The graphical presentations for the devices are generated on a central server and adapt to the very limited technical resources as well as to the user's cognitive constraints. In AREAL we are designing an outdoor pedestrian and bycicle navigation system based on head worn displays, a differential GPS and a wearable computer in a backpack. Since all presentations are generated on the wearable computer, the generation process itself has to deal with very limited resources. The information can be localized with a granularity of roughly one meter depending on the user's position, viewing or walking direction. The graphical presentations will be supported by some speech output using the results from phase 1 of the project REAL and we are also planning for simple speech input facilities for interaction with the navigation system. You will find more details about these scenarios on the respective pages about IRREAL and ARREAL.

Concentrating on the generation of natural language spatial descriptions in dialog situations the interaction of resource-limited object localization and incremental natural language generation is examined and modelled in a system which answers spatial orientation queries. First, the cognitive system should answer where-questions under resource limitations about its actual visual environment in different spatial scenarios.
Considering experimental results of cognitive psychology, a basis for the generation of more realistic localization expressions, e.g., in future driver navigation systems, mobile robot system, or route description systems, is developped.
The system should be parametrized in a way that it is able to start with the verbalization of localization expressions before the spatial search and the mapping of spatial relations onto spatial propositions has completely been finished. Considering such an incremental generation, an interesting spectrum of performance phenomena appears - from a successive refinement of the spatial description to the correction of already verbalized fragments - which should be examined in the forthcoming system. The image below shows a sketch of the architecture.

Therefore, anytime algorithms are developped which always deliver adequate but approximate results by varying the temporal and sensoric restrictions. The project especially focuses on the search for reference objects, the computation of spatial relations and the anticipation of the listener's imagery under resource limitations will be modelled and implemented in an integrated process model.