Graphical and linguistic dialogue for intelligent multimodal systems

摘要

In this paper a brief outline of the elements of a theory of multimodal representation and interpretation is presented. The theory is illustrated with the help of an interactive system called GRAFLOG which is able to support a multimodal graphical and linguistic dialogue. In this dialogue graphical symbols with their interpretation are introduced incrementally, and the system is able to assist the user in the solution of simple design problems. The theory has been used as a framework for the development of a number of master thesis and experimental systems addressing different aspects of representation and problem-solving in the context of multimodal dialogues. This line of work is also summarized. At the end of the paper the kind of dialogue that is currently being considered for the next version of the system is illustrated.