Interaction Metadata from Multimodal Interactions

With the advent of digital cameras, people have taken more pictures than ever [1], the size of our photo collection has exploded. Taking appropriate pictures from these huge collections is a clear problem. However, there is no one who is willing to do a hard job of marking these vast collections of photos and indexing them [2, 3, 4]. As Fleck M points out, when people add new images to the collection they do not see the usefulness of annotations and indexed pictures.

Multimodal: Use various interfaces in various situations. We are a keyboard for typing, a touch screen for drawing and design, voice for communicating with our digital personal assistant, radar based gesture control, muscle based game and VR gesture control, and your You can use the brain computer interface options tailored to your mood. musics

Multimodal application functionality includes voice and GUI interaction. The standard for multimode interfaces should be extensible to provide rich functionality for subsequent multimode devices. To promote rapid recruitment, you can design the same content to be used on old / simple / new / multimode devices. For example, a person with a new multimodal device will experience its multimodal function, but users with legacy simple devices will still use the keyboard and stylus as before. A user of the multi-mode device can provide input via voice, handwriting or buttons and provide input via display, pre-recorded and synthesized voice / voice, and haptic mechanisms such as vibrator and braille band It becomes possible to do. Application developers can provide a valid user interface for any mode selected by the user.

Multimodality and Arts Walsh (2010, p. 213) describes "multimodal literacy" as meaningful by "reading, observing, understanding, responding to, producing, and interacting with multimedia and digital text" It is defined. This interaction includes spoken language, listening, dramatization, composition, design, and text creation in voice and gesture mode (page 213). A multimodal presentation is realized by image, gesture, line of sight, body posture, sound, lighting, and positional composition of music and is realized as a systematic collection of symbolic resources, called patterns, or production I will. , 2008, p. 246). The essential concept of the multimodal concept is "pattern - occur if it exists" (Jewitt & Kress, 2003, Albers & Harste, 2007, p. 11), and there is a complicated relationship between them. Create patterns in text