• Formulate a set of clear research questions relative to a self-identified example of a communicative setting and resultant tangible communication products in which the interplay between words, texts, images, colours, and other modalities play a decisive role in ensuring the desired cognitive effect.
• Apply key concepts and empirical methods presented during the course to analyzing such examples (cases) with a view to both linguistic, visual, and other sensory perspectives and to cognitive impact.
• Suggest specific adjustments to existing multimodal communication products to optimize their communicative and cognitive effect.

Words, texts, pictures, colours, and sensory impressions all have a semiotic potential; i.e., they can be used to convey some form of communicative content. People use them not only to transfer information but to create new conceptual structures and, ultimately, states of affairs in their own lives and in society at large: new products, political beliefs, social behaviours, and so forth. However, the in-depth study of the working mechanisms behind these very different types of semiotic units are usually studied in isolation by general disciplines such as linguistics, cognitive psychology, visual communication, and sensory science. Yet in most real-life circumstances they are used and encountered in combination, instantiating what has been termed multimodal communication.  Archetypal examples include: newspaper front pages, printed advertisements, and web pages (combining verbal and visual information); films and videos (combining dynamic visual and auditory information); and product packages (combining verbal, visual, tactile, and other sensory information). Another example is a conversation between two people which combines speech and prosody together with facial expressions and bodily gestures.
The need to address the total “semiotic cocktail” in integration is well recognised by such practice-oriented fields as marketing, branding, and advertising. Yet the focus tends to be on other essential facets of these endeavours than the mixing of individual “cocktails” which is left to the intuition and creativeness of designers and art directors. The course provides more systematic theoretical and methodological tools for analysing the communicative potential of various elements of the “semiotic cocktail” and their potential interplay. The perspective is cross-disciplinary, drawing primarily on selected insights from the linguistic and cognitive sciences. The points are illustrated by real-life cases drawn from such fields as advertising, product packaging design, and web design.


The overall objectives of the course are to enable students to understand, reflect on, manage, and deal hands-on with key dimensions and challenges of multimodal communication in a business or institutional context. To achieve this, cross-disciplinary methods and models will be used, and problems will be analyzed from the recipient’s as well as the sender’s perspective. The course will naturally draw upon insights gained in relevant adjacent courses within the overall teaching programmes followed by the student, with room for individualization according to personal interests, particularly with regards to case work.
 
The course will provide the participants with a background to understand and deal with:

• Similarities and differences in the communicative potential of verbal vs. visual, and other sensory means of expression.
• The different mechanisms of conceptual and emotional processing and potential effects of various kinds of expression units on both levels.
• The interplay between stimulus- and goal-driven attention on the one hand, and semantic disambiguation on the other in real-life communicative situations.

Machin, D. (2007). Introduction to Multimodal Analysis. London: Arnold.

Kress, G. & van Leeuwen, G. (2001). Modes and Media of Contemporary Communication. London: Arnold.
 
The textbooks will be supplemented by concise course material in PowerPoint as well as selected journal articles, reports, and real-life examples of multimodal communication uploaded in CBS Learn.