Review: Multimodality

Nuts & Bolts

Condensed review focusing on the most salient arguments of the book, and how they are managed within Multimodality.

read more

Negative

What are the problems with Multimodality? This focused section provides analysis of the issues arising in the book.

read more

Positive

Interested in the positive aspects of Multimodality? This section provides the most positive arguments of the book.

read more

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Negative

 

The basics

Multimodality: A social semiotic approach to contemporary communication is my second experience with Gunther Kress. A heavy theoretical text, filled with terminology, it offers several ways to look at the different modes of communication; Kress offers lenses through which we can look at both communication itself and visual representation within communication.

Some of Kress's claims are broad-sweeping, whole arguments boiled down to easily digestible kernels such as “there is no meaning without framing” (p. 10, italics in original), “Knowledge and text are entirely linked” (p. 23), “Communication is a quintessentially social activity” (p. 51, italics in original), or my favorite: “All communication is movement” (p. 169). These statements – each well supported within the text – are wonderful markers along the journey Kress is taking, acting as sign posts to help get through the intense theory in the majority of the text.

There is some difficulty following Kress and knowing what he is trying to say. It's not that Kress isn't trying to be clear; if anything, the problem is that he is trying too hard to be clear. He takes so much time explaining and delving into all the tangential meanings that the original point ends up lost or, at best, heavily obfuscated by the surrounded text. While Kress says something as clear as “there is no meaning without framing” (p. 10, italics in original), he says it amongst a full four paragraphs of explanation. The quoted sentence is very clear, and does act as a signpost, but the amount of explanation required for Kress to get to such a simple statement muddies the waters.

This trail blazing mostly comes in the form of new terms and definitions. It seems like every other sentence contains a new term; there are around thirty in the first six pages of the book. Kress would have been more successful writing this book as an encyclopedia of social semiotics, rather than attempting to maintain an argument across a text that requires so many terms fired in such rapid succession.

However, this is Kress's method, and he makes no apologies for it. In fact, early on he says that “Developing ways of thinking about this at once simple and complex phenomenon – that is, setting out a social-semiotic theory of multimodality – is what this book is about” (p. 5, italics in original). In order to develop new ways of thinking, it makes sense that we need a lot of new terms. The terms themselves are not complicated, but the ways Kress uses them tend to be, so we need some explaining; Kress tells us what he means when he says “writing” “image” “colour [sic]” “names” “shows” “style” “mean” “frames” “highlights” and “read.” These definitions may be essential building blocks to the arguments Kress makes throughout the book, which is why he defines all of these before even beginning his argument. This allows him, later in the book, to make much more complex arguments without having to stop and redefine the words.

The argument

The definitions are not complex, and Kress throws them at us just so that we will be on the same page as he and can follow along his argument. There is no quiz at the end; as readers, we're able to look at these definitions as he shows them to us, then let them fade away, confident that, when they become important, Kress will bring them out again, and we will at least recognize them.

His argument is to extend the world of meaning, to show us that semiotics needs to stretch further than we have thus far allowed it to; that is, he is using the work of semiotics, which has thus far been applied primarily to text, and applying it across multiple modes of communication. He tells us that “'Language' isn't a big enough receptacle for all the semiotic stuff we felt sure we could pour into it” (p. 15). We have to accept that words are not enough; there are methods of communication we need to employ that aren't presented with just language. We have to reach beyond language and towards visual spaces, to the other modes of semiotic-conceptual work (p. 15). It is not an easy thing to do; there is much involved in what is essentially a reinterpretation of existing theory coupled with an exploration into new territory. As Kress says:

 

The task is to establish, with as much precision as we might, what these differences are, in specific cases and circumstances. What new kinds of questions emerge and are made possible; how do persistent, older questions get recast, in ways possibly that lead to more plausible answers; and who might benefit in what ways from the different answers (p. 15).

 

If he is successful, there will be new areas for communicators and rhetors to explore, new questions to ask, and, more importantly, new ways to answer old questions.

Images

The rise of images is pervasive; there is no way to live in our culture without seeing it. It's easy to find a mixture of text and image (there are images on this web page, for example), and easy to find images that stand alone; but finding text that stands by itself these days is a nigh impossible task. The simple fact is that images are good rhetorical tools, and so images are used more frequently. Kress tells us that “Rhetorical considerations do still organize all of communication, all semiotic interaction, at all times, even in times of stability and the dominance of convention” (p. 45). Those rhetorical considerations lead us to the use of images more and more frequently, and the shifting of the 'canonical mode par excellence' as Kress suggests.

Kress is not suggesting that the image is, or ever will, completely replace the written word. That is not his aim, and to think it is would be to miss the point. Kress is telling us that image (along with sound, moving pictures, etc.) are supplemental and add to writing. This is not to say that there is a superior/subordinate relationship; each mode is an independent form of communication that conveys messages differently. That's what multimodality is: literally, it is the use of multiple modes. Different modes work in different situations, and as rhetoricians, we need to understand those different modes. Kress takes this opportunity for another definition: “Multimodal design refers to the use of different modes – image, writing, colour [sic], layout – to present, to realize, at times to (re-)contextualize social positions and relations, as well as knowledge in specific arrangements for a specific audience” (p. 139, italics in original). By using multiple modes, we engage in multimodal design. As obvious as this sounds, it's an important idea. Once again, this is Kress showing us that the other modes add to text rather than replace it. Text remains the result of the sign-maker's choices. But Kress shows that all modes are the result of the sign maker's choices. He writes that “Texts are always multimodal, so the rhetorical and design decisions lead to the making of ensembles of modes; these are themselves assembled as orchestrations of modes, in which purposes and needs of the maker of sign-frames are brought together with affordances in the best possible manner, from the rhetor's and the designer's perspective” (p. 157, italics in original), showing us that the rhetor and the designer are often – if not always – the same person, and when they are not, they need to work in concert with one another to achieve the desired goals.

Most important here are the modes that Kress discusses. Writing and image are clearly modes, but the inclusion of color and layout draw our attention to things we might otherwise ignore. For example, looking at the text of Multimodality, we might not think about the font size, or about the margins that make it easier to read. We might not think about things like the location of pictures on the page and within the text. This seems to be one of Kress's biggest failings, which is odd, given how much attention Kress pays to the needs of the reader and the interpreter. Pictures are on different pages than the text that discusses them. There are cases (pp. 138-140) where Kress discusses an image, without mentioning that it is reproduced in the text, for two pages before actually showing it. And then there are the color plates in the middle of the book (pp. 78-79), which he refers to as examples throughout the text, both before and after the plates, referencing them only as which plate number he is speaking about. This necessitates either flipping through the book away from where he is making his argument so that the image can be seen, or ignoring the need to do so and taking his word for the parts of image he discusses. This is, at best, distracting and, at worst, rhetorically ineffective. The design and layout of a page is important, and while Kress separates them, saying that “Rhetoric is oriented to the social and political dimensions of communication; design is oriented to the semiotic” (p. 49, italics in original), in the case of his own use of images, the realm of design and rhetoric seem to be more of a Venn diagram, overlapping quite a bit. Kress is expanding our ideas of literacy, including the visual as something to be literate about as well as the written. Presumably, video and sound, two media that cannot be easily included in a book, will also fall under this expanded version of literacy.

Leaving aside the overwhelming tide of terminology, Kress seems to argue almost exclusively through anecdotes. More specifically, he tends to show us pictures that his children drew: one on page 16, two on 40, two on 55, two on 75, one on 118, 126, and 127, and two more on 180. The pictures drawn by children, whether his or others, are so prevalent that I was left wondering if Kress is suggesting that his theory can only be applied to the communication presented by children. He does suggest early on that there is a semiotic generation gap (p. 38), but I don't think he intends us to think that his theory only applies to the drawings of children. Understandably, he's trying to point to the drawings because they are so simplistic, and he can focus so clearly on specific parts of the images, however there are so many that they become distracting. Some of them, however, are far too complex for such references. The plates do have page numbers on them for where they are discussed, but this cross referencing is more of a tacit acknowledgement of the problem than it is an attempt at solution.

The anecdotes he tells along with the stories can also be very jarring. At one point (p. 135) he takes the time to talk about the potato peelers of his youth as compared to the newly designed type, and, digressing further into an examination of how the potato sits differently in his hand than it used to. These anecdotal tangents may have rhetorical purpose, but whatever that purpose may be is lost on this reviewer. Kress is a well established scholar, but these tangents threaten to detract from the authority he has.

On top of these problems is a nearly chronic ignoring of audience – ironic considering his loquaciousness on other topics. Kress sets up communication as taking place between multiple people, presumably a communicator and an audience, but he doesn't really mention audience in his argument until page 72, nearly half way through the book. And when he does mention audience, it is only as an aside, something that a sign-maker must consider when making sounds. The reaction of the audience, that entire half of the communication process, doesn't seem to be important to Kress, even though he says says that “a text is a multimodal semiotic entity, seen as 'having completeness' by those who engage with it” (p. 148, italics in original). Perhaps the problem isn't that Kress doesn't feel that audience is important; instead, the problem may be simply that he doesn't say enough about it. Whatever his reasoning for leaving audience out through so much of the book, it does present a glaring hole in his theory as it can be applied to audience.

Final word

All in all, this is a worthwhile book, an interesting project with many ideas and many, many terms that can be used for further examination into the field of social semiosis. While there are many holes in what Kress writes, those holes are not the result of sloppy argument; they are the places that Kress has opened up for further theorizing. Semiotics is, as Kress said, ever expanding, a continued process of meaning making. Multimodality is both a theoretical work and an exercise in what it theorizes about.

 

Reviewed by Joe Weinberg

University of Minnesota, Twin Cities

Reviewed by Joe Weinberg

University of Minnesota, Twin Cities