Introduction

It is evident that the way we participate in our social and professional lives, as mediated by language and culture, has drastically changed. This rapid evolution has led to the redefinition of “text,” “language,” “communication,” and “literacy.” Consequently, it has also prompted educators to reexamine how to best equip today’s students to navigate this rich, multimodal, semiotic landscape. (For a longer discussion of the socio-material realities that should inform the pedagogies we deploy, see Thorne, Hellermann, & Jakonen, 2021).

Kern (2000) argues that literacy be defined as “dynamic, culturally and historically situated practices of using and interpreting diverse written and spoken texts to fulfill particular social purposes” (p. 6). We see here the need to envision literacy as the ability to participate in the world around us, a message that constitutes the core of the New London Group’s preoccupations when they define the mission of education as “to ensure that all students benefit from learning in ways that allow them to participate fully in public, community, and economic life” (p. 60).

For students to fully engage with their lives and communities, they need the tools to interpret and understand the increasingly multimodal semiotic landscape around them. In this chapter, we unpack “multimodal literacies” by establishing the theoretical foundation for multiliteracies and examining the relationship between multiliteracies and multimodality. Second, we offer concrete examples to illustrate these concepts and approaches. We aim to show how multimodal literacy supports the goal of second and foreign language and culture^[1] learning, which is to catalyze “the development of anticipatory dispositions that enable complex, nuanced, recipient-aware, nimble and improvisational communicative capacities” (Thorne, 2011, p. 306). Achieving this goal requires learners to deliberately engage with what language means in their communities, in context, and in the world around them. Multimodality offers an apt avenue to make progress toward awareness of socially situated language, as it consists of “an orchestration of multiple modes to communicate, represent, and express meanings — [attending] systematically to the social interpretation of a wide range of communicational forms used in making meaning” (Rowsell, 2013, p. 6).

Multimodal Literacies

In this section, we define multimodality and multiliteracies as pertains to language teaching contexts. It is important to note that much of the foundational and empirical research on L2 multimodality and multiliteracies stems from Education (i.e., non-language teaching contexts), as well as English language teaching settings (see Pacheco et al., 2021, 2022 for a review of English language learning and bilingual contexts). However, the past several years have seen blossoming research on languages other than English (LOTE), the focus of this work. In what follows, we examine a handful of research findings from LOTE contexts that can be beneficial to educators hoping to implement such strategies in their classrooms.

Concepts and Definitions

Multimodality entails communicating and making meaning through “the full range of communication forms people use,” as well as the relationship between these modes (Jewitt, 2009, p. 14). The five modes typically evoked are textual, visual, audio (oral/aural), spatial, and gestural. According to Kress (2010), modes are socioculturally situated “semiotic resource[s] for making meaning” (p. 79). The same image, for example, may not be interpreted the same way across cultures or even within the same culture across different individuals.

Another important element of Jewitt’s definition of multimodality is that it posits modes as potentially interconnected and working together interactively within a given multimodal text. In other words, in the case of a multimodal sign, the combination of modes produces meaning that is often greater than or different from the meaning produced by a given individual mode. In the context of this contribution to multimodality, we envision text broadly to encompass more than simply written or verbal language but also other modes of representation (e.g., gestures, colors). Rowsell (2013) gives a vivid account of how far-reaching this take on text is, providing examples that are more immediate (e.g., film, visual signs, interface or video game design) as well as others that are too often forgotten, such as spatial design, performance (movement), or textiles. To expand from multimodality, multimodal literacy entails interpreting how different forms of communication function, how they are informed by culture, how different modalities interact to create meaning, and how to leverage modes to create and compose meaning in order to act on the world around you.

Multimodal Literacies and Multiliteracies

Multiliteracies is both a theory and a pedagogy originally proposed by the New London Group (1996) and further customized for the language learning context by Cope and Kalantzis (2009, 2015). Multiliteracies falls under the umbrella of multimodality in that it proposes communicating with multiple semiotic modes (e.g., images, sounds, texts) across multiple channels of communication (e.g., social media, texts, face-to-face conversation) across multiple contexts (e.g., formal, informal). The New London Group theorized a key dimension of multiliteracies, “designs of meaning” (p. 73), which holds three dimensions and lies at the heart of the multiliteracies approach.

First, Available Designs are the resources available to learners to interpret the world around them. For example, in a French-language classroom, an Available Design might be a French poem, music video, news article, or social media post. Designing, the second component, describes the process of harnessing the Available Design(s) to recontextualize and reconfigure multiple modes (image, text, sound, video, etc.). Designing could involve full-class discussions about the French poem, music video, or article, or other scaffolded processes of consuming and dissecting these materials. The Redesigned is the outcome of the Designing process and involves creating new meaning, which both reconfigures the set of available designs and transforms the designer of this new meaning through their agentic meaning making. Designing in a French class could involve learners designing a blog post, video reflection, or their own poem in response to the poems they viewed as Available Designs, for example. The entire multiliteracies meaning-making process is both individual and collective, a negotiation and construction of identity through the cultivation of voice, discursive repertoires, genres, and multiple modes of representation.

Previous Research on Multimodal Literacies

One way in which language learners participate in the multiliteracies acts of Available Designs, the Redesigned, and the Designing is through the semiotic practice of digital multimodal composition, which involves creating digital multimodal compositions such as digital stories (Oskoz & Elola, 2016), blogs (Ducate & Lomicka, 2008), and video blogs (vlogs) (Amgott, 2022). Creating digital multimodal compositions is an essential part of developing multimodal literacies, as such projects have been shown to support language and cultural learning (Ducate & Lomicka, 2008; Lopez et al., 2021), identity exploration (Amgott, 2022; Honeyford, 2014), critical digital literacy (de los Ríos, 2018), and identity development.

Research on multimodal projects in LOTE contexts is informed by the abundance of research on the benefits of multimodal composing in education, English as a Second Language (ESL), English as a Foreign Language (EFL), and English Language Learner (ELL) settings. In recent years, research on multiliteracies and multimodal composing has expanded in contexts of languages other than English, as described below. Multimodal projects are beneficial for promoting engagement, identity, and critical thinking in university language-learning contexts. In an intermediate German-learning context, Schmerbeck and Lucht (2017) described how learners participated in a multiliteracies curriculum involving a portfolio project in which the students viewed German advertisements (Available Designs) and reflected on their own cultural practices (Designing). Focusing on the Redesigned, Kumagai and colleagues (2015) studied Japanese language learners who designed two video projects to explain their university and life experiences. Although students were novice Japanese learners, they were able to leverage modes to communicate humor and to elaborate on their descriptions. The authors emphasized that multimodality was not a complementary part of the project, but rather that multimodality “creates new meaning” (p. 142). At the advanced level of language learning, Sagnier (2015) implemented an advanced French curriculum called Reading Images that encouraged students to interact with ads, propaganda, and other images prevalent in French media from the 19th century. The focus of this curriculum was to equip students with the steps needed to decipher the images (Available Designs) that they were interacting with, thus enhancing their skills in Designing.

Studies on high school students have additionally pointed to the benefits of implementing multimodal projects in the language classroom. Hellmich and colleagues’ (2021) study in a high school French learning context emphasizes the structure needed for implementing such projects. Students designed multimodal Instagram posts from the perspective of characters in a book excerpt they had read for class. Additionally, they collaboratively designed a multimodal video project about what the workplace would look like in the year 2070. The researchers found that the learners communicated with their peer audience, often by leveraging social media during these projects. Notably, the study highlighted the constraints involved with implementing multimodal projects, namely that some students viewed them as stressful and felt that the technology they used was superfluous to their “actual language skills” (p. 13). Research on scaffolding multimodal projects in ELL contexts (Pacheco et al. 2021) can also be applied to non-LOTE contexts to enhance alignment between student and teacher outcomes for digital multimodal projects. In another high-school learning context, Casteñeda (2013) examined how fourth-year Spanish students engaged in digital storytelling, finding that students had language and technology concerns before engaging in the project. Similar to Hellmich et al.’s (2021) study, Casteñeda found that the students were acutely aware of their audience.

In conclusion, studies on multimodal composition in university and high school contexts have illustrated the propensity of such projects to promote critical thinking, reflection on cultural practices, identity development, engagement, and connection to peers. Although research has emerged in digital multimodal composition in LOTE contexts, it still lags behind the proliferation of studies on the topic in English learning contexts. This disparity may be explained by the way that professional organizations and their standards have historically adhered to a division of language skills (speaking, reading, listening, writing, and “culture”) rather than a more multimodal, holistic approach.

Approaches, Tools, and Methodologies

When working with multimodality with the goal of examining or fostering multimodal literacy, there are two foundational elements to consider that fundamentally structure practices in both research and pedagogy. The first resides primarily in the multimodality of the material, meaning that both the instructional materials and the research data need to be multimodal. The second aspect stems from the emphasis on design and can be expressed as follows: there is no significant–or useful–distinction between learning to interpret signs and learning to make signs. This is apparent in the pedagogical approach of the multiliteracies framework: the act of interpreting is the activity of designing meaning with the ultimate goal, explicit or not, of creating new ones (redesigning) that will become available to others as available designs (the redesigned) using similar or completely different semiotic resources.

For similar reasons, conducting research on multimodality and multimodal literacy is itself a complex, multifaceted process, in which the researcher captures the recursivity, non-linearity, and layering of modes (Smith et al., 2017). For example, in their investigation of translanguaging in multimodal composition processes, Smith et al. (2017) and Smith (2018) collected computer screen recordings of students composing, video observations of the composition process, and design interviews (Dalton et al., 2015). They also collected all forms of artifacts created by the students during the composition process (e.g., visual brainstorms, drafts, final versions, metanarrative reflections).

Once collected, all these data have to be transcribed and coded to extract patterns. It is important to note here that the transcription itself is necessarily multimodal. Smith et al. (2017) build what they term “multimodal codemeshing timescapes,” which “served as an analytic tool and visual display for each student’s process for each class workshop” (p. 11) and enabled the researchers to gain a fine-grained understanding of the compositional process and patterns. These video recordings can also reveal learning moments in these microgenetic interactions (see, e.g., van Compernolle, 2014).

Even though the focus of this chapter is on teaching (with) multimodal literacy by offering models of how to unpack signs, what we hope to show is how teaching with multimodality, researching multimodality, or using it as part of action research essentially leverage the same analytical tools. Let us now turn to three examples grouped into two categories: multimodal signs in public spaces (linguistic landscape—examples #1 and #2) and multimodal compositions by LC2 students of French in study abroad contexts (example #3). We will first present each example in the form of an exercise to let you ask yourself questions or formulate hypotheses about signs and their meanings or about the potential of certain pedagogical approaches, as well as venture your own answers. Then, we will offer our own take on these examples.