My research starts from the recognition of the importance of teaching media literacy in Korean—a form of literacy which is becoming an essential part of language education today, since knowing how to “read” sophisticated layers of multimodal communication in the written mode of technology-mediated communication is becoming more and more a necessary skill for reading text in general, technology-mediated or not. For my project, I looked at a unique kind of media literacy in Korean. Specifically, I examined what I term “graphic overtitles” in Korean TV shows. In contrast to the traditional subtitles (which usually represent a direct captioning of the speech of the people in the movie) that are shown at the bottom of the screen for deaf or foreign viewers, “graphic overtitles” take on varied forms in terms of typography, colors, size and visual structuring, often appearing all over the screen rather than at the bottom. They also differ from traditional subtitles in that they represent selective utterances that participants make and use diverse semiotic features other than written language. They can be seen as a producer’s mediation in the form of active participation in as well as reporting of the activity of a show through written language as well as graphic images and symbols.
“Graphic overtitles” deserve attention in that (1) they represent multi-voicedness (Bakhtin, 1981) of an utterance in multileveled participation framework, (2) they play a part in the co-constructing of specific sociocultural genre/activity of TV-mediated discourse and add a new dimension to the show, serving the goals of entertainment, (3) as a written communication like the Internet and comic books, they illustrate many features of genre-specific usage of semiotic resources, such as graphic features using Hangul keystrokes and emoticons and cartoon-like symbols and images, and finally, (4) the understanding of this new semiotic dimension will play an important role in teaching the the same, leading to students better grasping the relationship between language and genre/discourse and better understanding the relationship between language and culture. In this article I will provide a brief summary of my project by illustrating semiotic resources and linguistic features frequently found in graphic overtitles in Korean TV shows. After an examination of graphic overtitles, I will make suggestions for teaching this genre to Korean-learning students and for how to appropriately view this new written mode of communication.
For the data of analysis, I used two different shows in Korean. One is called “Yasimmanman” (YSMM) (“We asked 10,000 people online”) in which the guests have to guess the responses to a question that has been asked of about 10,000 people online. The other show is “Mwuhantocen” (“Infinite Challenge”), in which 6 comedians challenge an adventurous activity or a game in each episode. Pictures 1 and 2 above give an example of graphic overtitles from each show.
Analysis
For the analysis, I will summarize semiotic resources for graphic overtitles observed in Korean TV shows and list linguistic features frequently found in graphic overtitles. Semiotic resources for graphic overtitles can be divided into two types: textual and visual.
Textual semiotic resources for graphic overtitles
- Typography – letter forms, color, texture, movement, three-dimensionality
- Punctuation marks – three dots, question marks, exclamation marks, tilde
Visual semiotic resources for graphic overtitles
- Emoticons (from Internet communication)
- Comic book symbols
- Cartoon-like pictures
In graphic overtitles, we can see that different typography is used to express different voices. For example, the block font is used mostly in commentaries or summaries by the producer, while the handwritten-style font is used in quotations and in describing emotions. The color and size variations with many punctuation marks are also effectively employed in delivering the delicate nuances or stances of the participants. The use of primary colors to emphasize the meaning of the utterances and the use of black and white to express the producer’s voice are also noticeable. Question marks and exclamation marks are used to express curiosity and surprise (or emphasis), respectively. Tilde is seen as an effort to convey the intonational characteristics of the utterance. Three dots represent something is to left to be said and sometimes suggest an unspoken reaction that the participant must be feeling to some surprising or strange situation.
Visual aspects of graphic overtitles include use of emoticons, comic book symbols and cartoon-like pictures (shown in picture 2). In picture 1, we see ‘^^;;’ (smiling eyes with sweat beads), which is an emoticon frequently used in Korean Internet communication to depict embarrassment in a light way. The computer keyboard has also given rise to a shorthand involving Hanguel, characters from the Korean alphabet, to illustrate the emotions of the speakers on the Internet. Some examples of these are shown below.
- ㅜㅜ the use of a vowel ‘ㅜ’ (sound of ‘u’) to depict a crying face
- ㅠㅠ the use of a vowel ‘ㅠ’ (sound of ‘yu’) to depict a crying face with more tears
- ㅋㅋ the use of a consonant ‘ㅋ’ (sound of ‘k’) to describe giggling
- ㅎㅎ the use of a consonant ‘ㅎ’ (shound of ‘h’) to describe a laughter
And we can see this Internet semiotic mode on the TV screen. Picture 3, the use of ㅋㅋ’ to illustrate the two participants’ giggling when they are watching the other participant’s comic interview, and picture 4, the use of ㅠㅠ when the producer depicts how touched (almost to the point of tears) he was when he heard that a famous actress (shown on the screen) said she’d been watching the show, are from Infinite Challenge.
The use of emoticons and the Korean alphabet as pictographs shows the extension of Internet-specific semiotic signs to another communicative context, asserting the validity of these pictorial signs as a new semiotic mode on their own.
Another intertextual usage of semiotic resources is observed in the comic book symbols and cartoon-like pictures found in graphic overtitles. The use of speech balloons and the frequent use of onomatopeia (sound symbolic words) with punctuation marks suggest that the growth of graphic overtitles has been influenced by the conventions of comic books.
Pictures 5 to 8 all manifest common linguistic and symbolic resources found in comic books. Picture 7 shows a drawing of sweat beads next to a host of Infinite Challenge, when his elementary school grades are revealed. His grades were pretty bad, and he seems to be little embarrassed; and, at that moment, the sweat beads appear within the figure of a cloud. And the ‘anger cross’ depicted in picture 8 appears when an interviewer says that the man on the screen is old. Sweat beads and anger crosses like these are used on these TV shows in the same ways they are used in comic books. (see picture 6)
In addition to textual and visual semiotic resources, there are some linguistic features frequently found in graphic overtitles. These linguistic features include sound symbolic words (or onomatopeia), stance-marking words, and headless relative clauses. Sound symbolic words are words that mimic sounds to depict physical and/or mental state. In picture 5, the words on the graphic overtitle “kkung” describes the “sound” people make when they are doing something that is really hard or difficult; and, in the picture, the person is eating really spicy kimchi, and the overtitle depicts the difficulty he’s having with it. The frequent use of sound symbolic words is a distinctive feature of comic books and we can see an example in picture 6, in which a woman is working on a very difficult essay, and the sound symbolic word “kkung” is used to describe her physical and mental state.
In graphic overtitles, we also many stance-marking words that are attached to speakers or to the utterances that the speakers make. These words describe the speaker’s emotive stance, which the producer is evaluating. In picture 9, after a participant in YSMM discovers his answer is not in the top 5 list, he asks, “It’s not on the list?” At this point, the word “unexpectedness” shows up on the screen to mark the speaker’s stance. In picture 10, one of the main hosts of Infinite Challenge asks the producer (who is off-screen) why he didn’t tell the hosts who’s coming on the show, and, over his utterance, which is in a speech balloon, the word “complaint” is seen, manifesting the host’s feelings at that moment.
The other linguistic feature that is salient in the graphic overtitles is the use of a headless relative clause, i.e., a relative clause that lacks the noun that it modifies. For example, the phrase “a girl who is pretty” is beheaded to read simply “…who is pretty.”
In picture 11, the graphic overtitle reads, “…got bowled over” below “big laughter.” On the show the participants are laughing out loud because of another participant’s story about his grades in elementary school, and instead of the words “the co-hosts who got bowled over,” the words “got bowled over” shows up on the screen. This use of a seemingly ungrammatical construction can be argued to be an adaptation to the new communicative context in the form of graphic overtitles on a TV screen. As you can actually view the modified entity on the screen, you can omit the modified entity in a linguistic construction.
I talked briefly about the semiotic resources- textual, visual, and linguistic – used on the new semiotic mode of graphic overtitles in Korean TV shows. Now I want to address more basic questions like what and how. What do graphic overtitles do? I argue that they bring the interaction close to the home audience by simulating a conversation between explicit (those who can be shown on TV) and implicit (the home audience, and the producer) participants of the show, and evolve the relationship between the audience and the show. And this evolving relationship between the show and the home audience sometime appears in a very interesting form. When Infinite Challenge aired an episode without graphic overtitles because of a union strike, the fan club for the show created their own graphic overtitles and uploaded the augmented video on the Internet. They said these “subtitles” are like a 7th member of the show. This anecdote casts light on how important the graphic overtitles are to the show. And the question of how these new practices came about is something to be studied further; but the use of graphic overtitles demonstrates how people adapt to a new communicative context involving advanced technology and how they adopt available semiotic resources from similar discourses, in this case, from the Internet and comic books.
Suggestions for Teaching Korean
Finally, I’d like to make a few suggestions for incorporating graphic overtitles into a Korean language curriculum. I surveyed 202 students (81 beginning level students, 55 intermediate level students, 29 advanced level students, and 27 4th and 5th level students) on the subject of graphic overtitles. I asked what the students thought about graphic overtitles after I showed a couple of clips from Korean TV shows. With respect to the positive aspects of graphic overtitles, the beginning and intermediate level students wrote about the addition of humor to the show and the educational effect of the overtitles. By educational effect, they meant that graphic overtitles helped contextualize the show and conveyed aspects of Korean culture. With respect to negative aspects, students said that graphic overtitles went by too fast and were confusing, and sometimes seemed childish. It is interesting to contrast these observations with those from more advanced Korean students. 4th year and 5th year Korean learners mentioned the creative aspects of the graphic overtitles and the educational effect. For negative aspects, they wrote about the overuse of graphic overtitles, possible manipulation by the producer, and limitations on audience interpretation. As the more advanced learners understand more, they seem to recognize more about the possible pluses and minuses of graphic overtitles, offering greater opportunity for in-depth discussion about the use of graphic overtitles. The following are a few suggestions for teaching Korean at each level using clips from Korean TV shows with graphic overtitles. More specific procedures will evolve over time, but for now these should suffice to help uncover the rich potential of graphic overtitles for teaching Korean language and culture in this age of technology-mediated communication.
Beginning level students
- awareness-raising of graphic overtitles
Intermediate level students
- use of clips without sounds for speaking and writing practice
- learning stance-marking words and sound symbolic words
- learning culture
Advanced level students
- learning more difficult stance-marking words and sound symbolic words
- creating their own graphic overtitles
- discussion of graphic overtitles
Conclusion
I have examined a new type of technology-mediated communication in Korea: graphic overtitles in Korean TV shows. This examination has revealed that various semiotic resources, including those found on the Internet and in comic books, are in play in the emergence of a new genre of written discourse composed of words and images in the form of graphic overtitles. Graphic overtitles add a new semiotic dimension to the multi-faceted social world of Korean TV shows, being inseparable from the shows themselves; and I argue that they are the product of people’s adaptation to a new communicative context to express different voices/stance in a multilevel participation framework. As I implied in the introduction, this new form of literacy should be integrated into the teaching of Korean language and culture, as proper understanding of how visual/graphic aspects of written language in technology-mediated communication is essential not just to get information but also to express one’s opinion and feelings and to communicate with others appropriately in a specific sociocultural discourse.
References
Bakhin, M. (1981). The Dialogic imagination: four essays. Austin: University of Texas Press.
Crystal, D. (2001). Language and the Internet. Cambridge: Cambridge University Press.
Finnegan, R. (1988). Literacy and orality: Studies in the technology of communication. Oxford: Basil Blackwell Ltd.
Goffman, E. (1981). Footing. In E. Goffman (Eds.), Forms of Talk (pp. 124-159). Philadelphia: University of Pensylvania Press.
Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 2, 1489-1522.
Goodwin, C & Goodwin, M. (2000). Emotion within situated activity. In A. Duranti (Eds.), A linguistic anthropology: A reader (pp. 239-257). Oxford: Blackwell.
Hutchby, I. (2001). Conversation and technology: From the telephone to the Internet. Cambridge: Polity Press.
Jewitt, D. & Kress, G. (Eds.) (2003). Multimodal literacy. New York: Peter Lang Publishing, Inc.
Kress, G. & Leeuwen, T.V. (2001). Multimodal discourse: The modes of media of contemporary communication. London: Arnold publishers/Oxford University Press.
Leeuwen, T.V. (2005). Introducing Social Semiotics. New York: Routledge.
Levine, P & Scollon, R. (Eds.) (2004). Discourse and technology: Multimodal discourse analysis. Washington D.C.: Georgetown University Press.
McLuhan, M. (1964). Understanding media: The extension of man. New York: McGraw-Hill.
Norris, S. & Jones, R. T. (Eds.) (2005). Discourse in Action. New York: Routledge Saraceni, M. (2003). The Language of Comics. New York: Routledge.