Keywords
Co-speech gestures
Prosody
Synchronization
Pragmatics
Information structure
How to Cite
Abstract
This paper was written with the aim of highlighting the functional and structural correlations between gesticulation and prosody, focusing on gesture / prosody synchronization in spontaneous spoken Italian. The gesture annotation used follows the LASG model (Bressem et al. 2013), while the prosodic annotation focuses on the identification of terminal and non-terminal prosodic breaks which, according to L-AcT (Cresti, 2000; Moneglia & Raso 2014), determine speech act boundaries and the information structure, respectively. Gesticulation co-occurs with speech in about 90% of the speech flow examined and gestural arcs are synchronous with prosodic boundaries. Gesture Phrases, which contain the expressive phase (Stroke) never cross terminal prosodic boundaries, finding in the utterance the maximum unit for gesture / speech correlation. Strokes may correlate with all information unit types, however only infrequently with Dialogic Units (i.e. those functional to the management of the communication). The identification of linguistic units via the marking of prosodic boundaries allows us to understand the linguistic scope of the gesture, supporting its interpretation. Gestures may be linked at different linguistic levels, namely those of: a) the word level; b) the information unit phrase; c) the information unit function; d) the illocutionary value.
References
Amir N, Vered SV, Izre'el S. Characteristics of Intonation Unit Boundaries in Spontaneous Spoken Hebrew: Perception and Acoustic Correlates. In Bel B, Marlien I. (Eds.), Proceedings of Speech Prosody 2004, ISCA, 2004, 677–680.
Austin J. How to Do Things with Words. Oxford: Oxford University Press, 1962.
Bressem J, Ladewig SH., Müller C. Linguistic Annotation System for Gestures (LASG). In Müller C, Cienki A, Fricke E. Ladewig SH., McNeill D, Teßendorf S. (Eds.), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction (Handbooks of Linguistics and Communication Science 38) Vol. 1, Berlin: De Gruyter Mouton, 2013, 1098–1125.
Beckman M, Hirschberg J, Shattuck-Hufnagel S. The original ToBI system and the Evolution of the ToBI Framework. In Sun-Ah J. (ed.), Prosodic Typology. The Phonology of Intonation and Phrasing. Oxford: Oxford University Press, 2005, 9–54.
Boersma P, Weenink D. Praat: Doing phonetics by computer. Software. 2005. Retrieved from: http://www.praat.org/.
Buhmann J, Caspers J, van Heuven V, Hoekstra H, Martens JP, Swerts M. Annotation of prominent words, prosodic boundaries and segmental lengthening by no-expert transcribers in the spoken Dutch corpus. In Gonzales Rodriguez M, Suarez Araujo C. (Eds.) Proceedings of the 2nd International Conference on Language Resources and Evaluation (LREC 2002). Paris: ELRA, 2002, 779–785.
Cantalini G. La gestualità co-verbale nel parlato spontaneo e nel recitato. Annotazione del gesto e correlati prosodici in campioni comparabili di attori italiani, PhD thesis - Università Roma Tre, 2018.
Cantalini G, Gagliardi G, Moneglia M, Proietti M. La correlazione gesto/prosodia e la sua variabilità: il parlato spontaneo di contro alla performance attorale. In: Atti del convegno: GSCP (Napoli, 12-14 Dicembre 2018), 2020.
Chafe W. Discourse, consciousness, and time: The flow and displacement of conscious experience in speaking and writing. Chicago: UCP, 1994.
Cheng W, Greaves Ch, Warren M. A Corpus-driven Study of Discourse Intonation: The Hong Kong Corpus of Spoken English. Amsterdam: Benjamins, 2005.
Cresti E. Corpus di italiano parlato. Firenze: Accademia della Crusca, 2000.
Cresti E, Moneglia M. (Eds.) C-ORAL-ROM. Integrated reference corpora for spoken romance languages. Amsterdam: Benjamins, 2005.
Cruttenden A. Intonation. Second edition. Cambridge: Cambridge University Press, 1997.
Crystal D. The English Tone of Voice. London: Edward Arnold, 1975.
Danieli M, Garrido JM, Moneglia M, Panizza A, Quazza S, Swerts M. Evaluation of Consensus on the Annotation of Prosodic Breaks in the Romance Corpus of Spontaneous Speech C-ORAL-ROM. In Lino MT, Xavier MF, Ferreira F, Costa R, Silva R. (Eds.), Proceedings of the 4th LREC Conference. Paris: ELRA, 2004, 1513–1516.
Du Bois JW, Schuetze-Coburn S, Cumming S, Paolino D. Outline of discourse transcription. In Edwards JA., Lampert MD. (Eds.): Talking data: Transcription and coding in discourse research. Hillsdale NJ: Lawrence Erlbaum, 1993, 45–89.
Duncan S. Annotative practice (under perpetual revision). Retrieved from: http://mcneilllab.uchicago.edu/pdfs/susan_duncan/Annotative_practice_REV-08.pdf, 2008.
ELAN (Version 5.9) [Computer software]. (2020). Nijmegen: Max Planck Institute for Psycholinguistics, The Language Archive. Retrieved from: https://archive.mpi.nl/tla/elan
Gagliardi G. Inter-Annotator Agreement in linguistica: una rassegna critica / Inter-Annotator Agreement in Linguistics: A Critical Review. In: Cabrio E, Mazzei A, Tamburini F. (Eds.) Proceedings of the Fifth Italian Conference on Computational Linguistics - CLiC-it 2018. (Torino, 10-12 Dicembre 2018). Torino: Accademia University Press, 2019, 206–212.
Halliday M, Kirkwood A. Intonation and Grammar in British English. The Hague: Mouton, 1967.
't Hart, J., Collier, R., Cohen, S., A Perceptual Study of Intonation. An Experimental-Phonetic Approach to Speech Melody. Cambridge: Cambridge University Press, 1990.
Helmic I, Lausberg H. Hand movements with a phase structure and gestures that depict action stem from a left hemispheric system of conceptualization. Experimental Brain Research, 232(10): 3159–3173, 2014.
Holle H, Rein R. The Modified Cohen's Kappa: Calculating Interrater Agreement for Segmentation and Annotation. In Lausberg H. (Ed.), Understanding Body Movement. A Guide to Empirical Research on Nonverbal Behaviour. With an Introduction to the NEUROGES Coding System. Frankfurt am Main: Peter Lang, 2013, 261–275.
Izre'el S, Mettouchi A. Representation of Speech in CorpAfroAs. Transcriptional Strategies and Prosodic Units. In Mettouchi A, Vanhove M, Caubet D. (Eds.) Corpus-based Studies of Lesser-described Languages: The CorpAfroAs corpus of spoken AfroAsiatic languages. Amsterdam: Benjamins, 2015, 13–41.
Izre'el S, Mello H, Panunzi A, Raso T. (Eds.), In Search of Basic Units of Spoken Language. Amsterdam: Benjamins, 2020.
Karcevsky S. Sur la phonologie de la phrase. Travaux du Cercle linguistique de Prague, IV: 188–228, 1931.
Kendon A. Some relation between body motion and speech: An analysis of an example. In Siegman, AW, Pope B. (Eds.), Studies in Dyadic Communication New York: Elsevier, 1972, 177–210.
Kendon A. Gesticulation and speech: Two aspects of the process of Utterance. In Key MR. (ed.), Nonverbal Communication and Language. The Hague: Mouton, 1980, 207–227.
Kendon A. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press, 2004.
Kita S, van Gijn I, van der Hulst H. Movement phases in signs and co-speech gestures, and their transcription by human coders. In Wachsmuth I, Fröhlich M. (eds), Gesture and Sign Language in Human-Computer Interaction Berlin: Springer, 1998, 23–35.
Krippendorff K. Content Analysis: an introduction to its Methodology. Thousand Oaks, CA: Sage Publications, 1980.
Ladewig SH, Bressem J. A linguistic perspective on the notation of gesture phases. In Müller C, Cienki A, Fricke E, Ladewig SH, McNeill D, Teßendorf S. (Eds.), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction (Handbooks of Linguistics and Communication Science 38) Vol. 1, Berlin: De Gruyter Mouton, 2013, 1060–1079.
Lausberg H. Understanding Body Movement. A Guide to Empirical Research on Nonverbal Behaviour, With an Introduction to the NEUROGES Coding System. Frankfurt am Main: Peter Lang, 2013.
Loehr D. Gesture and prosody. In Müller C, Cienki A, Fricke H, Ladewig SH, McNeill D. (Eds.), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction (Handbooks of Linguistics and Communication Science 38, Vol. 2. Berlin: De Gruyter Mouton, 2014, 1381–1391.
McClave E. Intonation and Gesture. Ph.D. dissertation. Washington, DC: Georgetown University, 1991.
McNeill D. Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press, 1992.
McNeill D. Gesture and Thought. Chicago: University of Chicago Press, 2005.
McNeill D. Why We Gesture: The Surprising Role of Hand Movements in Communication. Cambridge: Cambridge University Press, 2016.
Moneglia M, Cresti E. Intonazione e criteri di trascrizione del parlato. In Bortolini U, Pizzuto E. (a cura di) Il progetto CHILDES Italia. Pisa: Del Cerro, 1997, 57–90.
Moneglia M, Raso T, Malvessi-Mittmann M, Mello H. Challenging the perceptual relevance of prosodic breaks in multilingual spontaneous speech corpora: C-ORAL-BRASIL / C-ORAL-ROM. In Speech Prosody 2010, W1.09, Satellite workshop on Prosodic Prominence: Perceptual, Automatic Identification. Chicago. Retrieved from: https://www.isca-speech.org/archive/sp2010/sp10_2010.html, 2010.
Moneglia M, Raso T. Notes on the Language into Act Theory. In Raso T, Mello H. (eds), Spoken corpora and linguistics studies. Amsterdam: Benjamins, 2014, 468–494.
Nencioni G. Parlato-parlato, parlato-scritto, parlato-recitato. Strumenti critici, LX, 1–56, 1976.
Panunzi A, Gregori L, Rocha B. Comparing annotations for the prosodic segmentation of spontaneous speech: Focus on reference units. In Izre'el S, Mello H, Panunzi A, Raso T. (Eds.), In Search of Basic Units of Spoken Language. A corpus-driven approach. Amsterdam: John Benjamins, 2020, 403–431.
Quirk R, Greenbaum S, Leech G, Svartvik J. A Comprehensive Grammar of the English Language. London/New York: Longman, 1985.
Raso T, Mello H. (Eds.), C-ORAL-BRASIL I: Corpus de referência de português brasileiro falado informal. Belo Horizonte: Editora UFMG, 2012.
Raso T, Barbosa P, Cavalcante F, Malvessi-Mittmann M. Segmentation and analysis of the two English excerpts. In Izre'el S., Mello H., Panunzi A., Raso T. (eds), In Search of Basic Units of Spoken Language. A corpus-driven approach. Amsterdam: John Benjamins, 2020, 309–325.
Saccone V, Vieira M, Panunzi A. Complex Illocutive Unit in Language into Act Theory: an analysis of non-terminal prosodic breaks of Bound Comments and Lists. Journal of Speech Sciences, 7.2. 51–64, 2018.
Sorianello P. Per una definizione fonetica dei confini prosodici. In Pettorino M, Giannini A, Savy R. (eds) Atti del Convegno Internazionale, La comunicazione parlata. Napoli: Liguori, 2006, 310–330.
Spoken Dutch Corpus: available at: http://lands.let.ru.nl/cgn/doc_English/topics/project/pro_info.htm
WINPITCH. Software for prosodic research, with on the fly aligner, real-time spectrograph, multitracking F0 analysis, video and audio analysis, and much more (Free installation password required after 30 days of use). Available at: https://www.winpitch.com/.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright (c) 2020 Giorgina Cantalini, Massimo Moneglia
