Supplementary Files
Keywords
Brazilian portuguese
Mexican spanish
How to Cite
Abstract
The aim of this paper is to compare the multimodal production of questions in two different language varieties: Brazilian Portuguese and Mexican Spanish. Descriptions of the auditory and visual cues of two speech acts, assertions and questions, are presented based on Brazilian and Mexican corpora. The sentence “Como você sabe” was produced as an yes-no (echo) question and an assertion by ten speakers (five male) from Rio de Janeiro and the sentence “Apaga la tele” was produced as a yes-no question and an assertion by five speakers (three male) from Mexico City. The results show that, whereas the Brazilian Portuguese and Mexican Spanish assertions are produced with different F0 contours and different facial expressions, questions in both languages are produced with specific F0 contours but similar facial expressions. The outcome of this comparative study suggests that lowering the eyebrows, tightening the lid and wrinkling the nose can be considered question markers in both language varieties.
References
Abdi H, Williams, LJ. Correspondence Analysis. In: Salkind NJ (Ed.), Encyclopedia of Research Design, Thousand Oaks, CA: Sage, 2010.
Barkhuysen P, Krahmer E, Swerts M. Cross-modal and incremental perception of audiovisual cues to emotional speech. Language and speech, 53 (1), pp. 3-30, 2010.
Borràs-Comes J, Kaland C, Prieto P, Swerts M. Audiovisual Correlates of Interrogativity: A comparative analysis of Catalan and Dutch. Journal of Nonverbal Behavior, 38, pp. 53-66, 2013. DOI: https://doi.org/10.1007/s10919-013-0162-0
Borràs-Comes J, Prieto P. ‘Seeing tunes.’ The role of visual gestures in tune interpretation. Laboratory Phonology, 2 (2), pp. 355–380, 2011.
Cavé C, Guaitella I, Bertrand R, Santi S, Harlay F, Espesser R. About the relationship between eyebrow movements and F0 variations. Proceedings of the 4th International Conference on Spoken Language Processing, Philadelphia, EUA, ICSLP, pp. 2175–2179, 1996.
Cohen J. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37-46, 1960.
Crespo Sendra V, Kaland C, Swerts M, Prieto P. Perceiving incredulity: the role of intonation and facial gestures. Journal of Pragmatics, 47, pp. 1-13, 2013.
Cruz M, Swerts M, Frota S. Do visual cues to interrogativity vary between language modalities? Evidence from spoken Portuguese and Portuguese Sign Language. Proceedings of the 15th International Conference on Auditory-Visual Speech Processing 10-11 August 2019, Melbourne, Australia, pp. 1–5, 2019.
Cruz M, Swerts M, Frota S. The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties. Laboratory Phonology, 8 (1), 23, 2017.
Cruz M, Swerts M, Frota S. Variation in tone and gesture within language. Proceedings of the 18th International Congress of Phonetic Sciences, Glasgow, UK: The University of Glasgow, paper number 452, 2015.
Darwin C. The Expression of the Emotions in Man and Animals. London: J. Murray, 1872.
Debras C. The shrug: Forms and meanings of a compound enactment. Gesture, 16 (1), pp. 1–34, 2017. DOI: https://doi.org/10.1075/gest.16.1.01deb
De la Cruz-Pavía I, Werker JF, Vatikiotis-Bateson E, Gervain J. Finding phrases: The interplay of word frequency, phrasal prosody and co-speech visual information in chunking speech by monolingual and bilingual adults. Language and Speech, 2019. DOI: https://doi.org/10.1177/0023830919842353
De-la-Mota C, Butragueño PM, Orozco L, Prieto P. Mexican Spanish Intonation. In: Pietro P, Roseano P. (Org.). Transcription of Intonation of the Spanish Language. München: Lincom Europa, pp. 319-350, 2010.
Dohen M, Loevenbruck H. Interaction of audition and vision for the perception of prosodic contrastive focus. Language and Speech, 52 (2/3), pp. 177–206, 2009.
Ekman P, Friesen WV, Hager JC. The Facial Action Coding System. Salt Lake City: Research Nexus, 2002.
Field AP, Miles J, Field Z. Discovering Statistics Using R. London: SAGE Publications Ltd, 2012.
Frota S, Cruz M, Svartman FRF, Collischonn G, Fonseca A, Serra CR, Oliveira P, Vigario M. Intonational variation in Portuguese: European and Brazilian varieties. In: Frota S, Prieto P. (Org.). Intonation in Romance. 1ed. Oxford: Oxford University Press, v. 1, pp. 235-283, 2015a.
Frota S, Oliveira P, Cruz M, Vigário M. P-ToBI: tools for the transcription of Portuguese prosody. Lisboa: Laboratório de Fonética, CLUL/FLUL, 2015b. ISBN: 978-989-95713-9-6. [http://labfon.letras.ulisboa.pt/InAPoP/P-ToBI/]
Gili Fivela B. Multimodal analyses of audio-visual information: Some methods and issues in prosody research. In: Feldhausen I, Fliessbach J, Vanrell MM (Eds.). Methods in prosody: A Romance language perspective (Studies in Laboratory Phonology 4). Berlin: Language Science Press, pp. 83-122, 2018.
Gomes da Silva C. A prosódia de atos de fala no espanhol da Cidade do México. Rio de Janeiro, 2019. Tese de Doutorado em Língua Espanhola – Faculdade de Letras, Universidade Federal do Rio de Janeiro, Rio de Janeiro, 2019.
González-Fuente S. La prosodia audiovisual de la ironía verbal: un estudio de caso. Revista Española de Lingüística 45/2, pp. 73-103, 2015.
Guimarães DP. Análise prosódica de enunciados interrogativos totais de conversas coloquiais de fala espontânea na variedade mexicana. Dissertação de Mestrado em Língua Espanhola. Faculdade de Letras, Universidade Federal do Rio de Janeiro, Rio de Janeiro, 2018.
Hadar U, Steiner, TJ, Grant EC, Clifford Rose F. Head movement correlates of juncture and stress at sentence level. Language and Speech, 26, pp. 117–129, 1983.
House D. Intonational and visual cues in the perception of interrogative mode in Swedish. Proceedings of the 7th International Conference on Spoken Language Processing, Denver, Colorado, pp. 1957-1960, 2002.
Husson F, Lê S, Pagès J. Exploratory Multivariate Analysis by Example Using R. 2nd edition. Chapman & Hall/CRC, 2017.
Kaminski J, Hynds J, Morris P, Waller B. Human attention affects facial expressions in domestic dogs. Sci Rep 7, 12914, 2017. https://doi.org/10.1038/s41598-017-12781-x
Kendon A. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press, 2004.
Lalonde K, Holt RF. Preschoolers Benefit From Visually Salient Speech Cues. J Speech Lang Hear Res., 58 (1), pp. 135–150, 2014. Doi: 10.1044/2014_JSLHR-H-13-0343
Landis JR, Koch CG. The measurement of observer agreement for categorical data. Biometrics, 33 (1), pp. 159–174, 1977.
Lewkowicz DJ. Infants' Perception of the Audible, Visible, and Bimodal Attributes of Multimodal Syllables. Child Development, 71, pp. 1241-1257, 2003. doi:10.1111/1467-8624.00226
Lewkowiz DJ, Hansen-Tift, AM. Infants deploy selective attention to the mouth of a talking face when learning speech. Proceedings of the National Academy of Sciences, Jan 2012, 109 (5), pp. 1431-1436, 2012. DOI: 10.1073/pnas.1114783109
McGurk H, MacDonald J. Hearing lips and seeing voices. Nature, 264, pp. 746–748, 1976.
Miranda LS. Estudo fonético-perceptivo da entoação de enunciados assertivos, interrogativos e exclamativos do português do Brasil: uma análise multimodal. 243 f. Tese (Doutorado em Letras Vernáculas – Língua Portuguesa) – Faculdade de Letras, Universidade Federal do Rio de Janeiro, 2019.
Miranda LS. Análise da entoação do português do Brasil segundo o modelo IPO. 161f. Dissertação de mestrado (Letras Vernáculas – Língua Portuguesa) – Faculdade de Letras, Universidade Federal do Rio de Janeiro, Brazil.
Miranda L, Moraes J, Rilliard A. Statistical modeling of prosodic contours of four speech acts in Brazilian Portuguese. Proceedings of the 10th International Conference on Speech Prosody, May 25-28, Tokyo, Japan, p. 404-408, 2020a. DOI: 10.21437/SpeechProsody.2020-83.
Miranda L, Swerts M, Moraes J, Rilliard A. The role of the auditory and visual modalities in the perceptual identification of Brazilian Portuguese statements and echo questions. Language and Speech, 2020b. DOI: https://doi.org/10.1177/0023830919898886
Miranda LS, Moraes J, Rilliard A. Audiovisual perception of wh-questions and wh-exclamations in Brazilian Portuguese. Proceedings of the 19th International Congress of Phonetic Sciences, August 5-9, Melbourne, Australia, pp. 2941–2945, 2019.
Moraes J. The pitch accents in Brazilian Portuguese: analysis by synthesis. Proceedings of the 4th International Conference on Speech Prosody, Campinas, Brazil, pp. 389-397, 2008.
Moraes J. Intonation in Brazilian Portuguese. In: Hirst D, Di Cristo A. (eds.). Intonational Systems: a survey of twenty languages. Cambridge. MIT Press, 1998.
Moraes J, Rilliard A. Illocution, attitudes and prosody: a multimodal analysis. In: Raso T, Mello H. (Eds.). Spoken Corpora and Linguistic Studies. Amsterdam: John Benjamins, 2014.
Moraes J, Miranda LS, Rilliard A. Facial gestures in the expression of prosodic attitudes in Brazilian Portuguese. Proceedings of Seventh GSCP International Conference Speech and Corpora, Belo Horizonte, Brazil, pp. 157–161, 2012.
Moraes J, Rilliard A, Mota B, Shochi T. Multimodal Perception and production of attitudinal meaning in Brazilian Portuguese. Proceedings of the International Conference on Speech Prosody, 5, Chicago, USA, paper 340, 2010.
Paiva FAS, Martino JM, Barbosa PA, Benetti A, Silva IR. Um sistema de transcrição para língua de sinais brasileiras: o caso de um avatar. Revista do Gel, 13 (3), pp. 12-48, 2016.
Peres DO, Raposo de Medeiros B, Ferreira Netto W, Baia MFA. The role of the visual stimuli in the perception of prosody in Brazilian Portuguese. Proceedings of Fifth Conference on Laboratory Approaches to Romance Phonology, Somerville, MA, USA, pp. 136–141, 2011.
Pierrehumbert J. The phonology and phonetics of English intonation. Bloomington: Indiana University Linguistics Club. PhD thesis, MIT, 1980. [Published 1987 by IULC edition, Bloomington, IN.].
Prieto P, Roseano P. Prosody: Stress, Rhythm, and Intonation. In: Geeslin KL. (ed.) The Cambridge Handbook of Spanish Linguistics. Cambridge: Cambridge University Press, pp. 211-236, 2018.
Scheider L, Waller BM, Oña L, Burrows AM, Liebal K. Social use of facial expressions in hylobatids. PloS One 11, e0151733, 2016.
Schmidt KL, Cohn JF. 2018. Human Facial Expressions as Adaptations: Evolutionary Questions in Facial Expression Research. Am J Phys Anthropol. Suppl 33, pp. 3–24, 2001. Doi: 10.1002/ajpa.2001
Sosa JM. La entonación del español. Su estructura fónica, variabilidad y dialectología. Madrid: Cátedra, 1999.
Srinivasan RJ, Massaro DW. Perceiving prosody from the face and voice: Distinguishing statements from echoic questions in English. Language and Speech, 46 (1), pp. 1–22, 2003.
Swerts M, Krahmer E. Congruent and incongruent audiovisual cues to prominence. In: Bel B, Marlin I. (Eds.), Proceedings of 2nd International Conference on Speech Prosody, Nara, Japan, pp. 69-72, 2004.
Torreira F, Valtersson E. Phonetic and visual cues to questionhood in French. Phonetica, 72, pp. 20-42, 2015.
Vegas Pro software, Version 14 of Vegas Pro. Copyright © [2016] MAGIX. Software available at: https://www.vegascreativesoftware.com/.
Waller BM, Caeiro CC, Davila-Ross M. Orangutans modify facial displays depending on recipient attention. PeerJ 3, e827, 2015.
Waller BM, Peirce K, Caeiro CC, Scheider L, Burrows AM, McCune S, Kaminski J. Paedomorphic facial expressions give dogs a selective advantage. PloS one, 8 (12), e82686, 2013.
Willis, EW. Tonal levels in Puebla Mexico Spanish declaratives and absolute interrogatives. In: Gess R, Rubin EJ (Eds.), Theorical and experimental approaches to Romance languages, pp. 351–363, 2005.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright (c) 2020 Luma da Silva Miranda, Carolina Gomes da Silva, João Antônio de Moraes, Albert Rilliard
