Generative AI and the translation of colloquial speech: A comparative study of four systems
DOI:
https://doi.org/10.24197/exbewg03Keywords:
Machine translation, colloquial speech, generative artificial intelligence, language models, translation evaluationAbstract
Large language models (LLMs) such as ChatGPT, DeepSeek, Gemini, and Claude have shown significant progress in machine translation; however, translating colloquial speech remains challenging due to idiomatic expressions, slang, pragmatic markers, and cultural references. This study compares their performance in translating forty English colloquial expressions into Spanish, grouped into four categories. A comparative, descriptive, and exploratory design was adopted, using a validated corpus and evaluation by three expert judges through a rubric assessing semantic adequacy, fluency, linguistic naturalness, and idiomatic-cultural handling. Results show Gemini achieved the highest overall performance, followed by ChatGPT, DeepSeek, and Claude. The findings highlight the need for pragmatic sensitivity, cultural mediation, and post-editing in professional translation.
Downloads
References
Achiam, Josh, Adler, Steven, Agarwal, Sandhini, Ahmad, Lama, Akkaya, Ilge, Leoni Aleman, Florencia. OpenAI. (2023). GPT-4 Technical Report. arXiv. OpenAI. (2023). GPT-4 Technical Report. arXiv. https://doi.org/10.48550/arXiv.2303.08774
Andrade Preciado, Jahiro Samar (2024). La traducción de coloquialismos en textos audiovisuales: Estudio de la adquisición de la competencia traductora (1.ª ed.). Editorial Fontamara.
Anthropic (2024). The Claude 3 Model Family: Opus, Sonnet, Haiku. Anthropic. https://www.anthropic.com
Bai, Yuntao, Kadavath, Saurav, Kundu, Sandipan, Askell, Amanda, Kernion, Jackson, Jones, Andy, Amodei, Dario (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv.. https://doi.org/ 10.48550/arXiv.2212.08073
Beeching, Kate (2016). Pragmatic Markers in British English: Meaning in Social Interaction. Cambridge University Press.
Bowker, Lynne y Buitrago Ciro, Jairo (2021). Machine Translation and Global Research: Towards Improved Machine Translation Literacy in the Scholarly Community. Emerald Publishing. https://doi.org/ 10.1108/9781800435263
Castilho, Sheila, Guerreiro, Nuno M., Marinho, Zita y Way, Andy (2023). Is neural machine translation the new state of the art? A critical analysis of quality, fluency and acceptability. Language Resources and Evaluation, 57 (2), 513-543. https://doi.org/10.1007/s10579-022-09596-4
Castilho, Sheila, Moorkens, Joss, Gaspari, Federico, Sennrich, Rico, Way, Andy y Georgakopoulou, Panayota (2018). Evaluating machine translation in a user-centered paradigm: Reconciling usability and translation quality. The International Journal of Translation and Interpreting Research, 10 (1), 1-20. https://doi.org/10.12807/ti.11 0201.2018.a01
European Commission (2022). European Master’s in Translation (EMT) Competence Framework. Publications Office of the European Union. https://commission.europa.eu/education/skills-and-qualifications/ed ucation-and-training/emt_en
Gaspari, Federico, Toral, Antonio, Naskar, Sudip Kumar, Groves, Declan y Way, Andy (2022). Perception vs. reality: Measuring machine translation post-editing effort. Machine Translation, 36 (1), 1-29. https://doi.org/10.1007/s10590-021-09266-6
Gemini Team. (2024). Gemini 1.5: Unlocking Multimodal Understanding and Long-Context Reasoning. DeepMind.
Hansen-Schirra, Silvia, Hofmann, Sascha, Nitzke, Jean y Neumann, Stella (2021). Post-Editing of Machine Translation: Processes and Applications. Routledge. https://doi.org/10.4324/9781003038179
House, Juliane (2021). Translation as communication across languages and cultures. En Sue-Ann Harding y Ovidi Carbonell Cortés (Ed.), The Routledge Handbook of Translation and Culture (pp. 11-25). Routledge. https://doi.org/10.4324/9780367820739
Kenny, Dorothy y Doherty, Stephen (2023). Ethics and governance of artificial intelligence in translation. Translation Spaces, 12 (1), 1-26. https://doi.org/10.1075/ts.22018.ken
Liu, Aixin, Dong, Zhihong, Zheng, Zihui, Ma, Chenggang, Li, Xingkai, He, Jiawei y Liu, Zhiwei (2024). DeepSeek-V2: A strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv. https://doi.org/10.48550/arXiv.2405.04434
Liu, Emmy, Chaudhary, Aditi y Neubig, Graham (2023). Crossing the threshold: Idiomatic machine translation and the role of large language models. En Bouamor, Houda, Pino, Juan y Bali, Kalika (Eds.), Proceedings of EMNLP 2023 (pp. 15095–15111). Association for Computational Linguistics.
Michel, Paul y Neubig, Graham (2018). MTNT: A testbed for machine translation of noisy text. En Riloff, Ellen, Chiang, David, Hockenmaier, Julia y Tsujii, Jun’ichi (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 543–553). Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/D18-1050
Moorkens, Joss (2020). A usability study of post-editing productivity. Machine Translation, 34 (3), 257-275. https://doi.org/10.1007/s105 90-020-09246-5
Naveen, Palanichamy y Trojovský, Pavel (2024). Overview and challenges of machine translation for contextually appropriate translations. Heliyon, 10 (2)
O’Brien, Sharon (2022). Cognitive effort in translation and post-editing. En Alves, Fabio y Jakobsen, Arnt Lykke (Eds.), The Routledge Handbook of Translation and Cognition (pp. 191-205). Routledge. https://doi.org/10.4324/9781003177984-14
Ouyang, Long, Wu, Jeffrey, Jiang, Xu, Almeida, Diogo, Wainwright, Carroll L., Mishkin, Pamela, Zhang, Chong, Agarwal, Sandhini, Slama, Katarina, Ray, Alex, Schulman, John, Hilton, Jacob, Kelton, Fraser, Miller, Luke, Simens, Maddie, Askell, Amanda, Welinder, Peter, Christiano, Paul F., Leike, Jan y Lowe, Ryan (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744. Neural Information Processing Systems Foundation, Inc.PACTE Group. (2020). Competence Levels in Translation: Working Towards a European Framework. John Benjamins.
Pramodya, Ashmari, Sakai, Yusuke, Vasselli, Justin, Kamigaito, Hidetaka y Watanabe, Taro (2025). Translating Movie Subtitles by Large Language Models using Movie-meta Information. En Zhao, Jin, Wang, Mingyang y Liu, Zhu (Eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) (pp. 315–330). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.acl-srw.22
Pym, Anthony (2021). Exploring Translation Theories (3.ª ed.). Routledge.
Toury, Gideon (2012). Descriptive Translation Studies – and Beyond. John Benjamins. (Primera edición de esta obra publicada en 1995).
Vieira, Lucas Nunes, O’Brien, Sharon y Teixeira, Carlos S. C. (2024). User perception of quality and trust in machine translation and large language models. Translation Spaces, 13 (1), 1-25. https://doi.org/ 10.1075/ts.22023.vie
Vieira, Lucas Nunes, O’Brien, Sharon y Teixeira, Carlos S. C. (2024). User perception of quality and trust in machine translation and large language models. Translation Spaces, 13 (1), 1-25. https://doi.org/ 10.1075/ts.22023.vie
Zhang, Li, Wang, Chang y Zhao, Harry (2024). Evaluating the Translation Performance of Large Language Models in Specialized Domains. arXiv. https://doi.org/10.48550/arXiv.2402.12345
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
The articles published at Hermēneus. Revista de Traducción e Interpretación will have a Creative Commons Attribution 4.0 International License (CC BY 4.0).
The journal allows the authors to retain publishing rights. Authors may reprint their articles in other media without having to request authorization, provided they indicate that the article was originally published in Hermēneus. Revista de Traducción e Interpretación.
