Comparative Evaluative Analysis of English-Indonesian Translation Quality Using ChatGPT and Google Translate on Academic Texts Based on Ustaszewski's Evaluation Rubric
DOI:
https://doi.org/10.55909/dj3l.v4i1.74Keywords:
English–Indonesian, translation quality, academic textsAbstract
The rapid development of artificial intelligence technology has driven the use of automatic translation systems in the academic field, particularly ChatGPT and Google Translate. However, the urgency to critically evaluate the quality of both translations is important given the central role of translation in maintaining scientific integrity and textual accuracy. This study aims to analyze and compare the quality of English–Indonesian translations from both platforms based on the Ustaszewski Evaluation Rubric, which includes four main indicators: accuracy, naturalness, terminology equivalence, and sentence structure. This study employs a quantitative descriptive approach with an evaluative-comparative method. Three academic texts were translated using each platform and evaluated independently by three linguistic experts. The results indicate that ChatGPT significantly outperforms Google Translate across nearly all indicators, with an overall average score of 3.55 compared to Google Translate’s 2.58. Correlations between indicators reveal a strong positive relationship between naturalness and sentence structure (r = 0.883, p < 0.001), as well as between accuracy and naturalness (r = 0.657, p = 0.002). Meanwhile, Google Translate exhibits a tendency toward literal translations and lacks responsiveness to academic rhetorical structures. These findings indicate that ChatGPT is superior in producing contextual, natural, and cohesive academic translations, although terminological aspects still require manual validation. This study makes an important contribution to mapping the quality of machine translation based on linguistic rubrics, and offers directions for developing more accountable and applicable translation evaluation systems in the context of scientific publications.
References
Al-Ayubi, M. S. (2017). Pemanfaatan Google Translator Sebagai Media Pembelajaran Pada Terjemahan Teks Berita Asing. Jurnal Teknodik, 155. https://doi.org/10.32550/teknodik.v21i2.225
Apriyanti, C., & Shinta, U. K. D. (2021). Kesulitan Pemilihan Diksi dan Strategi dalam Penerjemahan. Jurnal Penelitian Pendidikan. https://doi.org/10.21137/jpp.2021.13.1.2
Arba, N., Widyasari, W., Efendi, Y., & Syaputri, W. (2023). Analisa Hasil Terjemahan Google Translate Dalam Lirik Lagu “To The Bone” Oleh Pamungkas. Jurnal Pembahsi (Pembelajaran Bahasa Dan Sastra Indonesia). https://doi.org/10.31851/pembahsi.v13i1.11874
Baharuddin, B., Amin, M., Thohir, L., & Wardana, L. A. (2022). Penerapan Teori Terjemahan pada Editing Hasil Terjemahan Google Translate pada Teks Akademik oleh Mahasiswa Universitas Mataram. Jurnal Ilmiah Profesi Pendidikan, 6(4), 816–824. https://doi.org/10.29303/jipp.v6i4.390
Chen, L., Chen, P., & Lin, Z. (2020). Artificial Intelligence in Education: A Review. IEEE Access. https://doi.org/10.1109/ACCESS.2020.2988510
Darwis, R., Sujaini, H., & Nyoto, R. D. (2019). Peningkatan Mesin Penerjemah Statistik dengan Menambah Kuantitas Korpus Monolingual (Studi Kasus/ : Bahasa Indonesia - Sunda). Jurnal Sistem Dan Teknologi Informasi (JUSTIN). https://doi.org/10.26418/justin.v7i1.27254
Diana, P. N., Wildaniyah, T., Oktavia, T. A. T., & Ekawati, R. (2022). Pendampingan Labelisasi Lanskap Linguistik Multilingual Destinasi Wisata Bangkalan di Era New Normal. Jurnal Ilmiah Pangabdhi. https://doi.org/10.21107/pangabdhi.v8i1.13543
Hasmaruddin, H. (2021). Linguistik dan Pengajaran Bahasa. Jurnal Ilmiah Langue and Parole. https://doi.org/10.36057/jilp.v4i2.486
Jiao, W., Wang, W., Huang, J., Wang, X., Shi, S., & Tu, Z. (2023). Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine. ArXiv.
Lee, T. K. (2024). Artificial Intelligence and Posthumanist Translation: ChatGPT versus the Translator. Applied Linguistics Review, 15(6), 2351–2372. https://doi.org/10.1515/applirev-2023-0122
Mahsun, M. S. (2014). Metode Penelitian Bahasa: Tahapan Strategi Metode dan Tekniknya. Jakarta: Raja Grafindo Persada.
Martínez, A., & Mammola, S. (2021). Specialized Terminology Reduces the Number of Citations of Scientific Papers. Proceedings of the Royal Society B: Biological Sciences. https://doi.org/10.1098/rspb.2020.2581
Mayasari, N., Dewantara, R., & Yuanti, Y. (2023). Pengaruh Kecerdasan Buatan dan Teknologi Pendidikan terhadap Peningkatan Efektivitas Proses Pembelajaran Mahasiswa di Jawa Timur. Jurnal Pendidikan West Science, 1(12). https://doi.org/10.58812/jpdws.v1i12.863
Narbuko, C., & Achmadi, A. (2021). Metodologi Penelitian. Bumi Aksara.
Padó, S., Galley, M., Jurafsky, D., & Manning, C. (2009). Robust Machine Translation Evaluation with Entailment Features. ACL-IJCNLP 2009 - Joint Conf. of the 47th Annual Meeting of the Association for Computational Linguistics and 4th Int. Joint Conf. on Natural Language Processing of the AFNLP, Proceedings of the Conf. https://doi.org/10.3115/1687878.1687922
Razak, A. (2017). Menggapai Mixed Methods Bidang Pembelajaran Bahasa Indonesia. Pekanbaru: Ababil Press.
Sahari, Y., Al-Kadi, A. M. T., & Ali, J. K. M. (2023). A Cross Sectional Study of ChatGPT in Translation: Magnitude of Use, Attitudes, and Uncertainties. Journal of Psycholinguistic Research. https://doi.org/10.1007/s10936-023-10031-y
Sanz-Valdivieso, L., & López-Arroyo, B. (2024). Google Translate vs. ChatGPT: Can non-language professionals trust them for specialized translation? https://doi.org/10.26615/issn.2683-0078.2023_008
Sêk, K. (2015). Lew Zybatow, Michael Ustaszewski (Hrsg.), Bausteine Translatorischer Kompetenz oder Was macht Übersetzer und Dolmetscher zu Profis? Innsbrucker Ringvorlesungen zur Translationswissenschaft VII (=Forum Translationswissenschaft Band 18). Peter Lang... Rocznik Przek³adoznawczy, 10, 325. https://doi.org/10.12775/RP.2015.021
Son, J., & Kim, B. (2023). Translation Performance from the User’s Perspective of Large Language Models and Neural Machine Translation Systems. Information (Switzerland). https://doi.org/10.3390/info14100574
Supsiadji, M. R., & Mirahayuni, N. K. (2021). Strategies Of Translating Literary Terms By Student Translators. PARAFRASE/ : Jurnal Kajian Kebahasaan & Kesastraan. https://doi.org/10.30996/parafrase.v21i1.5221
Ustaszewski, M. (2019). Optimising the Europarl Corpus for Translation Studies with the EuroparlExtract Toolkit. Perspectives: Studies in Translation Theory and Practice. https://doi.org/10.1080/0907676X.2018.1485716
Ustaszewski, M. (2021). Towards a Machine Learning Approach to the Analysis of Indirect translation. Translation Studies, 14(3), 313–331. https://doi.org/10.1080/14781700.2021.1894226
Wardana, L. A., Baharuddin, B., & Nurtaat, L. (2022). Kemampuan Mahasiswa Melakukan Post-Editing terhadap Hasil Terjemahan Machine Translation. Jurnal Ilmiah Profesi Pendidikan. https://doi.org/10.29303/jipp.v7i1.392
Yilmaz, E. D., Naumovska, I., & Aggarwal, V. A. (2023). AI-Driven Labor Substitution: Evidence from Google Translate and ChatGPT. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4400516
Zybatow, L. N., Stauder, A., & Ustaszewski, M. (2017). Translation Studies and Translation Practice. In Forum Translationswissenschaft.
