Developing an AI-Generated Language-Specific Epistemic Justification Inventory: A Preliminary Validation Using a Machine-Translation Error-Detection Task
Foreign-language learners now treat generative AI as a routine source of information about how a language works, even though what these tools produce is fluent yet not always correct. How learners decide whether to trust such output (in epistemic terms) has therefore become a practical concern for language education. Adapting Bråten and colleagues’ Internet-Specific Epistemic Justification Inventory (ISEJI), we wrote a 12-item inventory covering three theorized ways of justifying AI-generated language information: against one’s own knowledge, by cross-checking several sources, and by appeal to authority. Eighty-four non-language-major undergraduates completed it alongside a machine-translation (MT) error-detection task. Internal consistency was reasonable for a first draft (overall α = .79; subscales .68–.75). An exploratory factor analysis recovered the personal-justification items cleanly but folded the multiple-sources and authority items together, hinting that in the AI setting learners may distinguish only between internal and external grounds for belief. More usefully, total scores told apart students who caught the translation error from those who missed it (4.98 vs. 4.65; t = 2.08, p = .041; d = 0.47), and a one-SD rise in justification raised the odds of a successful catch by roughly 58% (OR = 1.58). We read these results as encouraging but preliminary: the sample is small, the structure is unsettled, and the criterion is thin. The inventory is offered as a starting point, not a finished scale.
[1] Xu J, 2025, Harnessing AI in Academic Writing: The Complex Interplay of AI Literacy and Self-Directed Learning Among University L2 Learners. System, 135: 103855.
[2] Bråten I, Brandmo C, Kammerer Y, 2019, A Validation Study of the Internet-Specific Epistemic Justification Inventory with Norwegian Preservice Teachers. Journal of Educational Computing Research, 57(4): 877–900.
[3] Bråten I, Strømsø HI, Samuelstuen MS, 2005, The Relationship Between Internet-Specific Epistemological Beliefs and Learning Within Internet Technologies. Journal of Educational Computing Research, 33(2): 141–171.
[4] Rind IA, Bhatti J, Chellappan K, 2026, From Blind Trust to Critical Inquiry: Epistemic Beliefs and Student Engagement with ChatGPT in Higher Education. Jurnal Kejuruteraan.
[5] Horbach A, Pehlke J, Laarmann-Quante R, Ding Y, 2023, Crosslingual Content Scoring in Five Languages Using Machine-Translation and Multilingual Transformer Models. International Journal of Artificial Intelligence in Education.
[6] Bråten I, Britt MA, Strømsø HI, Rouet JF, 2011, The Role of Epistemic Beliefs in the Comprehension of Multiple Expository Texts: Toward an Integrated Model. Educational Psychologist, 46(1): 48–70.
[7] Ståhl T, 2024, Justification for Knowing in a Digitalised Landscape: An Empirical Exploration. Journal of Digital Social Research, 6(3): 59–77.
[8] Maggioni L, Parkinson MM, 2008, The Role of Teacher Epistemic Cognition, Epistemic Beliefs, and Calibration in Instruction. Educational Psychology Review, 20(4): 445–461.
[9] Guilfoyle L, McCormack O, Erduran S, 2024, The Role of Research Experiences in Developing Pre-Service Teachers' Epistemic Beliefs. Teaching and Teacher Education, 144: 104599.
[10] Ng DTK, Wu W, Leung JKL, Chiu TKF, Chu SKW, 2023, Design and Validation of the AI Literacy Questionnaire. British Journal of Educational Technology, 55(3): 1082–1104.
[11] Carolus A, Koch MJ, Straka S, et al., 2023, MAILS – Meta AI Literacy Scale. Computers in Human Behavior: Artificial Humans, 1(2): 100014.
