An Empirical Evaluation of Generative AI Integration in Instructional Material Development: Its Impact on Teacher Performance and the Moderating Role of Digital Literacy
DOI:
https://doi.org/10.37251/ijoer.v7i3.2903Keywords:
Digital Literacy, Educational Evaluation, Generative AI, SEM-PLS, Teacher Performance, Teaching MaterialsAbstract
Purpose of the study: The use of generative artificial intelligence (AI) in education is growing rapidly, particularly in the development of instructional materials. However, research providing an empirical evaluation of its effectiveness on teacher performance—taking digital literacy into account as a readiness factor—remains limited. This study is an evaluative research aimed at assessing the effectiveness of using generative AI in instructional material development on teacher performance, as well as evaluating the moderating role of digital literacy within the framework of educators’ professional accountability.
Methodology: This study employs a quantitative approach using an explanatory survey design involving teachers from formal educational institutions in Lhokseumawe City. Data were collected via a Likert-scale questionnaire and analyzed using Partial Least Squares-Structural Equation Modeling (SEM-PLS) through SmartPLS 4 software to evaluate the structural relationships among variables.
Main Findings: The analysis results provide evaluative evidence that the use of generative AI significantly contributes to improving teachers’ performance in the aspect of instructional material development. Digital literacy was also found to have a positive and significant effect on performance, with a more dominant influence. However, the evaluation results indicate that digital literacy does not moderate the relationship between the use of generative AI and teacher performance, suggesting that the effectiveness of AI is independent of the level of digital literacy in this sample.
Novelty/Originality of this study: This study contributes to the field of educational evaluation by offering a framework for assessing evidence-based technology integration. The implications for evaluation practice are the need to develop teacher performance assessment standards that include digital technology competencies as part of professional accountability. These results can serve as a policy foundation for educational institutions in evaluating the effectiveness of technology training programs to ensure the adoption of AI that impacts instructional quality.
References
S. Zaimoğlu and A. Dağtaş, “Teacher Cognition and Practices in Using Generative AI Tools to Support Student Engagement in EFL Higher-Education Contexts,” Behav. Sci. (Basel)., vol. 15, no. 9, p. 1202, Sep. 2025, doi: 10.3390/bs15091202.
A. Yadav, M. Lachney, A. Hu, and L. Tavernier, “Integrating Generative AI in Teacher Education: Pre-Service Teachers’ Perspectives, Attitudes, and Design Challenges,” J. Res. Child. Educ., pp. 1–22, Dec. 2025, doi: 10.1080/02568543.2025.2581712.
K. Acevedo, “Exploring the Impact of Generative AI to Mitigate Educator Burnout,” Abilene Christian University, 2025.
R. C. Kerr and H. Kim, “From Prompts to Plans: A Case Study of Pre-Service EFL Teachers’ Use of Generative AI for Lesson Planning,” English Teach., vol. 80, no. 1, pp. 95–118, Mar. 2025, doi: 10.15858/engtea.80.1.202503.95.
M. Nelly and Yusdiana, “Application of Albert Bandura’s Social-Cognitive Theories in Teaching and Learning,” Edukasi Islam. J. Pendidik. Islam, vol. 12, no. 02, p. 2131−2146, 2023, doi: 10.30868/ei.v12i02.4585.
I. P. Soko and D. D. Samo, “The Analysis of In-Service Teachers’ Practices of Implementing Technological Pedagogical Content Knowledge (TPACK),” Eur. J. Educ. Pedagog., vol. 4, no. 2, pp. 64–71, Mar. 2023, doi: 10.24018/ejedu.2023.4.2.585.
P. N. Widowati and R. Purbojo, “Pengaruh Kompetensi Integrasi Teknologi, Konten Dan Pedagogik (TPACK), Motivasi Kerja Dan Teachers’ Work Engagement Terhadap Kinerja Guru SD XYZ Cabang Bekasi,” Cetta J. Ilmu Pendidik., vol. 7, no. 3, pp. 216–226, Jul. 2024, doi: 10.37329/cetta.v7i3.3137.
L. Chang, “Reconstruction of Writing instruction with Generative AI in SAMR Framework: A Multimodal Empirical Study,” in Proceedings of the 2nd Guangdong-Hong Kong-Macao Greater Bay Area Education Digitalization and Computer Science International Conference, New York, NY, USA: ACM, Apr. 2025, pp. 877–881. doi: 10.1145/3746469.3746605.
M. Cheng, “Quality in higher education: Developing a virtue of professional practice,” Qual. High. Educ. Dev. a Virtue Prof. Pract., pp. 1–104, 2016, doi: 10.1007/978-94-6300-666-8.
I. Celik, M. Dindar, H. Muukkonen, and S. Järvelä, “The Promises and Challenges of Artificial Intelligence for Teachers: a Systematic Review of Research,” TechTrends, vol. 66, no. 4, pp. 616–630, Jul. 2022, doi: 10.1007/s11528-022-00715-y.
OECD, OECD Digital Education Outlook 2023. in OECD Digital Education Outlook. OECD Publishing, 2023. doi: 10.1787/c74f03de-en.
R. Luckin, Machine Learning and Human Intelligence: The Future of Education for the 21st Century. 2018.
O. Zawacki-Richter, V. I. Marín, M. Bond, and F. Gouverneur, “Systematic review of research on artificial intelligence applications in higher education – where are the educators?,” Int. J. Educ. Technol. High. Educ., vol. 16, no. 1, p. 39, Dec. 2019, doi: 10.1186/s41239-019-0171-0.
L. Darling-Hammond, Empowering Teachers: Lessons from Five High-Performing Systems. San Francisco, CA / New York: Jossey Bass, 2017.
A. Adnan, A. Zohriah, and A. Muín, “Evaluasi Kinerja Tenaga Pendidik,” JIIP - J. Ilm. Ilmu Pendidik., vol. 7, no. 2, pp. 1463–1468, 2024, doi: 10.54371/jiip.v7i2.3446.
C. Dalsgaard and T. Ryberg, “A theoretical framework for digital learning spaces: learning in individual spaces, working groups, communities of interest, and open connections,” Res. Learn. Technol., vol. 31, Sep. 2023, doi: 10.25304/rlt.v31.3084.
E. Yeşilyurt and R. Vezne, “Digital literacy, technological literacy, and internet literacy as predictors of attitude toward applying computer-supported education,” Educ. Inf. Technol., vol. 28, no. 8, pp. 9885–9911, Aug. 2023, doi: 10.1007/s10639-022-11311-1.
N. Hidayati and F. Nugrahani, “Pengaruh Kemampuan Berpikir Kritis dan Minat Baca Terhadap Kemampuan Literasi Digital Pendahuluan,” vol. 13, no. 3, pp. 3201–3212, 2024, doi: 10.58230/27454312.760.
D. T. K. Ng, J. Su, J. K. L. Leung, and S. K. W. Chu, “Artificial intelligence (AI) literacy education in secondary schools: a review,” Interact. Learn. Environ., vol. 32, no. 10, pp. 6204–6224, Nov. 2024, doi: 10.1080/10494820.2023.2255228.
Daniel L. Stufflebeam and Chris L. S. Coryn, Evaluation theory, models, and applications, Edition-2. San Francisco: San Francisco : Jossey-Bass & Pfeiffer Imprints, Wiley, 2014. Accessed: Apr. 26, 2026.
A. Tlili et al., “What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education,” Smart Learn. Environ., vol. 10, no. 1, p. 15, Feb. 2023, doi: 10.1186/s40561-023-00237-x.
R. Scherer, S. K. Howard, J. Tondeur, and F. Siddiq, “Profiling teachers’ readiness for online teaching and learning in higher education: Who’s ready?,” Comput. Human Behav., vol. 118, p. 106675, May 2021, doi: 10.1016/j.chb.2020.106675.
C. Lang, G. Siemens, A. F. Wise, D. Gasevic, and A. Merceron, Handbook of Learning Analytics, Second. Society for Learning Analytics Research (SoLAR), 2022. doi: 10.18608/hla17.
J. W. Creswell and D. C. J, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 5th ed. Los Angeles: SAGE Publications, 2018.
J. F. Hair, J. J. Risher, M. Sarstedt, and C. M. Ringle, “When to use and how to report the results of PLS-SEM,” Eur. Bus. Rev., vol. 31, no. 1, pp. 2–24, Jan. 2019, doi: 10.1108/EBR-11-2018-0203.
H. Latan and I. Ghozali, Partial Least Squares: Concepts, Methods and Applications using WarpPLS 4. 2021.
J. Blake, “Unleashing the Potential,” in Generative AI in Teaching and Learning, Japan: IGI Global, 2023, ch. 2, pp. 31–45. doi: 10.4018/979-8-3693-0074-9.ch002.
C. L. M. FX. Risang Baskara, “Mengoptimalkan Reciprocal Teaching Dengan Generative AI: Kerangka Teori ... - FX. Risang Baskara, Concilianus Laos Mbato - Google Buku,” in E Book, M. Budiraharjo, Ed., Yogyakarta: Sanata Dharma University Press, 2024. Accessed: Dec. 20, 2025.
P. MacDowell, K. Moskalyk, K. Korchinski, and D. Morrison, “Preparing Educators to Teach and Create With Generative Artificial Intelligence,” Can. J. Learn. Technol., vol. 50, no. 4, pp. 1–23, Nov. 2024, doi: 10.21432/cjlt28606.
I. Molenaar, “Towards hybrid human-AI learning technologies,” Eur. J. Educ., vol. 57, no. 4, pp. 632–645, Dec. 2022, doi: 10.1111/ejed.12527.
C. N. Prilop, D.-K. Mah, L. J. Jacobsen, R. R. Hansen, K. E. Weber, and F. Hoya, “Generative AI in teacher education: Educators’ perceptions of transformative potentials and the triadic nature of AI literacy explored through AI-enhanced methods,” Comput. Educ. Artif. Intell., vol. 9, p. 100471, Dec. 2025, doi: 10.1016/j.caeai.2025.100471.
W. Holmes and I. Tuomi, “State of the art and practice in AI in education,” Eur. J. Educ., vol. 57, no. 4, pp. 542–570, Dec. 2022, doi: 10.1111/ejed.12533.
F. Miao, W. Holmes, R. Huang, and H. Zhang, AI and education: guidance for policy-makers. UNESCO, 2021. doi: 10.54675/PCSP7350.
A. Matere, “Effectiveness of Artificial Intelligence Tools in Teaching and Learning in Higher Education Institutions in Kenya,” J. Kenya Natl. Comm. UNESCO, vol. 5, no. 1, Dec. 2024, doi: 10.62049/jkncu.v5i1.177.
F. A. Wilujeng and Setiyawan, “The Effect of Digital Literacy on Teacher Performance With Creativity and Professionality As An Intervening Variable Fitri,” IJAEM, vol. 5, no. 12, pp. 328–336, 2023, doi: 10.35629/5252-0512328336.
Z. Mao, S. Tong, C. Jiang, S. Yan, and Y. Bai, “How university teachers’ digital literacy influences their innovative ability: a system dynamics theoretical modeling and simulation study,” Front. Psychol., vol. 16, Feb. 2026, doi: 10.3389/fpsyg.2025.1665337.
K. Sperling, C.-J. Stenberg, C. McGrath, A. Åkerfeldt, F. Heintz, and L. Stenliden, “In search of artificial intelligence (AI) literacy in teacher education: A scoping review,” Comput. Educ. Open, vol. 6, p. 100169, Jun. 2024, doi: 10.1016/j.caeo.2024.100169.
H. Du, Y. Sun, H. Jiang, A. Y. M. A. Islam, and X. Gu, “Exploring the effects of AI literacy in teacher learning: an empirical study,” Humanit. Soc. Sci. Commun., vol. 11, no. 1, p. 559, May 2024, doi: 10.1057/s41599-024-03101-6.
O. V. Sergeeva, A. R. Masalimova, M. R. Zheltukhina, L. S. Chikileva, L. Y. Lutskovskai, and A. Luzin, “Impact of digital media literacy on attitude toward generative AI acceptance in higher education,” Front. Educ., vol. 10, Jun. 2025, doi: 10.3389/feduc.2025.1563148.
F. ul Haq, M. Asim, N. M. Suki, N. Zakaria, and S. Hussain, “AI Adoption and Educational Effectiveness in Emerging Higher Education Institutions: The Moderating Role of Digital Literacy and Institutional Support,” J. Inf. Knowl. Manag., vol. 25, no. 03, Mar. 2026, doi: 10.1142/S021964922550090X.
H. Yaseen, A. S. Mohammad, N. Ashal, H. Abusaimeh, A. Ali, and A.-A. A. Sharabati, “The Impact of Adaptive Learning Technologies, Personalized Feedback, and Interactive AI Tools on Student Engagement: The Moderating Role of Digital Literacy,” Sustainability, vol. 17, no. 3, p. 1133, Jan. 2025, doi: 10.3390/su17031133.
C. K. Lo, “What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature,” Educ. Sci., vol. 13, no. 4, p. 410, Apr. 2023, doi: 10.3390/educsci13040410.
D. Siahaan, A. N. Azhari, and D. S. Soegoto, “Transformasi Digital dalam Pendidikan dan Produktivitas Kerja Pendidik : Peran Artificial Intelligence sebagai Enabler Efisiensi Pedagogi efisiensi pedagogi dalam hubungan antara transformasi digital pendidikan dan produktivitas menyeluruh terhadap metode,” J. Ilm. Manaj. dan Kewirausahaan, vol. 5, 2026, doi: 10.55606/jimak.v5i2.6730.
T. Rachbauer, J. Graup, and E. Rutter, “Digital literacy and artificial intelligence literacy in teacher training,” Forum Educ. Stud., vol. 3, no. 1, pp. 1842–1842, Mar. 2025, doi: 10.59400/FES1842.
F. Miao and M. Cukurova, AI competency framework for teachers. France: UNESCO, 2024. doi: 10.54675/ZJTE2084.
X. Tan, G. Cheng, and M. H. Ling, “Enhancing teachers’ AI competency: A professional development intervention study based on intelligent-TPACK framework,” Comput. Educ. Artif. Intell., vol. 9, p. 100521, Dec. 2025, doi: 10.1016/j.caeai.2025.100521.
I. Tenberga and L. Daniela, “Artificial Intelligence Literacy Competencies for Teachers Through Self-Assessment Tools,” Sustainability, vol. 16, no. 23, p. 10386, Nov. 2024, doi: 10.3390/su162310386.
E. Tipton, K. Hallberg, L. V. Hedges, and W. Chan, “Implications of Small Samples for Generalization: Adjustments and Rules of Thumb,” Eval. Rev., vol. 41, no. 5, pp. 472–505, Oct. 2017, doi: 10.1177/0193841X16655665.
L. Parker, A. J. Loper, C. W. Carter, J. Hayes, and A. Karakas, “Longitudinal insights into AI in education: Usage, ethics, and policy development in higher education,” Comput. Educ. Open, vol. 10, p. 100329, Jun. 2026, doi: 10.1016/j.caeo.2025.100329.
McGehee Nikolas, “Artificial Intelligence and Student Usage in Online Learning: A Longitudinal Analysis of Usage Patterns, Achievement, and Perceptions in K-12 Virtual Education | Michigan Virtual,” Michigan Virtual Learning. Accessed: May 01, 2026.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Muhammad Dhuha Masyhuri, Muhammad Ilham
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and acknowledge that the Indonesian Journal of Education Research (IJoER) is the first publisher licensed under a Creative Commons Attribution 4.0 International License.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges and earlier and greater citation of published work.
.png)
.png)
