Hybrid Modeling to Predict the Impact of Generative AI on Students' Learning Retention

Abstract

The growing adoption of Generative Artificial Intelligence (GenAI) technology in the field of education has sparked global concerns regarding the potential decline in students’ cognitive abilities and the loss of their analytical independence. On the other hand, the majority of previous studies have employed a one-size-fits-all approach that generalizes the impact of artificial intelligence without accounting for the specific behavioral heterogeneity of its users. This gap in the literature serves as the research gap for this study, which proposes Hybrid Machine Learning to predict fluctuations in the Skill Retention Score metric. The K-Means algorithm was implemented to segment the data, predictive modeling using the CatBoost Regressor through SHAP (Semi-Supervised Heterogeneous Adaptive Predictor) explainable AI. The segmentation results confirmed the existence of the following profiles: Cluster 0 (The Heavy Dependent) and Cluster 1 (The Traditional User). Based on these two clusters, the STEM field was found to be the top field in the use of Artificial Intelligence (AI) for the purpose of debugging computational code. Model evaluation revealed that AI adoption behavior variables significantly dictate skill degradation only among extreme users (R² = 0.3131), compared to conventional users (R² = 0.1797). Furthermore, the Shapley value analysis found that AI is proven to be safe as a support assistant or learning assistant if the dependency level remains between 1 and 6; however, if usage exceeds 6, it will affect cognitive retention. In other words, the Shapley value analysis successfully identified the tipping point of the cognitive offloading phenomenon. Nevertheless, a total ban on AI use was found to be ineffective in preserving academic retention scores. Therefore, a transition toward institutional regulations regarding the adoption of artificial intelligence is needed, one that is more adaptive, accountable, and specifically tailored to high-risk demographics, particularly in STEM fields.