Log Data Indicators for Identifying Learner Engagement in MOOCs
Abstract
Engagement during learning is crucial as the instructor and even management can see how well and understood is learner about the topic or course they are up to. As Engagement happens in Massive Open Online Courses (MOOCs), a lot of research papers discuss it and it is more interesting as the pandemic happens and most students must adapt to online learning. Log data in MOOCs have recorded engagement indicators and it is called log data indicators. However, it comes to the question of what is considered engagement in MOOCs as engagement has three parts which are affective, cognitive, and behavioral. A Systematic Literature Review (SLR) was used in this study to find out what engagement indicators for MOOCs can be captured. The result showed for this study is the indicators that can be used to analyze to explore student engagement which include the description of the indicators accordingly. From an online education perspective and improvement of the MOOCs platform, The engagement indicators are really necessary as it is part of learning. Able to identify the engagement and view the level of engagement for every participant in the course. The exploration of the engagement itself makes the instructor well knowledge of what to improve in their courses materials.
References
Alemayehu, L., & Chen, H. L. (2021). Learner and instructor-related challenges for learners’ engagement in MOOCs: a review of 2014–2020 publications in selected SSCI indexed journals.
Almukhaylid, M., & Suleman, H. (2020). Socially-Motivated Discussion Forum Models for Learning Management Systems. ACM International Conference Proceeding Series, 1–11.
Baethge, C., Goldbeck-Wood, S., & Mertens, S. (2019). SANRA—a scale for the quality assessment of narrative review articles. Research Integrity and Peer Review 2019 4:1, 4(1), 1–7.
Bonafini, F., Bonafini, F., Chae, C., Park, E., & Jablokow, K. (2017). How Much Does Student Engagement with Videos and Forums in a MOOC Affect... Online Learning Journal, 21(4).
Carlon, M. K. J., Keerativoranan, N., & Cross, J. S. (2020). Content Type Distribution and Readability of MOOCs. L@S 2020 - Proceedings of the 7th ACM Conference on Learning @ Scale, 401–404.
Clark, K. R., Vealé, B. L., Watts, L. K., Kr, C., Bl, V., & Watts, L. A. (2017). A review of the use of massive open online courses (MOOCs) in medical imaging education. Nsuworks.Nova.Edu, 15(2).
Cohen, S. S., Madsen, J., Touchan, G., Robles, D., Lima, S. F. A., Henin, S., & Parra, L. C. (2018). Neural engagement with online educational videos predicts learning performance for individual students. Neurobiology of Learning and Memory, 155, 60–64.
Collins, C., Dennehy, D., Conboy, K., & Mikalef, P. (2021). Artificial intelligence in information systems research: A systematic literature review and research agenda. International Journal of Information Management, 60, 102383.
Crues, R. W., Bosch, N., Perry, M., Angrave, L., Shaik, N., & Bhat, S. (2018). Refocusing the Lens on Engagement in MOOCs. Proceedings of the Fifth Annual ACM Conference on Learning at Scale.
Deng, R., Benckendorff, P., & Gannaway, D. (2019). Learner engagement in MOOCs: Scale development and validation. British Journal of Educational Technology, 51(1), 245–262.
Deng, R., Benckendorff, P., & Gannaway, D. (2020). Linking learner factors, teaching context, and engagement patterns with MOOC learning outcomes. Journal of Computer Assisted Learning, 36(5), 688–708.
Ding, Y., & Zhao, T. (2020). Emotions, engagement, and self-perceived achievement in a small private online course. Journal of Computer Assisted Learning, 36(4), 449–457.
Doherty, K., & Doherty, G. (2019). Engagement in HCI: Conception, theory and measurement. ACM Computing Surveys, 51(5).
Estrada-Molina, O., & Fuentes-Cancell. (2022). Engagement and Desertion in MOOCs: Systematic Review. ERIC, 70, 2022.
Farrow, E., Moore, J., & Gašević, D. (2019). Analysing discussion forum data: A replication study avoiding data contamination. ACM International Conference Proceeding Series, 170–179.
Farrow, E., Moore, J., & Gaševic, D. (2020). Dialogue attributes that inform depth and quality of participation in course discussion forums. ACM International Conference Proceeding Series, 129–134.
Galikyan, I., Admiraal, W., & Kester, L. (2021). MOOC discussion forums: The interplay of the cognitive and the social. Computers & Education, 165, 104133.
Geigle, C., conference, C. Z.-P. of the fourth (2017) A., & 2017. (2017). Modeling MOOC student behavior with two-layer hidden Markov models. Dl.Acm.Org, 205–208.
Gledson, A., Apaolaza, A., Barthold, S., Günther, F., Yu, H., & Vigo, M. (2018). Instructor perspectives on comparative heatmap visualizations of student engagement with lecture video. SIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Computer Science Education, 2018-January, 251–256.
Gledson, A., Apaolaza, A., Barthold, S., Günther, F., Yu, H., & Vigo, M. (2021). Characterising student engagement modes through low-level activity patterns. UMAP 2021 - Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, 88–97.
Guajardo Leal, B. E. , Navarro-Corona, C. , & Valenzuela González, J. R. (2019). Systematic mapping study of academic engagement in MOOC. Erudit.Org, 20(2).
Halverson, L., & CR Graham. (2019). Learner engagement in blended learning environments: A conceptual framework. ERIC, 16(1).
Hardt, R. (2022). A System to Motivate Sustained Lecture Video Engagement in Small Private Online Courses. Koli Calling ’22: 22nd Koli Calling International Conference on Computing Education Research, 1.
Henrie, C., Halverson, L., & CR Graham. (2015). Measuring student engagement in technology-mediated learning: A review. Elsevier.
Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology, 47(2), 320–341.
Hew, K., Qiao, C., & Y Tang. (2018). Understanding student engagement in large-scale open online courses: A machine learning facilitated analysis of student’s reflections in 18 highly rated MOOCs. Erudit.Org.
Huang, T., Mei, Y., Zhang, H., Liu, S., & Yang, H. (2019). Fine-grained engagement recognition in online learning environment. ICEIEC 2019 - Proceedings of 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication, 338–341.
Hussain, M., Zhu, W., Zhang, W., & Abidi, S. M. R. (2018). Student Engagement Predictions in an e-Learning System and Their Impact on Student Course Assessment Scores. Computational Intelligence and Neuroscience, 2018.
Joksimović, S., Poquet, O., Kovanović, V., Dowell, N., Mills, C., Gašević, D., Dawson, S., Graesser, A. C., & Brooks, C. (2018). How do we model learning at scale? A systematic review of research on moocs. Review of Educational Research, 88(1), 43–86.
KF Hew. (2015). Towards a model of engaging online students: lessons from MOOCs and four policy documents. Ijiet.Org.
Kőrösi, G., & Farkas, R. (2020). MOOC Performance Prediction by Deep Learning from Raw Clickstream Data. Communications in Computer and Information Science, 1244 CCIS, 474–485.
Liao, B., Ali, Y., Nazir, S., He, L., & Khan, H. U. (2020). Security analysis of IoT devices by using mobile computing: a systematic literature review. Ieeexplore.Ieee.Org.
Liu, Z., Mu, R., Liu, S., Peng, X., & Liu, S. (2021). Modeling Temporal Association of Cognition-Topic in MOOC Discussion to Track Learners’ Cognitive Engagement Dynamics. Conference of the South African Institute of Computer Scientists and Information Technologists 2020.
Mattick, K., Johnston, J., & de la Croix, A. (2018). How to write a good research question. Wiley Online Library, 15(2), 104–108.
Min, L., & Foon, H. K. (2019). Self-Regulated Learning Process in MOOCs: Examining the Indicators of Behavioral, Emotional, and Cognitive Engagement. Proceedings of the 2019 4th International Conference on Distance Education and Learning.
Mongkhonvanit, K., Kanopka, K., & Lang, D. (2019). Deep knowledge tracing and engagement with MOOCs. ACM International Conference Proceeding Series, 340–342.
Moubayed, A., Injadat, M., Shami, A., & Lutfiyya, H. (2020). Student engagement level in an e-learning environment: Clustering using k-means. Taylor & Francis, 34(2), 137–156.
Ng, K., & Lei, P. (2022). A Lightweight Method using LightGBM Model with Optuna in MOOCs Dropout Prediction. 2022 6th International Conference on Education and Multimedia Technology, 1, 53–59.
Paton, R. M., Fluck, A. E., & Scanlan, J. D. (2018). Engagement and retention in VET MOOCs and online courses: A systematic review of literature from 2013 to 2017. Computers & Education, 125, 191–201.
Poquet, O., Lim, L., Mirriahi, N., & Dawson, S. (2018). Video and learning: A systematic review (2007-2017). ACM International Conference Proceeding Series, 20(6), 151–160.
Romero-Rodriguez, L., & MS Ramirez-Montoya. (2019). Gamification in MOOCs: Engagement application test in energy sustainability courses. Ieeexplore.Ieee.Org.
Shukor, N. A., & Abdullah, Z. (2019). Using learning analytics to improve MOOC instructional design. Learntechlib.Org. https://doi.org/10.3991/ijet.v14i24.12185
Soffer, T., & Cohen, A. (2019). Students’ engagement characteristics predict success and completion of online courses. Journal of Computer Assisted Learning, 35(3), 378–389.
Sooryanarayan, D. G., & Gupta, D. (2015). Impact of learner motivation on MOOC preferences: Transfer vs. made MOOCs. 2015 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2015, 929–934.
Stracke, C. M. (2017). Why We Need High Drop-Out Rates in MOOCs: New Evaluation and Personalization Strategies for the Quality of Open Education. Proceedings - IEEE 17th International Conference on Advanced Learning Technologies, ICALT 2017, 13–15.
Stracke, C. M., Tan, E., Moreira Texeira, A., Texeira Pinto, M. D. C., Vassiliadis, B., Kameas, A., & Sgouropoulou, C. (2018). Gap between MOOC Designers’ and MOOC Learners’ Perspectives on Interaction and Experiences in MOOCs: Findings from the Global MOOC Quality Survey. Proceedings - IEEE 18th International Conference on Advanced Learning Technologies, ICALT 2018, 1–5.
Swinnerton, B., Hotchkiss, S., & Morris, N. P. (2017). Comments in MOOCs: who is doing the talking and does it help? Journal of Computer Assisted Learning, 33(1), 51–64.
Tang, W., Chen, Y., & Guan, B. (n.d.). An Analysis of the Characteristics of Teacher Talk on MOOCs Based on Interpersonal Function Take the “National Excellent Course” of English Writing as an Example KEYWORDS. 2022 6th International Conference on Education and Multimedia Technology, 1.
Thi, N., Ho, T., Thi, A., & Pham, V. (2022). Student Perceptions towards Positive and Negative Aspects of MOOC-based Blended Learning. 2022 6th International Conference on Education and Multimedia Technology, 1, 20–26.
Thomas, C., Nair, N., & Jayagopi, D. B. (2018). Predicting Engagement Intensity in the Wild Using Temporal Convolutional Network. 604–610.
Tseng, S. F., Tsao, Y. W., Yu, L. C., Chan, C. L., & Lai, K. R. (2016). Who will pass? Analyzing learner behaviors in MOOCs. Research and Practice in Technology Enhanced Learning, 11(1), 1–11.
Vellukunnel, M., Buffum, P., Boyer, K. E., Forbes, J., Heckman, S., & Mayer-Patel, K. (2017). Deconstructing the discussion forum: Student questions and computer science learning. Proceedings of the Conference on Integrating Technology into Computer Science Education, ITiCSE, 603–608.
Wang, W., Guo, L., He, L., & Wu, Y. J. (2018). Effects of social-interactive engagement on the dropout ratio in online learning: insights from MOOC.
Xiao, Y., and, M. W.-J. of P. E., & 2019, undefined. (2019). Guidance on conducting a systematic literature review. Journals.Sagepub.Com, 39(1), 93–112.
Xiao, Y., & M Watson. (2019). Guidance on conducting a systematic literature review. Journals.Sagepub.Com, 39(1), 93–112.
Yan, W., Dowell, N., Holman, C., Welsh, S. S., Choi, H., & Brooks, C. (2018). Unpacking the relationship between discussion forum participation and learning in MOOCs: Content is key. ACM International Conference Proceeding Series, 330–339.
Yan, W., Welsh, S. S., Dowell, N., Choi, H., Holman, C., & Brooks, C. (2019). Exploring learner engagement patterns in teach-outs. ACM International Conference Proceeding Series, 180–184.
Almukhaylid, M., & Suleman, H. (2020). Socially-Motivated Discussion Forum Models for Learning Management Systems. ACM International Conference Proceeding Series, 1–11.
Baethge, C., Goldbeck-Wood, S., & Mertens, S. (2019). SANRA—a scale for the quality assessment of narrative review articles. Research Integrity and Peer Review 2019 4:1, 4(1), 1–7.
Bonafini, F., Bonafini, F., Chae, C., Park, E., & Jablokow, K. (2017). How Much Does Student Engagement with Videos and Forums in a MOOC Affect... Online Learning Journal, 21(4).
Carlon, M. K. J., Keerativoranan, N., & Cross, J. S. (2020). Content Type Distribution and Readability of MOOCs. L@S 2020 - Proceedings of the 7th ACM Conference on Learning @ Scale, 401–404.
Clark, K. R., Vealé, B. L., Watts, L. K., Kr, C., Bl, V., & Watts, L. A. (2017). A review of the use of massive open online courses (MOOCs) in medical imaging education. Nsuworks.Nova.Edu, 15(2).
Cohen, S. S., Madsen, J., Touchan, G., Robles, D., Lima, S. F. A., Henin, S., & Parra, L. C. (2018). Neural engagement with online educational videos predicts learning performance for individual students. Neurobiology of Learning and Memory, 155, 60–64.
Collins, C., Dennehy, D., Conboy, K., & Mikalef, P. (2021). Artificial intelligence in information systems research: A systematic literature review and research agenda. International Journal of Information Management, 60, 102383.
Crues, R. W., Bosch, N., Perry, M., Angrave, L., Shaik, N., & Bhat, S. (2018). Refocusing the Lens on Engagement in MOOCs. Proceedings of the Fifth Annual ACM Conference on Learning at Scale.
Deng, R., Benckendorff, P., & Gannaway, D. (2019). Learner engagement in MOOCs: Scale development and validation. British Journal of Educational Technology, 51(1), 245–262.
Deng, R., Benckendorff, P., & Gannaway, D. (2020). Linking learner factors, teaching context, and engagement patterns with MOOC learning outcomes. Journal of Computer Assisted Learning, 36(5), 688–708.
Ding, Y., & Zhao, T. (2020). Emotions, engagement, and self-perceived achievement in a small private online course. Journal of Computer Assisted Learning, 36(4), 449–457.
Doherty, K., & Doherty, G. (2019). Engagement in HCI: Conception, theory and measurement. ACM Computing Surveys, 51(5).
Estrada-Molina, O., & Fuentes-Cancell. (2022). Engagement and Desertion in MOOCs: Systematic Review. ERIC, 70, 2022.
Farrow, E., Moore, J., & Gašević, D. (2019). Analysing discussion forum data: A replication study avoiding data contamination. ACM International Conference Proceeding Series, 170–179.
Farrow, E., Moore, J., & Gaševic, D. (2020). Dialogue attributes that inform depth and quality of participation in course discussion forums. ACM International Conference Proceeding Series, 129–134.
Galikyan, I., Admiraal, W., & Kester, L. (2021). MOOC discussion forums: The interplay of the cognitive and the social. Computers & Education, 165, 104133.
Geigle, C., conference, C. Z.-P. of the fourth (2017) A., & 2017. (2017). Modeling MOOC student behavior with two-layer hidden Markov models. Dl.Acm.Org, 205–208.
Gledson, A., Apaolaza, A., Barthold, S., Günther, F., Yu, H., & Vigo, M. (2018). Instructor perspectives on comparative heatmap visualizations of student engagement with lecture video. SIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Computer Science Education, 2018-January, 251–256.
Gledson, A., Apaolaza, A., Barthold, S., Günther, F., Yu, H., & Vigo, M. (2021). Characterising student engagement modes through low-level activity patterns. UMAP 2021 - Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, 88–97.
Guajardo Leal, B. E. , Navarro-Corona, C. , & Valenzuela González, J. R. (2019). Systematic mapping study of academic engagement in MOOC. Erudit.Org, 20(2).
Halverson, L., & CR Graham. (2019). Learner engagement in blended learning environments: A conceptual framework. ERIC, 16(1).
Hardt, R. (2022). A System to Motivate Sustained Lecture Video Engagement in Small Private Online Courses. Koli Calling ’22: 22nd Koli Calling International Conference on Computing Education Research, 1.
Henrie, C., Halverson, L., & CR Graham. (2015). Measuring student engagement in technology-mediated learning: A review. Elsevier.
Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology, 47(2), 320–341.
Hew, K., Qiao, C., & Y Tang. (2018). Understanding student engagement in large-scale open online courses: A machine learning facilitated analysis of student’s reflections in 18 highly rated MOOCs. Erudit.Org.
Huang, T., Mei, Y., Zhang, H., Liu, S., & Yang, H. (2019). Fine-grained engagement recognition in online learning environment. ICEIEC 2019 - Proceedings of 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication, 338–341.
Hussain, M., Zhu, W., Zhang, W., & Abidi, S. M. R. (2018). Student Engagement Predictions in an e-Learning System and Their Impact on Student Course Assessment Scores. Computational Intelligence and Neuroscience, 2018.
Joksimović, S., Poquet, O., Kovanović, V., Dowell, N., Mills, C., Gašević, D., Dawson, S., Graesser, A. C., & Brooks, C. (2018). How do we model learning at scale? A systematic review of research on moocs. Review of Educational Research, 88(1), 43–86.
KF Hew. (2015). Towards a model of engaging online students: lessons from MOOCs and four policy documents. Ijiet.Org.
Kőrösi, G., & Farkas, R. (2020). MOOC Performance Prediction by Deep Learning from Raw Clickstream Data. Communications in Computer and Information Science, 1244 CCIS, 474–485.
Liao, B., Ali, Y., Nazir, S., He, L., & Khan, H. U. (2020). Security analysis of IoT devices by using mobile computing: a systematic literature review. Ieeexplore.Ieee.Org.
Liu, Z., Mu, R., Liu, S., Peng, X., & Liu, S. (2021). Modeling Temporal Association of Cognition-Topic in MOOC Discussion to Track Learners’ Cognitive Engagement Dynamics. Conference of the South African Institute of Computer Scientists and Information Technologists 2020.
Mattick, K., Johnston, J., & de la Croix, A. (2018). How to write a good research question. Wiley Online Library, 15(2), 104–108.
Min, L., & Foon, H. K. (2019). Self-Regulated Learning Process in MOOCs: Examining the Indicators of Behavioral, Emotional, and Cognitive Engagement. Proceedings of the 2019 4th International Conference on Distance Education and Learning.
Mongkhonvanit, K., Kanopka, K., & Lang, D. (2019). Deep knowledge tracing and engagement with MOOCs. ACM International Conference Proceeding Series, 340–342.
Moubayed, A., Injadat, M., Shami, A., & Lutfiyya, H. (2020). Student engagement level in an e-learning environment: Clustering using k-means. Taylor & Francis, 34(2), 137–156.
Ng, K., & Lei, P. (2022). A Lightweight Method using LightGBM Model with Optuna in MOOCs Dropout Prediction. 2022 6th International Conference on Education and Multimedia Technology, 1, 53–59.
Paton, R. M., Fluck, A. E., & Scanlan, J. D. (2018). Engagement and retention in VET MOOCs and online courses: A systematic review of literature from 2013 to 2017. Computers & Education, 125, 191–201.
Poquet, O., Lim, L., Mirriahi, N., & Dawson, S. (2018). Video and learning: A systematic review (2007-2017). ACM International Conference Proceeding Series, 20(6), 151–160.
Romero-Rodriguez, L., & MS Ramirez-Montoya. (2019). Gamification in MOOCs: Engagement application test in energy sustainability courses. Ieeexplore.Ieee.Org.
Shukor, N. A., & Abdullah, Z. (2019). Using learning analytics to improve MOOC instructional design. Learntechlib.Org. https://doi.org/10.3991/ijet.v14i24.12185
Soffer, T., & Cohen, A. (2019). Students’ engagement characteristics predict success and completion of online courses. Journal of Computer Assisted Learning, 35(3), 378–389.
Sooryanarayan, D. G., & Gupta, D. (2015). Impact of learner motivation on MOOC preferences: Transfer vs. made MOOCs. 2015 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2015, 929–934.
Stracke, C. M. (2017). Why We Need High Drop-Out Rates in MOOCs: New Evaluation and Personalization Strategies for the Quality of Open Education. Proceedings - IEEE 17th International Conference on Advanced Learning Technologies, ICALT 2017, 13–15.
Stracke, C. M., Tan, E., Moreira Texeira, A., Texeira Pinto, M. D. C., Vassiliadis, B., Kameas, A., & Sgouropoulou, C. (2018). Gap between MOOC Designers’ and MOOC Learners’ Perspectives on Interaction and Experiences in MOOCs: Findings from the Global MOOC Quality Survey. Proceedings - IEEE 18th International Conference on Advanced Learning Technologies, ICALT 2018, 1–5.
Swinnerton, B., Hotchkiss, S., & Morris, N. P. (2017). Comments in MOOCs: who is doing the talking and does it help? Journal of Computer Assisted Learning, 33(1), 51–64.
Tang, W., Chen, Y., & Guan, B. (n.d.). An Analysis of the Characteristics of Teacher Talk on MOOCs Based on Interpersonal Function Take the “National Excellent Course” of English Writing as an Example KEYWORDS. 2022 6th International Conference on Education and Multimedia Technology, 1.
Thi, N., Ho, T., Thi, A., & Pham, V. (2022). Student Perceptions towards Positive and Negative Aspects of MOOC-based Blended Learning. 2022 6th International Conference on Education and Multimedia Technology, 1, 20–26.
Thomas, C., Nair, N., & Jayagopi, D. B. (2018). Predicting Engagement Intensity in the Wild Using Temporal Convolutional Network. 604–610.
Tseng, S. F., Tsao, Y. W., Yu, L. C., Chan, C. L., & Lai, K. R. (2016). Who will pass? Analyzing learner behaviors in MOOCs. Research and Practice in Technology Enhanced Learning, 11(1), 1–11.
Vellukunnel, M., Buffum, P., Boyer, K. E., Forbes, J., Heckman, S., & Mayer-Patel, K. (2017). Deconstructing the discussion forum: Student questions and computer science learning. Proceedings of the Conference on Integrating Technology into Computer Science Education, ITiCSE, 603–608.
Wang, W., Guo, L., He, L., & Wu, Y. J. (2018). Effects of social-interactive engagement on the dropout ratio in online learning: insights from MOOC.
Xiao, Y., and, M. W.-J. of P. E., & 2019, undefined. (2019). Guidance on conducting a systematic literature review. Journals.Sagepub.Com, 39(1), 93–112.
Xiao, Y., & M Watson. (2019). Guidance on conducting a systematic literature review. Journals.Sagepub.Com, 39(1), 93–112.
Yan, W., Dowell, N., Holman, C., Welsh, S. S., Choi, H., & Brooks, C. (2018). Unpacking the relationship between discussion forum participation and learning in MOOCs: Content is key. ACM International Conference Proceeding Series, 330–339.
Yan, W., Welsh, S. S., Dowell, N., Choi, H., Holman, C., & Brooks, C. (2019). Exploring learner engagement patterns in teach-outs. ACM International Conference Proceeding Series, 180–184.
Published
2023-03-01
How to Cite
MOHD SHARIF, Mohamad Shafri Amir; RAMAKRISNAN, Prasanna.
Log Data Indicators for Identifying Learner Engagement in MOOCs.
International Journal of Advanced Research in Education and Society, [S.l.], v. 5, n. 1, p. 35-51, mar. 2023.
ISSN 2682-8138.
Available at: <https://myjms.mohe.gov.my/index.php/ijares/article/view/21663>. Date accessed: 05 oct. 2023.
Section
Articles
