Keywords
NLP, machine learning, cognitive science, self-regulated learning, LLM, CoALA
Abstract
As large language models (LLMs) demonstrate increasingly sophisticated language understanding capabilities, they offer transformative potential for educational technology, particularly in developing more responsive intelligent tutoring systems (ITS). To develop these systems, a synthesis that balances theoretical frameworks with practical, cutting-edge, and deployable technology would be useful. This thesis aims to advance the field of ITSs theoretically by viewing the modern advancements in natural language processing within the context of the foundational frameworks important to the fields of self-regulated learning, ITSs, and cognitive agents. Then, building on prior research on the ITS MetaTutor \citep{Azevedo2022}, this research focuses on how functions of the pedagogical agent "Sam the Strategizer" could be deployed and accessible through local on-device models. % Specifically, I fine-tuned the encoder models ModernBERT-base and ModernBERT-large, and decoder-only models such as Llama 3.2, Phi-4, and Qwen 2.5, and evaluated their performance on a custom benchmark dataset for fact-checking claims and detecting student compliance. The following research questions guided this investigation: % (1) How can theoretical frameworks from cognitive architectures and intelligent tutoring systems be integrated with emergent abilities of LLMs to design modern pedagogical agents that scaffold self-regulated learning, and what cognitive architectural modules could be used to construct an SRL scaffolding agent (Sam) based on these principles? % (2) How do small, consumer-hardware-compatible encoder-only models compare to small decoder-only models in accuracy and efficiency when fact-checking claims, and how does domain-specific fine-tuning affect these results? % (3) How do these models perform on student compliance detection, and does fine-tuning a domain-specific expert improve these results? This research demonstrates the potential for integrating theoretical cognitive architecture frameworks with modern LLM abilities in the context of ITSs. % The experimental findings suggest that small fine-tuned decoder models outperform encoder models in fact checking domain-specific claims, while small encoder models are more effective at determining student compliance, which suggests that different model architectures may be optimal for different reasoning tasks that an agent performs. % Additionally, the results indicate that although larger models tend to work better for complex fine-grained tasks, small locally-deployable models can be effective with fine-tuning, potentially increasing privacy and accessibility for students. Since the parameter count appeared to significantly increase performance, future research could more thoroughly explore the trade-off between efficiency and accuracy by deploying larger models (7-8B parameters) that still fit on some consumer hardware. % Additionally, expanding beyond the domain of the circulatory system and exploring more the performance of models on different reasoning tasks could increase the agent's completeness.
Thesis Completion Year
2025
Thesis Completion Semester
Spring
Thesis Chair
Azevedo, Roger
College
College of Engineering and Computer Science
Department
Computer Science
Thesis Discipline
Computer Science
Language
English
Access Status
Open Access
Length of Campus Access
None
Campus Location
Orlando (Main) Campus
STARS Citation
Naderi, Evan Vahid, "Intelligent Tutoring Systems In The Age Of Llm-Based Agentic Frameworks - Adapting Small On-Device Language Models For Fact-Checking And Student Compliance Detection" (2025). Honors Undergraduate Theses. 357.
https://stars.library.ucf.edu/hut2024/357
Included in
Rights Statement
