Jie LinFollow

ORCID

0009-0007-1176-6115

Keywords

LLMs, Vulnerability Detection, Vulnerability Localization, Hallucination, Static Code Analysis

Abstract

Large Language Models (LLMs) have recently emerged as promising tools for analyzing source code; however, their capabilities for automated vulnerability detection and localization remain underexplored, particularly given the widespread reliance on closed-source models which introduce significant privacy, security, and transparency risks. This dissertation rigorously examines the feasibility and key influencing factors determining the efficacy of open-source LLMs in source code security analysis. Initially, we investigate the impact of tokenized input length on vulnerability detection accuracy and explicitness across Java vulnerability datasets using ten distinct LLM architectures, identifying robust performance in specific models while others demonstrate considerable accuracy deterioration. Subsequently, we comprehensively evaluate the true vulnerability detection effectiveness of 38 open-source LLM configurations using both vulnerable and non-vulnerable datasets spanning Java and C/C++, applying realistic zero-shot and few-shot prompting scenarios and meticulously assessing model performance by specific vulnerability types. Furthermore, extending beyond mere detection capabilities, we conduct an extensive empirical study involving 66 LLM configurations for Java and 63 for C/C++ to examine hallucination behaviors during vulnerability localization. Our findings highlight that structured, assumption-based prompts substantially mitigate hallucinations, particularly among mid-sized models; nevertheless, precise vulnerability localization remains notably challenging, especially within C/C++ contexts. Finally, by evaluating precision, recall, and F1 metrics for various open-source LLMs at the function-level localization task, we systematically characterize their true effectiveness and practical applicability. Collectively, this dissertation establishes that through strategic prompt engineering, meticulous model selection, and rigorous empirical validation, open-source LLMs can reliably, transparently, and securely enhance vulnerability detection and localization processes, thereby substantially contributing to secure software development practices.

Completion Date

2025

Semester

Summer

Committee Chair

Mohaisen, David

Degree

Doctor of Philosophy (Ph.D.)

College

College of Engineering and Computer Science

Department

Computer Science

Format

PDF

Identifier

DP0029581

Language

English

Document Type

Thesis

Campus Location

Orlando (Main) Campus

Download

Share

COinS