Keywords

Forensic fire debris analysis, Decision-making workflow, Human (analyst) performance validation, Machine learning performance validation, Uncertainty, Decision theory, Subjective logic, Linear sequential unmasking, Receiver operating characteristic curve, Evaluative reporting, Performance validation protocol benefits

Abstract

This work presents a background on the chemical complexity of fire debris analysis, including an ever-present matrix of pyrolysis products as the catalyst that led to the creation of the National Center for Forensic Science's Fire Debris Database. A selection of these 1,000+ casework-relevant ground truth samples was used to create two newly proposed analyst workflows to connect the current method of categorical reporting with evaluative reporting practices reflective of the strength of the evidence. Both workflows use linear sequential unmasking to help mitigate bias, a discrete scoring system for quantification of the analysis, and receiver operating characteristic (ROC) curves to bridge together categorical and probabilistic reporting by indicating the optimum decision threshold the analysts are operating from when they make a decision. Both workflows also allow a machine-learning component to be included in evaluating the evidence and are practical methods for obtaining validated performances for human and machine decisions. The second workflow includes subjective logic, which provides a means of determining the uncertainty inherent to the opinion made by the analyst and the machine learning computational model. ‘Fuzzy categories' and an opinion triangle connect the opinion offered by the analyst given their perceived uncertainty to the ROC curve so a categorical decision can be made. For each workflow, three analysts independently assessed 20 randomly chosen samples from the Fire Debris Database and followed the ASTM E1618-19 standard fire debris analysis method. The resultant area under the ROC curve for each analyst for each workflow was 0.90 or higher, indicating that all were in the very good to excellent range for diagnostic classifiers, as was the machine learning model tested in the second workflow. Recommendations for implementing a performance validation workflow, how repetitive engagement can help the individual analyst and insights on using these for performance validation and training purposes are also included.

Completion Date

2024

Semester

Spring

Committee Chair

Sigman, Michael

Degree

Doctor of Philosophy (Ph.D.)

College

College of Sciences

Department

Chemistry

Degree Program

Chemistry

Format

application/pdf

Identifier

DP0028396

URL

https://purls.library.ucf.edu/go/DP0028396

Language

English

Rights

In copyright

Release Date

May 2024

Length of Campus-only Access

None

Access Status

Doctoral Dissertation (Open Access)

Campus Location

Orlando (Main) Campus

Accessibility Status

Meets minimum standards for ETDs/HUTs

Download

Share

COinS