Oracle Analysis and Study Mapping

← Back to Home

← Technique-to-Study Mapping

This page provides the extended analysis and complete study-level mapping for the test oracles examined in the survey Fuzzing AI Systems: Foundations, Techniques, and Open Challenges.

RQ3 asks how fuzzing studies determine whether a generated test exposes a failure. This question is particularly important for AI systems because an exact expected output is often unavailable for arbitrary inputs.

We analyze two related but distinct dimensions:

1. Oracle type — the high-level principle used to identify a failure; and
2. Oracle construction — the mechanism used to implement or approximate that principle in practice.

The categories are not mutually exclusive because a study may use multiple oracle types or combine several construction mechanisms.

Distribution Overview

Oracle Types

Oracle type	Number of studies
Specification-based	44
Differential	35
Crash/Exception	26
Inconsistency-based	25

Oracle-Construction Mechanisms

Oracle construction	Number of studies
Rules/constraints	62
Heuristic	34
Multiple models	14
Reference implementation	12
Multiple backends	2
Multiple frameworks/libraries	2

The complete clickable PID mapping is provided in the Detailed Study-Level Mapping section.

---

Oracle Types

Specification-Based Oracles

Specification-based oracles check whether system behavior satisfies predefined:

• rules;
• constraints;
• metamorphic relations;
• safety requirements;
• validity conditions; or
• domain-specific expectations.

They are the most common oracle type, appearing in 44 studies.

Many of these specifications are partial rather than complete. Instead of providing an exact expected output for every test input, they define conditions that acceptable behavior should satisfy.

Examples include:

• MDPFuzz (P019), which uses task- and policy-level failure conditions;
• DriveFuzz (P066), which evaluates driving behavior against scenario-level safety and quality requirements; and
• system-level autonomous-driving and reinforcement-learning studies that use trajectory, policy, safety, or environment-level conditions.

The dominance of specification-based oracles shows that AI fuzzing frequently relies on partial specifications and domain knowledge rather than complete input-output ground truth.

Differential Oracles

Differential oracles compare behavior across:

• models;
• model variants;
• implementations;
• frameworks;
• libraries;
• compiler backends;
• software versions;
• devices; or
• reference executions.

They appear in 35 studies.

A differential oracle treats unexpected disagreement among comparable executions as a potential failure signal.

Examples include:

• NNsmith (P001), which compares compiled model outputs with a reference implementation;
• TensorJSFuzz (P003), which uses comparison-based checking in DL framework testing;
• FlashFuzz (P080), which compares CPU and GPU executions for output or exception divergence; and
• DeepDiffer (P091), which applies priority-guided differential fuzzing to DL compilers.

Differential oracles are especially useful for framework/library and compiler/backend targets because these targets often provide multiple comparable execution paths.

Their main challenge is distinguishing true defects from acceptable differences caused by floating-point variation, approximation, nondeterminism, or device-specific behavior.

Crash/Exception Oracles

Crash/exception oracles treat the following as failures:

• runtime crashes;
• uncaught exceptions;
• assertion failures;
• segmentation faults;
• abnormal termination;
• invalid execution states; or
• other severe runtime errors.

They appear in 26 studies.

This oracle type is particularly common in framework/library and compiler/backend fuzzing because generated:

• API calls;
• tensor operations;
• computation graphs;
• compiler programs;
• optimization sequences; or
• backend executions

may trigger immediate runtime failures.

For example, DocTer (P017) generates DL API inputs and reports severe failures such as segmentation faults, floating-point exceptions, aborts, and bus errors.

Crash/exception oracles are practical because they do not require a complete semantic specification of the expected output.

However, they mainly expose explicit runtime failures and may miss silent numerical, behavioral, or semantic defects.

Inconsistency-Based Oracles

Inconsistency-based oracles detect:

• unstable predictions;
• unexpected behavioral changes;
• numerical instability;
• output inconsistency;
• semantic deviations; or
• violations of expected behavioral relations.

They appear in 25 studies and are common in model-level testing.

Examples include:

• DeepCNP (P004);
• DLRegion (P007);
• DeepHunter (P032);
• DeepRoad (P049); and
• CAGFuzz (P143).

These studies identify failures through prediction changes, behavioral deviations, transformed-input inconsistencies, or other forms of unstable behavior when exact expected labels are unavailable.

Inconsistency-based oracles directly address a central problem in AI testing: many failures cannot be identified using one exact expected output.

---

Target-Dependent Oracle Patterns

Oracle choice varies substantially by testing target.

Framework/Library and Compiler/Backend Targets

These studies frequently use:

• differential comparison;
• reference implementations;
• multiple backends;
• multiple frameworks or libraries; and
• crash/exception detection.

These targets often provide comparable execution paths and observable runtime failures.

For example:

• NNsmith (P001) compares compiled outputs against reference execution;
• TzER (P046) checks result inconsistency, performance degradation, crashes, and unexpected exceptions; and
• HIRGEN (P028) checks crashes, IR-level result inconsistencies, and hardware-level differences.

Model-Level Targets

Model-level studies more often use:

• specification-based checking;
• inconsistency-based checking;
• metamorphic relations;
• robustness constraints;
• prediction differences; and
• numerical-deviation thresholds.

These mechanisms help compensate for the absence of exact ground truth for arbitrary generated inputs.

System-Level Targets

System-level studies typically use:

• domain-specific safety rules;
• scenario-level constraints;
• policy-level behavior;
• trajectory requirements;
• environment-level validity conditions; and
• task-specific failure definitions.

This pattern appears in autonomous-driving and reinforcement-learning fuzzing, including DriveFuzz (P066), AV-FUZZER (P069), and ScenarioFuzz-LLM (P127).

---

Oracle Construction

Rules and Constraints

Rules/constraints are the most common oracle-construction mechanism, appearing in 62 studies.

This category includes:

• manually defined specifications;
• domain rules;
• validity constraints;
• safety requirements;
• metamorphic relations;
• behavioral invariants;
• threshold-based rules; and
• task-specific acceptance conditions.

The frequency of this mechanism reflects the need to express partial, target-specific expectations when exact outputs are unavailable.

Heuristic Construction

Heuristic construction appears in 34 studies.

Heuristic oracles use practical approximations such as:

• suspicious-behavior indicators;
• empirical thresholds;
• tool-specific checks;
• approximate comparison criteria;
• prediction-difference signals;
• abnormal-performance indicators; or
• custom failure scores.

For example:

• TzER (P046) uses practical checks for inconsistency, performance degradation, crashes, and unexpected exceptions;
• FlashFuzz (P080) uses tolerance-based output comparison and exception-divergence checks across devices; and
• model-level fuzzers may use confidence, prediction difference, coverage, or other approximate signals to identify suspicious behavior.

Heuristic construction is flexible, but its reliability depends on the quality and calibration of the selected signal.

Multiple Models

Comparison across multiple models appears in 14 studies.

This construction mechanism checks whether several models or model variants behave consistently on the same or related inputs.

It is useful when no single model can serve as an unquestioned source of truth, but agreement or disagreement across models provides an informative failure signal.

Reference Implementations

Reference-implementation oracles appear in 12 studies.

A trusted implementation, unoptimized execution, baseline system, or reference backend is used as the comparator.

Examples include:

• NNsmith (P001);
• ExAIS (P020);
• Predoo (P021);
• TzER (P046); and
• TorchProbe (P089).

This mechanism can provide a strong comparison baseline, although its reliability depends on the correctness and independence of the reference.

Multiple Backends

Comparison across multiple backends appears in 2 studies:

• FlashFuzz (P080); and
• FUEL (P082).

This mechanism checks whether equivalent executions remain consistent across hardware or backend configurations.

Multiple Frameworks or Libraries

Comparison across multiple frameworks or libraries appears in 2 studies:

• Gandalf (P119); and
• COMET (P120).

Cross-framework comparison is useful when semantically equivalent implementations are available in more than one AI software ecosystem.

---

Hybrid and Multi-Signal Oracle Design

Several studies combine multiple oracle types or construction mechanisms.

For example:

• NNsmith (P001) uses differential comparison against a reference execution while also detecting compiler crashes and inconsistent results.
• HIRGEN (P028) combines crash detection, result inconsistency among original, optimized, and semantically mutated high-level IRs, and cross-device comparison.
• TzER (P046) checks optimized and non-optimized executions for inconsistency, detects performance degradation, and records crashes or unexpected exceptions.
• FlashFuzz (P080) performs differential checking across CPU and GPU executions and treats both output divergence and exception divergence as potential defects.

These examples show that oracle design in AI fuzzing is frequently hybrid and target-dependent rather than tied to one failure signal.

---

Main Findings

The oracle analysis supports five main observations:

1. Specification-based oracles are the most common, but they are usually partial rather than complete.
2. Differential oracles are especially useful for framework/library and compiler/backend targets.
3. Crash/exception oracles remain important for component-level AI software testing.
4. Inconsistency-based oracles help address the lack of exact expected outputs in model-level testing.
5. Oracle construction is frequently hybrid and strongly dependent on the target, available comparators, and observable failure behavior.

Overall, no single oracle type provides a complete solution across all AI-system targets.

The results motivate stronger:

• semantic oracles;
• numerical oracles;
• metamorphic relations;
• safety-aware checks;
• domain-specific specifications; and
• hybrid oracle designs

that can better capture failures across models, frameworks/libraries, compiler backends, and integrated AI applications.

---

Detailed Study-Level Mapping

Each PID below links to the corresponding entry in the Primary Studies catalogue, which provides the full title, authors, publication year, venue, BibTeX key, and DOI or publication link.

Oracle Types

Oracle type	# Studies	Primary studies
Specification-based	44	P012, P019, P020, P026, P035, P043, P050, P058, P066, P068, P069, P088, P099, P106, P122, P125, P127, P130, P133, P136, P137, P140, P145, P147, P148, P152, P153, P157, P158, P160, P162, P164, P165, P169, P172, P174, P178, P180, P182, P197, P200, P201, P202, P208
Differential	35	P001, P003, P008, P009, P010, P013, P014, P015, P016, P018, P028, P029, P036, P038, P042, P045, P046, P051, P060, P077, P080, P082, P086, P089, P090, P091, P095, P100, P119, P120, P156, P170, P176, P193, P204
Crash/Exception	26	P002, P005, P006, P011, P017, P023, P028, P034, P046, P076, P077, P078, P079, P080, P081, P083, P087, P096, P101, P103, P104, P111, P113, P114, P116, P121
Inconsistency-based	25	P004, P007, P021, P022, P024, P025, P027, P028, P030, P031, P032, P033, P039, P040, P041, P044, P049, P061, P142, P143, P144, P146, P154, P167, P198

Oracle-Construction Mechanisms

Oracle construction	# Studies	Primary studies
Rules/constraints	62	P004, P005, P009, P010, P012, P017, P019, P026, P028, P035, P039, P041, P043, P049, P050, P058, P061, P066, P068, P069, P077, P079, P081, P083, P088, P095, P099, P100, P106, P122, P125, P127, P130, P133, P136, P137, P140, P143, P144, P147, P148, P152, P153, P157, P158, P160, P162, P164, P165, P169, P172, P174, P176, P178, P180, P182, P198, P200, P201, P202, P204, P208
Heuristic	34	P002, P007, P011, P022, P023, P024, P025, P027, P030, P031, P032, P033, P034, P040, P044, P045, P046, P076, P078, P087, P096, P101, P103, P104, P111, P113, P114, P116, P121, P142, P145, P146, P154, P167
Multiple models	14	P003, P006, P008, P013, P014, P016, P018, P036, P038, P051, P060, P156, P170, P193
Reference implementation	12	P001, P015, P020, P021, P029, P042, P046, P086, P089, P090, P091, P197
Multiple backends	2	P080, P082
Multiple frameworks/libraries	2	P119, P120

Note: Categories are not mutually exclusive because some studies use multiple oracle types or combine several oracle-construction mechanisms.