Fuzzing OpenSSL with AI Test Agent
In this guide for UNIX-like platforms, we will automatically fuzz OpenSSL with CI Fuzz Spark.
OpenSSL is built using configure and make, for which CI Fuzz needs a custom build script outlined by the steps below.
Projects that use CMake or Bazel are usually easier to set up for fuzzing.
Requirements
- cifuzz version >=6.1.0
- clang llvm lcov libclang-rt-dev
- bear - install using
apt install bearor compile it from source https://github.com/rizsotto/Bear - OpenSSL is building
- Access to a state of the art LLM. See the LLM Setup guide to learn which LLMs are supported and how to set them up for use with cifuzz.
Download and build OpenSSL
First, download OpenSSL:
$ git clone --depth=1 https://github.com/openssl/openssl.git
Build it once and ensure that your system fulfills all prerequisites:
$ cd openssl
$ mkdir build
$ cd build
$ ../Configure no-tests
$ make -j
$ ls *.so *.a
libcrypto.a libcrypto.so libssl.a libssl.so
Now the build/ folder contains shared and static libraries and CI Fuzz should also be able to build OpenSSL.
The build/ folder will not be needed for the rest of this tutorial and can be removed.
Configure OpenSSL for CI Fuzz
In the top-level OpenSSL directory, initialize a cifuzz project by invoking:
$ cifuzz init
cifuzz version v5.8.1
✅ configuration saved in cifuzz.yaml
This step has created the file cifuzz.yaml, from which cifuzz loads its configuration.
In the next steps we will configure it to enable fuzzing of OpenSSL library.
Configure cifuzz.yaml for building OpenSSL
cifuzz has no information about how to compile projects like OpenSSL that use configure and make.
In this step, we will point cifuzz to a user bash script that can do the compilation.
Scroll to the following line in the cifuzz.yaml:
#build-command: make my_fuzz_test
Replace it by these lines instead:
build-command: cifuzz-spark/spark.sh ${FUZZ_TEST}
build-dir: build-cifuzz-fuzzing # directory containing compile_commands.json
build-file: cifuzz-spark/spark.sh
list-fuzz-tests-command: grep '^if \[ "$FUZZ_TEST" == ".*"' cifuzz-spark/spark.sh | awk '{print $5}' | sed 's/"//g' | sort | uniq
build-command-- tells cifuzz how to build a given fuzz test. Here we call the user scriptspark.shthat we will add in the next step.build-dir-- points cifuzz to the directory containingcompile_commands.jsonfor the OpenSSL project. The compile commands don't exist yet---we will generated them later when building OpenSSL.build-file-- tells cifuzz where the CI Fuzz Spark template resides that is used for building fuzz tests.list-fuzz-tests-command-- this expression will be used to find all fuzz tests generated by CI Fuzz. It is tailored to the CI Fuzz Spark template that we will add in the next step.
Make a user build script with a CI Fuzz Spark template
Make a new folder named cifuzz-spark in the top-level OpenSSL directory and copy the following into the script cifuzz-spark/spark.sh:
#!/usr/bin/env bash
set -e
SRC_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
TOP=`pwd`
##############################################################
## Part 1: Configure & build OpenSSL for fuzzing and coverage
##############################################################
# build in either: build-cifuzz-fuzzing or build-cifuzz-coverage
BUILD_DIR="build-cifuzz-${CIFUZZ_BUILD_STEP}"
CONFIGURE_COMMAND="../Configure no-shared no-err -fno-sanitize=alignment"
BUILD_COMMAND="make -j 8"
CFLAGS+=" -fno-sanitize=undefined"
CXXFLAGS+=" -fno-sanitize=undefined"
# To reconfigure OpenSSL, remove the build directories:
# rm -rf build-cifuzz-fuzzing build-cifuzz-coverage
if [ ! -d ${BUILD_DIR} ]; then
# configure and build OpenSSL once
mkdir -p ${BUILD_DIR}
cd ${BUILD_DIR}
if [ "${CIFUZZ_BUILD_STEP}" == "fuzzing" ]; then # build for fuzzing
${CONFIGURE_COMMAND}
# extract compile_commands.json while building for fuzzing
bear -- ${BUILD_COMMAND}
else # build for coverage
${CONFIGURE_COMMAND} no-tests
${BUILD_COMMAND}
fi
else # rebuild OpenSSL if needed
cd ${BUILD_DIR}
${BUILD_COMMAND}
fi
cd ${TOP}
##############################################################
## Part 2: Spark template for building fuzz targets
##############################################################
OBJ_DIR="${SRC_DIR}/.obj"
mkdir -p ${OBJ_DIR}
INCLUDES="-I${TOP}/include -I${TOP}/${BUILD_DIR}/include -I${SRC_DIR}"
LIBS="${BUILD_DIR}/libssl.a ${BUILD_DIR}/libcrypto.a"
# cifuzz:build-template:begin
#if [ "$FUZZ_TEST" == "{{ .FuzzTestName }}" ]; then
# echo "==== [Spark] building fuzz test for ${CIFUZZ_BUILD_STEP}: {{ .FuzzTestName }}"
# ${CC} ${CFLAGS} ${FUZZ_TEST_CFLAGS} -DOPENSSL_NO_ERR -c ${SRC_DIR}/${FUZZ_TEST}.cpp \
# -o ${OBJ_DIR}/${FUZZ_TEST}.o ${INCLUDES}
# ${CXX} ${CXXFLAGS} ${FUZZ_TEST_CXXFLAGS} ${FUZZ_TEST_LDFLAGS} ${OBJ_DIR}/${FUZZ_TEST}.o \
# -o ${SRC_DIR}/${FUZZ_TEST} ${LIBS}
#fi
#
# cifuzz:build-template:end
Make the script executable:
$ chmod +x cifuzz-spark/spark.sh
This script is called for every fuzz test CI Fuzz Spark tries to generate a fuzz test and calculate coverage for it.
In the first part of the script (lines 11-38), OpenSSL is built for fuzzing or for coverage.
CI Fuzz Spark signals the current step in the variable CIFUZZ_BUILD_STEP that is either "fuzzing" or "coverage".
If for some reason you need to rebuild OpenSSL (e.g. usually when you want to compile a different configuration, or use different compilation flags), you can delete build-cifuzz-fuzzing and build-cifuzz-coverage directories.
By default, cifuzz builds with active UBSAN.
We build OpenSSL without undefined sanitizer by changing environmental variables CFLAGS and CXXFLAGS in lines 15 and 16.
Running bear -- ${BUILD_COMMAND} in line 28 extracts the compile commands and saves them in build-cifuzz-fuzzing/compile_commands.json.
We build OpenSSL for fuzzing with examples and unit tests so that they appear in the compile commands and CI Fuzz Spark can use them when generating fuzz tests.
This usually results in higher quality fuzz tests.
The second part contains the CI Fuzz Spark template that is used for building individual fuzz tests. This script will be dynamically extended as Spark generates more fuzz tests. Newest fuzz tests appear at the bottom.
Ready to Spark
Let's run Spark on OpenSSL for at most 2 hours and try to write fuzz tests for at most 30 functions:
$ cifuzz spark --candidate-includes="crypto/**;ssl/**" -n 30 --max-runtime=2h -v
🚀 24 successful fuzz tests.
🚀 2 Findings detected.
🚀 2.25k Unique Test Cases.
🚀 12.02% total code coverage.
Used 4.43M LLM tokens in 1h19m8s.
🔍 Failed to generate fuzz tests for 6 function(s).
Increased code coverage (from fuzz tests) from 0.00% to 12.02%
We instruct Spark to only build fuzz tests for function in crypto/ and ssl/ folders by setting the flag --candidate-includes.
The first time Spark runs, it will take a few minutes to build OpenSSL for fuzzing in build-cifuzz-fuzzing/ folder and to analyze the code base for good fuzzing candidates.
After Spark has found a running fuzz test that can generate coverage, it will again take some time to build OpenSSL in the build-cifuzz-coverage/ instrumented for coverage.
According to the output above, Spark took about 1 hour 20 minutes to generate 24 fuzz tests (out of 30 target functions) and used around 4.43 million tokens. In this tutorial we used the gpt4o model.
Analyze the candidates
Take a look at the list of function candidates for which Spark automatically tries to generate fuzz tests. Spark starts with the top-most function that has the highest score in terms of the total number of lines of code (LoC) reachable from the function, its total cyclomatic complexity (CC), the number of dangerous expressions (DE), and the number of dangerous function calls (DFC).
$ cifuzz candidates --candidate-includes="crypto/**;ssl/**" -n 10
FUNCTION SCORE LoC ULoC CC DE DFC LOCATION
1 OSSL_CMP_CTX_server_perform 1.00 29785 - 11698 1116 120 crypto/cmp/cmp_server.c:762
2 OSSL_CMP_SRV_process_request 1.00 29775 - 11692 1116 120 crypto/cmp/cmp_server.c:581
3 OSSL_CMP_try_certreq 0.99 28833 - 11416 1140 120 crypto/cmp/cmp_client.c:823
4 OSSL_CMP_exec_certreq 0.99 28826 - 11414 1140 120 crypto/cmp/cmp_client.c:867
5 OSSL_CMP_exec_RR_ses 0.97 28278 - 11176 1123 120 crypto/cmp/cmp_client.c:897
6 ossl_statem_server_process_message 0.97 28149 - 11116 1087 156 ssl/statem/statem_srvr.c:1280
7 OSSL_CMP_get1_certReqTemplate 0.97 28120 - 11109 1119 120 crypto/cmp/cmp_genm.c:410
8 OSSL_CMP_get1_rootCaKeyUpdate 0.97 28123 - 11115 1117 120 crypto/cmp/cmp_genm.c:287
9 OSSL_CMP_get1_crlUpdate 0.97 28171 - 11136 1108 120 crypto/cmp/cmp_genm.c:350
10 OSSL_CMP_get1_caCerts 0.96 28084 - 11093 1109 120 crypto/cmp/cmp_genm.c:141
LoC - Total lines of code reachable from this function
ULoC - Total uncovered lines of code reachable from this function
CC - Total cyclomatic complexity reachable from this function
DE - Total number of dangerous expressions (pointer dereferences) reachable from this function
DFC - Total number of dangerous function calls (e.g. memcpy, strcpy, etc.) reachable from this function
To show the metrics for all possible functions in OpenSSL, use -n 0.
Manually improve the corpus
Let's start with a simple fuzz test in cifuzz-spark/fuzz_d2i_X509.cpp:
#include <openssl/x509.h>
#include <fuzzer/FuzzedDataProvider.h>
#include <cifuzz/cifuzz.h>
#include <vector>
FUZZ_TEST(const uint8_t *data, size_t size) {
FuzzedDataProvider fdp(data, size);
std::vector<uint8_t> cert = fdp.ConsumeRemainingBytes<uint8_t>();
const uint8_t *certData = cert.data();
X509 *x = d2i_X509(NULL, &certData, cert.size());
if (x != NULL) {
cert.data()[size+1] = 0; // intentional heap-buffer-overflow
X509_free(x);
}
}
To build the fuzz test, add the following to the build script cifuzz-spark/spark.sh:
# added at the end of the file
if [ "$FUZZ_TEST" == "fuzz_d2i_X509" ]; then
echo "==== [Spark] building fuzz test for ${CIFUZZ_BUILD_STEP}: fuzz_d2i_X509"
${CC} ${CFLAGS} ${FUZZ_TEST_CFLAGS} -DOPENSSL_NO_ERR -c ${SRC_DIR}/${FUZZ_TEST}.cpp \
-o ${OBJ_DIR}/${FUZZ_TEST}.o ${INCLUDES}
${CXX} ${CXXFLAGS} ${FUZZ_TEST_CXXFLAGS} ${FUZZ_TEST_LDFLAGS} ${OBJ_DIR}/${FUZZ_TEST}.o \
-o ${SRC_DIR}/${FUZZ_TEST} ${LIBS}
fi
Once a valid certificate is generated in line 10 of the fuzz test, we expect cifuzz to stop with a heap buffer overflow finding, because the fuzz test contains an intentional flaw in line 12. However, in some cases the fuzzer takes a while to generate a valid X509:
$ cifuzz run fuzz_d2i_X509 -v
...
#18326669 NEW cov: 1189 ft: 3114 corp: 526/54Kb lim: 4096 exec/s: 32039 rss: 585Mb L: 1558/3130 MS: 4 EraseBytes-ChangeBit-CopyPart-CopyPart-
#18330220 REDUCE cov: 1189 ft: 3114 corp: 526/54Kb lim: 4096 exec/s: 32045 rss: 585Mb L: 178/3130 MS: 1 EraseBytes-
#18331350 REDUCE cov: 1189 ft: 3114 corp: 526/54Kb lim: 4096 exec/s: 32047 rss: 585Mb L: 518/3130 MS: 5 ChangeByte-ChangeByte-ShuffleBytes-CopyPart-EraseBytes-
#18338885 REDUCE cov: 1189 ft: 3114 corp: 526/54Kb lim: 4096 exec/s: 32060 rss: 585Mb L: 12/3130 MS: 5 EraseBytes-ChangeASCIIInt-CopyPart-ChangeASCIIInt-InsertByte-
#18353782 REDUCE cov: 1189 ft: 3114 corp: 526/54Kb lim: 4096 exec/s: 32031 rss: 585Mb L: 57/3130 MS: 2 InsertByte-EraseBytes-
#18355054 REDUCE cov: 1189 ft: 3114 corp: 526/54Kb lim: 4096 exec/s: 32033 rss: 585Mb L: 596/3130 MS: 2 ChangeByte-EraseBytes-
Let's help it out by manually adding a valid certificate to the corpus directory of the fuzz test.
Each fuzz test has it's corresponding corpus directory in .cifuzz-corpus/
For our fuzz test, the corpus is in .cifuzz-corpus/fuzz_d2i_X509/ directory.
Let's manually add a valid X509 certificate to the corpus using OpenSSL that we have built in build-cifuzz-fuzzing/apps/openssl:
export OPENSSL_CONF=./apps/openssl.cnf
corpus_dir=".cifuzz-corpus/fuzz_d2i_X509"
./build-cifuzz-fuzzing/apps/openssl genrsa -out $corpus_dir/private_key.pem 2048
./build-cifuzz-fuzzing/apps/openssl req -new -key $corpus_dir/private_key.pem -out $corpus_dir/csr.pem -subj "/C=US/ST=California/L=San Francisco/O=My Company/CN=mydomain.com"
./build-cifuzz-fuzzing/apps/openssl x509 -req -days 365 -in $corpus_dir/csr.pem -signkey $corpus_dir/private_key.pem -outform der -out $corpus_dir/certificate.pem
Running the bash script in the top-level directory of OpenSSL project adds a X509 certificate to the corpus directory of our fuzz test:
$ bash improve_fuzz_d2i_X509_corpus.sh
Certificate request self-signature ok
subject=C=US, ST=California, L=San Francisco, O=My Company, CN=mydomain.com
If we run the fuzz test again, the coverage rapidly increases and the heap buffer overflow is found:
$ cifuzz run fuzz_d2i_X509 -v
#768541 REDUCE cov: 1189 ft: 3122 corp: 398/41Kb lim: 4096 exec/s: 33414 rss: 488Mb L: 91/3130 MS: 4 ChangeASCIIInt-InsertByte-PersAutoDict-EraseBytes- DE: "\376\326\232\233"-
NEW_FUNC[1/592]: 0x555b62616bd0 in X509_ALGOR_get0 /openssl/build-cifuzz-fuzzing/../crypto/asn1/x_algor.c:74
NEW_FUNC[2/592]: 0x555b62618520 in BIO_new_ex /openssl/build-cifuzz-fuzzing/../crypto/bio/bio_lib.c:82
#769855 RELOAD cov: 3016 ft: 5376 corp: 399/42Kb lim: 4096 exec/s: 33471 rss: 489Mb
#769857 NEW cov: 3016 ft: 5377 corp: 400/42Kb lim: 4096 exec/s: 33472 rss: 489Mb L: 71/3130 MS: 1 ChangeBit-
NEW_FUNC[1/7]: 0x555b62626c00 in BN_clear_free /openssl/build-cifuzz-fuzzing/../crypto/bn/bn_lib.c:215
NEW_FUNC[2/7]: 0x555b6262d790 in BN_MONT_CTX_free /openssl/build-cifuzz-fuzzing/../crypto/bn/bn_mont.c:251
#770258 NEW cov: 3061 ft: 5725 corp: 401/42Kb lim: 4096 exec/s: 33489 rss: 490Mb L: 859/3130 MS: 1 PersAutoDict- DE: "\376\326\232\233"-
#774205 NEW cov: 3062 ft: 5726 corp: 402/43Kb lim: 4096 exec/s: 33661 rss: 490Mb L: 867/3130 MS: 1 CMP- DE: "\000\000\000\000\000\000\000\000"-
#774850 NEW cov: 3062 ft: 5728 corp: 403/44Kb lim: 4096 exec/s: 33689 rss: 490Mb L: 879/3130 MS: 3 ChangeASCIIInt-PersAutoDict-PersAutoDict- DE: "\377\377\377\027"-"\001\000\000\000\000\000\0002"-
#774876 NEW cov: 3062 ft: 5807 corp: 404/45Kb lim: 4096 exec/s: 33690 rss: 490Mb L: 884/3130 MS: 1 InsertRepeatedBytes-
#774952 NEW cov: 3063 ft: 5809 corp: 405/46Kb lim: 4096 exec/s: 33693 rss: 490Mb L: 883/3130 MS: 1 PersAutoDict- DE: "\377\377\377\027"-
#775020 NEW cov: 3063 ft: 5812 corp: 406/47Kb lim: 4096 exec/s: 33696 rss: 490Mb L: 880/3130 MS: 3 ChangeBinInt-ShuffleBytes-InsertByte-
#775191 NEW cov: 3063 ft: 5813 corp: 407/48Kb lim: 4096 exec/s: 33703 rss: 490Mb L: 940/3130 MS: 1 InsertRepeatedBytes-
#775285 NEW cov: 3063 ft: 5814 corp: 408/49Kb lim: 4096 exec/s: 33708 rss: 490Mb L: 859/3130 MS: 3 ShuffleBytes-ChangeBit-PersAutoDict- DE: "\376\326\232\233"-
==22663==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5180000273d8 at pc 0x5609466237a2 bp 0x7ffe42f84d50 sp 0x7ffe42f84d48
WRITE of size 1 at 0x5180000273d8 thread T0
#0 0x5609466237a1 in LLVMFuzzerTestOneInputNoReturn(unsigned char const*, unsigned long) /openssl/cifuzz-spark/fuzz_d2i_X509.cpp:12:29
#1 0x5609466237a1 in LLVMFuzzerTestOneInput /.local/share/cifuzz/include/cifuzz/cifuzz.h:36:3
#2 0x5609464a4df8 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614
#3 0x5609464a8fe3 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:516
#4 0x5609464aa0c7 in fuzzer::Fuzzer::RereadOutputCorpus(unsigned long) (.part.389) /llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:433
#5 0x5609464ac8ce in fuzzer::Fuzzer::RereadOutputCorpus(unsigned long) /usr/include/c++/7/ext/new_allocator.h:125
#6 0x5609464ac8ce in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) /llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:883
#7 0x560946490140 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:915
#8 0x56094647af32 in main /llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20
#9 0x7f0ca023e24c in __libc_start_main (/lib64/libc.so.6+0x3524c) (BuildId: f732026552f6adff988b338e92d466bc81a01c37)
#10 0x56094647afb9 in _start /home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:120
0x5180000273d8 is located 1 bytes after 855-byte region [0x518000027080,0x5180000273d7)
allocated by thread T0 here:
#0 0x560946620028 in operator new(unsigned long) /llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:86
#1 0x560946623894 in __gnu_cxx::new_allocator<unsigned char>::allocate(unsigned long, void const*) /usr/lib64/gcc/x86_64-suse-linux/11/../../../../include/c++/11/ext/new_allocator.h:127:27
#2 0x560946623894 in std::allocator_traits<std::allocator<unsigned char>>::allocate(std::allocator<unsigned char>&, unsigned long) /usr/lib64/gcc/x86_64-suse-linux/11/../../../../include/c++/11/bits/alloc_traits.h:464:20
#3 0x560946623894 in std::_Vector_base<unsigned char, std::allocator<unsigned char>>::_M_allocate(unsigned long) /usr/lib64/gcc/x86_64-suse-linux/11/../../../../include/c++/11/bits/stl_vector.h:346:20
#4 0x560946623894 in std::_Vector_base<unsigned char, std::allocator<unsigned char>>::_M_create_storage(unsigned long) /usr/lib64/gcc/x86_64-suse-linux/11/../../../../include/c++/11/bits/stl_vector.h:361:33
#5 0x560946623894 in std::_Vector_base<unsigned char, std::allocator<unsigned char>>::_Vector_base(unsigned long, std::allocator<unsigned char> const&) /usr/lib64/gcc/x86_64-suse-linux/11/../../../../include/c++/11/bits/stl_vector.h:305:9
#6 0x560946623894 in std::vector<unsigned char, std::allocator<unsigned char>>::vector(unsigned long, std::allocator<unsigned char> const&) /usr/lib64/gcc/x86_64-suse-linux/11/../../../../include/c++/11/bits/stl_vector.h:511:9
#7 0x560946623894 in std::vector<unsigned char, std::allocator<unsigned char>> FuzzedDataProvider::ConsumeBytes<unsigned char>(unsigned long, unsigned long) /.local/share/cifuzz/include/fuzzer/FuzzedDataProvider.h:373:18
#8 0x5609466236a5 in std::vector<unsigned char, std::allocator<unsigned char>> FuzzedDataProvider::ConsumeBytes<unsigned char>(unsigned long) /.local/share/cifuzz/include/fuzzer/FuzzedDataProvider.h:122:10
#9 0x5609466236a5 in std::vector<unsigned char, std::allocator<unsigned char>> FuzzedDataProvider::ConsumeRemainingBytes<unsigned char>() /.local/share/cifuzz/include/fuzzer/FuzzedDataProvider.h:141:10
#10 0x5609466236a5 in LLVMFuzzerTestOneInputNoReturn(unsigned char const*, unsigned long) /openssl/cifuzz-spark/fuzz_d2i_X509.cpp:8:37
Debugging findings in Visual Studio Code
To see the list of findings, run cifuzz finding:
$ cifuzz finding
cifuzz version v6.3.0-3-gb2f0b950
DATE SEVERITY NAME DESCRIPTION FUZZ TEST LOCATION ASSESSMENT
3m ago 9.0 quiet_roadrunner heap_buffer_overflow fuzz_d2i_X509 cifuzz-spark/fuzz_d2i_X509.cpp:12:29 n/a
Debugging with VSCode can be started by adding a configuration to .vscode/launch.json file:
{
"version": "0.2.0",
"configurations": [
{
"name": "Debug fuzz_d2i_X509",
"type": "cppdbg",
"request": "launch",
"program": "${workspaceFolder}/cifuzz-spark/fuzz_d2i_X509",
"args": [
"${workspaceFolder}/.cifuzz-findings/quiet_roadrunner/crashing-input",
"-runs=0"
],
"cwd": "${workspaceFolder}",
"environment": [
{
"name": "ASAN_OPTIONS",
"value": "symbolize=1:halt_on_error=1:detect_leaks=1"
},
],
"externalConsole": false,
"MIMode": "gdb",
"setupCommands": [
{
"description": "Enable pretty-printing for gdb",
"text": "-enable-pretty-printing",
"ignoreFailures": true
}
]
},
]
}
Now press Run and Debug button on the left of VSCode, and then Start Debugging (F5) after selecting "Debug fuzz_d2i_X509" as target:
Now you can set breakpoints, step through the code, see the values of variables, etc.
Giving hints to the LLM
Additional hints can be given to the LLM by pointing it to the user hint files in the cifuzz.yaml config:
llm-hint-files:
- llm-hits.txt
To double-check that the hints are actually sent to the LLM, run Spark with the flag --log-llm-messages:
$ cifuzz spark --candidate-includes="crypto/**;ssl/**" -n 90 --max-runtime=20m --log-llm-messages
All the messages sent to the LLM and their responses are logged in .cifuzz/logs/llm.log.
Skipping candidates
It is possible to tell Spark to skip individual or already candidates by adding the following to cifuzz.yaml:
# This skips all functions that Spark has tried already to generate a fuzz test for:
skip-already-tried-candidates: true
# This skips individual functions:
skip-candidates:
- OSSL_CMP_CTX_server_perform
- OSSL_CMP_try_certreq
Calculating coverage
# For all fuzz tests at once
$ cifuzz coverage --format lcov
# Only for fuzz_OSSL_CMP_exec_certreq_2
$ cifuzz coverage fuzz_OSSL_CMP_exec_certreq_2 --format lcov