CI Fuzz Spark

This page is still under construction.

You can automatically generate fuzz tests for your project using the AI Test Agent with the cifuzz spark command:

cifuzz spark [--option...]

Requirements

An LLM deployment is necessary for CI Fuzz Spark usage. Please refer to LLM-Setup for setup instructions.

CI Fuzz Spark generates, builds and runs new fuzz tests. To be able to do that, an integration in your build system is necessary. This consists of three parameters:

• The build directory containing a compile_commands.json
• The build file containing a build target template
• The build command (not necessary for CMake and Bazel projects)

Build directory

In order to analyze your projects code, you must specify a build directory that contains your projects build information in form of a compile_commands.json.

Generate compile_commands.json

CMake

For CMake projects, CI Fuzz can generate the compile commands on its own, but it needs a CMake build directory.

Bazel

For Bazel projects, you can use the Hedron Bazel Compile Commands Extractor to create a compile_commands.json.

Other

For other projects, you can use the command bear to generate a compile_commands.json. If you are using make, for example, use

bear -- make

Use the flag --build-dir <string> or set build-dir: in your cifuzz.yaml to specify the build directory.

Build file and templating

In order to add new fuzz tests to your build system, you must specify a build file that contains a target template.

The build file template syntax is

#cifuzz:build-template:begin
#<specify build target here>
#cifuzz:build-template:end

The following templating parameters are available:

• {{.FileName}}: The filename of the fuzz test, for example: MyFuzzTest.cpp.
• {{.FuzzTestName}}: The fuzz test name and the name of the resulting binary, for example: MyFuzzTest.

The build target template should create build targets that respect the following environment variables:

• FUZZ_TEST_CFLAGS / FUZZ_TEST_CXXFLAGS and FUZZ_TEST_LDFLAGS for compiling the fuzz tests,
• CFLAGS and LDFLAGS for compiling C projects (alternatively CIFUZZ_CFLAGSand CIFUZZ_LDFLAGS),
• CXXFLAGS and LDFLAGS for compiling C++ projects (alternatively CIFUZZ_CXXFLAGSand CIFUZZ_LDFLAGS).

Use the flag --build-file <string> or set build-file: in your cifuzz.yaml to specify the build file.

Examples

CMake

#cifuzz:build-template:begin
#add_fuzz_test({{ .FuzzTestName }} {{ .FileName }})
#target_link_libraries({{ .FuzzTestName }} PRIVATE "$<LINK_LIBRARY:WHOLE_ARCHIVE,${CMAKE_PROJECT_NAME}>")
#cifuzz:build-template:end

Bazel

#cifuzz:build-template:begin
#cc_fuzz_test(
#    name = "{{ .FuzzTestName }}",
#    srcs = [
#        "{{ .FileName }}",
#    ],
#    corpus = glob(
#        ["{{ .FuzzTestName }}_inputs/**"],
#        allow_empty = True,
#    ) + select({
#        "@cifuzz//:collect_coverage": glob([".{{ .FuzzTestName }}_cifuzz_corpus/**"], allow_empty = True),
#        "//conditions:default": [],
#    }),
#    deps = [
#        "@cifuzz",
#    ],
#)
#cifuzz:build-template:end

Makefile

#cifuzz:build-template:begin
#{{.FuzzTestName}}: {{.FileName}}
#       $(CXX) $(CXXFLAGS) $(FUZZ_TEST_CXX_FLAGS) $(FUZZ_TEST_LDFLAGS) -I<include directories> {{.FileName}} <project artifact> -o {{.FuzzTestName}}
#cifuzz:build-template:end

Build command for other build systems

If you aren't using CMake or Bazel, you must specify a build command that will build a fuzz test to allow CI Fuzz to build the generated fuzz tests. The build command must compile the fuzz tests given in $FUZZ_TEST into a binary called $CIFUZZ_BUILD_LOCATION.

The following environment variables will be set:

• FUZZ_TEST: The fuzz test name,
• CIFUZZ_BUILD_LOCATION: The name of the resulting binary,
• CIFUZZ_BUILD_STEP: The build step, can be fuzzing or coverage,
• CIFUZZ_COMMAND: The cifuzz command, will always be spark for invocations of cifuzz spark.

Furthermore the following compile flags will be set that should be respected by the build target template or the build command

• FUZZ_TEST_CFLAGS
• FUZZ_TEST_CXXFLAGS
• FUZZ_TEST_LDFLAGS
• CIFUZZ_CFLAGS
• CIFUZZ_CXXFLAGS
• CIFUZZ_LDFLAGS

The build command will be executed in a POSIX compatible shell, so you can concatenate multiple commands with ;, && and ||. Of course you can also specify a build script.

Use the flag --build-command <string> or set build-command: in your cifuzz.yaml to specify the build command.

Note: The build command must always re-build the binary and all dependencies. For coverage-calculation, the same build steps are executed with a different CIFUZZ_BUILD_STEP and different compile flags.

Example

build-command: "make -C lib clean && rm -f lib/$FUZZ_TEST && make -C lib -j 6 && make -C lib V=1 $FUZZ_TEST"

Options

The cifuzz spark command supports the following command specific options:

Flag	Description
--ast-compiler-arg <stringArray>	Additional compiler arguments passed to clang tooling when analyzing the source code.
--build-command <string>	Command to build the fuzz test for build system "other".
-b <string> --build-dir <string>	Directory containing compilation database "compile_commands.json".
--build-file <string>	Build file containing a template used for fuzz test generation.
--build-jobs <uint>	Maximum number of concurrent processes used for building. If set to zero, the default of the native build tool is used.Defaults to the number of cores on your machine. (default 12)
-e <stringArray> --candidate-excludes <stringArray>	Exclude files from analysis via glob patterns. Multiple exclude glob patterns are possible by setting the flag several times or by using ";" as a separator. Example: --candidate-excludes "examples/**;tests/**" (default **/*_test.*;test*/**;.cifuzz/**/_deps/**)
-i <stringArray> --candidate-includes <stringArray>	Include files to analyze for candidate selection via glob patterns. By default, only the project directory is included. Any values provided here will overwrite the default pattern. To include the project directory again, add it using an absolute path. Multiple include glob patterns are possible by setting the flag several times or by using ";" as a separator. Example: To include only files in "<projectRoot>/src" and "/home/<user>/project" for all users in candidate analysis: --candidate-includes src/**;/home/*/project/**
--coverage-excludes <stringArray>	Exclude files from coverage reporting via glob patterns. Double asterisks (**) match any number of directories.
--coverage-includes <stringArray>	Include files in coverage reporting via glob patterns. Double asterisks (**) match any number of directories. (default **/**)
--detect-system-includes	Automatically detect system include directories for the compiler. (default true)
--engine-arg <stringArray>	Command-line arguments to pass to the fuzzing engine. Thisflag can be used multiple times. For libFuzzer see: https://llvm.org/docs/LibFuzzer.html#options For AFL see: https://www.mankier.com/8/afl-fuzz
--list-fuzz-tests-command <string>	Provide a command that lists the names of already existing fuzz tests in your project. The listed names should be separated by whitespace and can't include whitespaces themselves, i.e. fuzz_test1 fuzz_test_2 ...
--llm-hint-file <stringArray>	Path to hint file with custom prompts for the LLM.
--log-llm-messages	Log messages sent to the LLM server.
--max-fuzzing-duration <duration>	Maximum time to run all fuzz tests ("5s", "30m", "1h", ...) Time will be split up evenly if multiple fuzz tests are run. To keep running indefinitely, set value to 0.
--max-idle-time <duration>	Maximum time until fuzz test is aborted if no progress is detected ("5s", "30m", "1h", ...) (default 10s)
--max-runtime <duration>	Maximum runtime for spark ("5s", "30m", "1h", ...). No new fuzz tests will be generated after time exceeds. To keep running indefinitely, set value to 0. (default 30m0s)
--max-variants <uint>	Number of fuzz test variants to evaluate for a single function. More variants can lead to higher quality fuzz tests but also increase the LLM token count (default 3)
--min-severity <string>	Minimum severity of Findings: "low", "medium", "high" or "critical" Command will exit with status code 99 if Finding with matching severity was detected. Findings with lower severity are marked as a warning. (default "medium")
-n <uint> --num-candidates <uint>	Number of candidates to select. (default 5)
-r <uint> --num-retries <uint>	Number of times cifuzz tries to generate a valid fuzz test for a function candidate. (default 3)
--preserve-present-findings	Findings marked as "present" from other fuzz tests will still remain marked. This can be imprecise if fuzz tests are renamed, deleted or ambiguously identified.
--skip-already-tried-candidates	Skip candidates that have already been tried in previous runs. Reset tried candidates by deleting ".cifuzz/spark/tried-candidates.log"
--skip-candidate <stringArray>	Skip candidate during spark generation.
--target-coverage <int>	Specify target coverage percentage (%) at which to stop generating new fuzz tests. Includes coverage of pre-existing fuzz tests. (default 100)
--usage-excludes <stringArray>	Exclude files from source file analysis via glob patterns. This can be helpful to exclude third party dependencies to speed up fuzzing. Multiple exclude glob patterns are possible by setting the flag several times or by using ";" as a separator. Example: --usage-excludes "examples/**;build/**" (default .cifuzz/**)
-u <stringArray> --usage-includes <stringArray>	Include files in source file analysis via glob patterns. By default the project directory is included. Multiple exclude glob patterns are possible by setting the flag several times or by using ";" as a separator. Example: --usage-includes "src/**;test/**"