Automatically create fuzz tests

This page is still under construction.

You can create a fuzz test for a specific function in your project or a general fuzz test stub with the following command:

cifuzz create [flags] [function] [--] [<build system arg>...]

Command options

Flag	Default	Description
--ast-compiler-arg <stringArray>		Additional compiler arguments passed to clang tooling when analyzing the source code.
--build-command <string>		Command to build the fuzz test for build system "other".
-b, --build-dir <string>		Directory containing compilation database "compile_commands.json".
--build-jobs <uint>	8	Maximum number of concurrent processes used for building. If set to zero, the default of the native build tool is used. Defaults to the number of cores on your machine.
--detect-system-includes	true	Automatically detect system include directories for the compiler.
--engine-arg <stringArray>		Command-line arguments to pass to the fuzzing engine. This flag can be used multiple times. For libFuzzer see: https://llvm.org/docs/LibFuzzer.html#options For AFL see: https://www.mankier.com/8/afl-fuzz
--fuzz-test-name <string>		Fuzz test to execute when validating LLM generated test.
--interactive	true	Toggle interactive prompting in the terminal.
--json-summary	false	Additionally print summary of the command as JSON.
--list-fuzz-tests-command <string>		Provide a command that lists the names of already existing fuzz tests in your project. The listed names should be separated by whitespace and can't include whitespaces themselves, i.e. 'fuzz_test1 fuzz_test_2 ...'
--llm-hint-file <stringArray>		Path to hint file with custom prompts for the LLM.
--log-llm-messages	false	Log messages sent to the LLM server.
--max-fuzzing-duration <duration>	10m	Maximum time to run all fuzz tests ("5s", "30m", "1h", ...) Time will be split up evenly if multiple fuzz tests are run. To keep running indefinitely, set value to 0.
--max-idle-time <duration>	10s	Maximum time until fuzz test is aborted if no progress is detected ("5s", "30m", "1h", ...)
--max-variants <uint>	3	Number of fuzz test variants to evaluate for a single function. More variants can lead to higher quality fuzz tests but also increase the LLM token count.
--min-severity <string>	medium	Minimum severity of Findings: "low", "medium", "high" or "critical" Command will exit with status code 99 if Finding with matching severity was detected. Findings with lower severity are marked as a warning.
-r, --num-retries <uint>	3	Number of times cifuzz tries to generate a valid fuzz test for a function candidate.
-o, --output <string>		Output path of created fuzz test.
--preserve-present-findings	false	Findings marked as "present" from other fuzz tests will still remain marked. This can be imprecise if fuzz tests are renamed, deleted or ambiguously identified.
--usage-excludes <stringArray>	[.cifuzz/**]	Exclude files from source file analysis via glob patterns. This can be helpful to exclude third party dependencies to speed up fuzzing. Multiple exclude glob patterns are possible by setting the flag several times or by using ";" as a separator. Example: --usage-excludes "examples/**;build/**"
-u, --usage-includes <stringArray>		Include files in source file analysis via glob patterns. By default the project directory is included. Multiple exclude glob patterns are possible by setting the flag several times or by using ";" as a separator. Example: --usage-includes "src/**;test/**"
--validate	false	Enable automatic validation and refinement of the generated fuzz test.

Create fuzz test for a specific function

You can add a function name as an argument to the create command to generate a specific fuzz test. The candidates command is a useful resource to create a list of suitable functions.

cifuzz create exampleFunction

If multiple functions with the same name exist, you can choose the function with the right signature in an interactive selection during the command (if --interactive wasn't turned off).

The command prints the created fuzz test on the terminal, or you can specify an output path with the -o/--output flag.

cifuzz create exampleFunction -o=example_fuzz_test.cpp

To be able to build and run the fuzz test with the cifuzz run command afterward, you have to integrate it into your project depending on your build system. You can find a guide on the "Write A Fuzz Tests" page.

Validate generated fuzz test

You can automatically validate and run the created fuzz test during the create command with the --validate flag. This process requires a functioning LLM and AI test agent setup. Please follow the instructions on the linked pages.

To test if the created fuzz test compiles and runs as expected, you have to integrate the desired fuzz test into your project before running the command so that the AI test agent can build it. The test file doesn't have to exist, only the configuration in your build system. If your project uses a build system other than Bazel or CMake, you have to specify a command to build fuzz tests with the --build-command flag (or configure it in your projects cifuzz.yaml).

cifuzz create exampleFunction -o=example_fuzz_test.cpp --validate

This command generates the fuzz test, validates and corrects it until it compiles without errors, saves it in the specified path and prints the results of the fuzzing run on the console. You can specify the number of times the command tries to generate and correct the fuzz test until it's valid with the --num-retries flag (default is 3).

Include/exclude specific usage examples

You can specify source files the LLM analyses and uses as examples for fuzz test generation with the -u/ --usage-includes flag. This overwrites the default set to the project directory. You can exclude multiple files either by using the flag multiple times or separate multiple glob patterns with a semicolon (";").

--usage-includes="src/**" --usage-includes="test/**"

--usage-includes="src/**;test/**"

You can also configure the flag in the projects cifuzz.yaml.

cifuzz.yaml

usage-includes: 
  - src/**
  - test/**

Analog you can also exclude specific files with the --usage-excludes flag.

LLM hints

If you need to adjust the LLM with specific instructions, you can add a path to a file with custom prompts with the --llm-hint-file flag. If you want to specify more than one file, you can use the flag multiple times or configure a llm-hint-files list in the projects cifuzz.yaml.