Run a fuzz test with CI Sense
This page describes how to run a fuzz test on CI Sense, both manually and as part of a CI/CD pipeline. This primarily applies to non web API fuzz tests. For details on how to run web API fuzz tests, see the relevant sections in web API fuzzing.
Overview
To run a fuzz test on CI Sense you need to follow these steps:
- Create a bundle, which is your software built with instrumentation the fuzzer can use to run the fuzz tests and evaluate coverage.
- Upload the bundle to CI Sense.
- Start the fuzz tests on CI Sense.
- Monitor the fuzz test (only needed for CI/CD runs)
A fuzz run on CI Sense consists of 3 jobs:
- Fuzz job
- Coverage job
- Cleanup job
The fuzz job
For fuzzers that support value profiling (libfuzzer and Jazzer), you need to create two fuzz jobs for each fuzz test that's part of the bundle. One fuzz job has value profiling enabled, the other doesn't. Value profiling can lead to improved code coverage, but also slows down the execution speed of the job. Both runs share the same corpus. Each fuzz job executes inside of its own Docker container.
The coverage job
A coverage job starts after all fuzz runs have completed. During the coverage job, the coverage binaries created as part of the bundle are executed using the generated corpus to determine the total coverage for the run.
Cleanup job
The cleanup job removes coverage information generated during web API fuzzing.
View the Job status
In the CI Sense UI, on the Dashboard or on a specific Project Overview, you can see the Job Status of the most recent fuzz run. You can see the status of previous fuzz runs by selecting the dropdown in the top right of the Project Overview. See an example below.
The Job Status can have the following values:
- FAILED - all jobs for the run failed to start.
- INCOMPLETE - at least 1 job started, but not all jobs were successful.
- IN PROGRESS - at least 1 job is still running.
- SUCCEEDED - all jobs completed.
Manually upload fuzz tests to CI Sense
There are a few ways to manually upload fuzz tests to your CI Sense server. This is useful when you are initially setting up your CI/CD pipeline or simply want to test a new fuzz test.
Create a bundle with CI Fuzz
If your project is already configured with CI Fuzz, you can create a bundle by running the following command from the root of the project directory:
cifuzz bundle
- This command builds and bundles the runtime artifacts needed to run your fuzz tests on CI Sense
- You can specify the name of one or more fuzz tests. If you don't specify any
fuzz tests, then it builds all fuzz tests for the project. If the
build-system
isother
, then you must specify the fuzz tests explicitly. - You can specify various arguments telling CI Sense how to run your fuzz tests. See this page for additional details.
- The output is a tarball of the form
fuzz_tests.tar.gz
or<name_of_specific_fuzz_test>.tar.gz
if you specified one.
After creating the bundle file, you need to upload it to the CI Sense server and start the fuzz tests. You can either:
- Drag and drop it in the Web UI on the project card you want to run it under. You can do this from the Dashboard or from the Overview after selecting your project.
- Use the
cictl
command (located inci-app-install-directory/bin/cictl
) to upload or import the bundle to the CI Sense server and then start it.
Here is an example bash script that uses cictl
to upload and run the fuzz
tests, created using cifuzz bundle
, to the CI Sense server.
CI_FUZZ_API_TOKEN=<token>
FUZZING_SERVER="<address of fuzzing server>"
PROJECT="<project name>"
CICTL_CMD="cictl -s $FUZZING_SERVER"
ARTIFACT_NAME=$($CICTL_CMD import artifact fuzz_tests.tar.gz --project-name "$PROJECT")
$CICTL_CMD start ${ARTIFACT_NAME}
- Most
cictl
commands require the--server, -s
flag to specify the location of the fuzzing server, so be sure to include it - The
CI_FUZZ_API_TOKEN
is either a token you generated in the CI Sense UI or the password you specified if you configure CI Sense that way. - You can also obtain the project name by running
cictl -s <fuzzing server> list projects
. The project name should be of the formprojects/my_project-391A177B
.
CI Fuzz remote run
The CI Fuzz remote-run command bundles, uploads, and starts your fuzz tests
with one command. This method doesn't require cictl
to upload and start the
fuzz tests, but if you want to monitor the fuzz test, you still need
cictl
.
cifuzz remote-run --server <server address>
- You can authenticate to the server automatically by passing the environment
variable
CIFUZZ_API_TOKEN
as part of your command (e.g.CIFUZZ_API_TOKEN=<token> cifuzz remote-run --server "https://cifuzz.server.com"
). - You can specify the name of one or more fuzz tests. If you don't specify any,
then
remote-run
bundles and runs all the fuzz tests for a project. - You can specify various arguments telling CI Sense how to run your fuzz tests. See this page for additional details.
- You can pass an existing bundle by simply specifying the
--bundle
flag and the path to the bundle. When passing an existing bundle, CI Fuzz ignores any runtime flags passed as part of theremote-run
command (e.g.--timeout
,--docker-image
).
Run fuzz tests with CI/CD
To create a CI/CD pipeline, you need to install both
CI Fuzz and the cictl
command
line tool. CI Sense must be reachable from the CI/CD server.
The general workflow is:
- Build/bundle the artifacts that run on the server
- Upload the artifacts to the server and run them
- Monitor the results
Here is a script you can use to help write and test your CI/CD pipeline. This
script uses cifuzz bundle
in conjunction with cictl
to create the bundle,
upload it, start it, and then monitor it.
#! /usr/bin/bash
# how long to monitor the fuzz test
TIMEOUT=300
# this should match wherever CI Sense is currently reachable
FUZZING_SERVER="127.0.0.1:8080"
# address of the web application
WEB_APP_ADDRESS="http://127.0.0.1:8080"
# project name of the form: projects/c-cpp-demo-0b5fbe28
# this can be obtained by using cictl -s <server> list projects
PROJECT=<project name>
# access token generated in the CI Sense web app
CI_FUZZ_API_TOKEN=
# local directory where the project is located
LOCAL_REPO=
# create bundle from local repo
# you can specify additional bundle options at the command line, or include them in cifuzz.yaml
cd $LOCAL_REPO
cifuzz bundle
CICTL_CMD="cictl -s $FUZZING_SERVER"
# login to the CI Sense server
echo $CI_FUZZ_API_TOKEN | $CICTL_CMD login
# upload the bundle to the CI Sense server
ARTIFACT_NAME=$($CICTL_CMD import artifact fuzz_tests.tar.gz --project-name "$PROJECT")
# start the fuzz tests in the bundle
CAMPAIGN_RUN=$($CICTL_CMD start --application-base-url "${WEB_APP_ADDRESS}" "${ARTIFACT_NAME}")
# monitor the output from the campaign_run
$CICTL_CMD monitor_campaign_run --dashboard_address="${WEB_APP_ADDRESS}" --keep_running --duration="${TIMEOUT}" ${CAMPAIGN_RUN}