Skip to main content

Run a fuzz test with CI Sense

This page describes how to run a fuzz test on CI Sense, both manually and as part of a CI/CD pipeline. This primarily applies to non web API fuzz tests. For details on how to run web API fuzz tests, see the relevant sections in web API fuzzing.


Overview

To run a fuzz test on CI Sense you need to follow these steps:

  • Create a bundle, which is your software built with instrumentation the fuzzer can use to run the fuzz tests and evaluate coverage.
  • Upload the bundle to CI Sense.
  • Start the fuzz tests on CI Sense.
  • Monitor the fuzz test (only needed for CI/CD runs)

A fuzz run on CI Sense consists of 3 jobs:

  • Fuzz job
  • Coverage job
  • Cleanup job

The fuzz job

For fuzzers that support value profiling (libfuzzer and Jazzer), you need to create two fuzz jobs for each fuzz test that's part of the bundle. One fuzz job has value profiling enabled, the other doesn't. Value profiling can lead to improved code coverage, but also slows down the execution speed of the job. Both runs share the same corpus. Each fuzz job executes inside of its own Docker container.

The coverage job

A coverage job starts after all fuzz runs have completed. During the coverage job, the coverage binaries created as part of the bundle are executed using the generated corpus to determine the total coverage for the run.

Cleanup job

The cleanup job removes coverage information generated during web API fuzzing.

View the Job status

In the CI Sense UI, on the Dashboard or on a specific Project Overview, you can see the Job Status of the most recent fuzz run. You can see the status of previous fuzz runs by selecting the dropdown in the top right of the Project Overview. See an example below.

The Job Status can have the following values:

  • FAILED - all jobs for the run failed to start.
  • INCOMPLETE - at least 1 job started, but not all jobs were successful.
  • IN PROGRESS - at least 1 job is still running.
  • SUCCEEDED - all jobs completed.

succeeded job status of private project at right corner of project tile

Manually upload fuzz tests to CI Sense

There are a few ways to manually upload fuzz tests to your CI Sense server. This is useful when you are initially setting up your CI/CD pipeline or simply want to test a new fuzz test.

Create a bundle with CI Fuzz

If your project is already configured with CI Fuzz, you can create a bundle by running the following command from the root of the project directory:

cifuzz bundle
  • This command builds and bundles the runtime artifacts needed to run your fuzz tests on CI Sense
  • You can specify the name of one or more fuzz tests. If you don't specify any fuzz tests, then it builds all fuzz tests for the project. If the build-system is other, then you must specify the fuzz tests explicitly.
  • You can specify various arguments telling CI Sense how to run your fuzz tests. See this page for additional details.
  • The output is a tarball of the form fuzz_tests.tar.gz or <name_of_specific_fuzz_test>.tar.gz if you specified one.

After creating the bundle file, you need to upload it to the CI Sense server and start the fuzz tests. You can either:

  • Drag and drop it in the Web UI on the project card you want to run it under. You can do this from the Dashboard or from the Overview after selecting your project.
  • Use the cictl command (located in ci-app-install-directory/bin/cictl) to upload or import the bundle to the CI Sense server and then start it.

Here is an example bash script that uses cictl to upload and run the fuzz tests, created using cifuzz bundle, to the CI Sense server.

CI_FUZZ_API_TOKEN=<token>
FUZZING_SERVER="<address of fuzzing server>"
PROJECT="<project name>"
CICTL_CMD="cictl -s $FUZZING_SERVER"
ARTIFACT_NAME=$($CICTL_CMD import artifact fuzz_tests.tar.gz --project-name "$PROJECT")
$CICTL_CMD start ${ARTIFACT_NAME}
  • Most cictl commands require the --server, -s flag to specify the location of the fuzzing server, so be sure to include it
  • The CI_FUZZ_API_TOKEN is either a token you generated in the CI Sense UI or the password you specified if you configure CI Sense that way.
  • You can also obtain the project name by running cictl -s <fuzzing server> list projects. The project name should be of the form projects/my_project-391A177B.

CI Fuzz remote run

The CI Fuzz remote-run command bundles, uploads, and starts your fuzz tests with one command. This method doesn't require cictl to upload and start the fuzz tests, but if you want to monitor the fuzz test, you still need cictl.

cifuzz remote-run --server <server address>
  • You can authenticate to the server automatically by passing the environment variable CIFUZZ_API_TOKEN as part of your command (e.g. CIFUZZ_API_TOKEN=<token> cifuzz remote-run --server "https://cifuzz.server.com").
  • You can specify the name of one or more fuzz tests. If you don't specify any, then remote-run bundles and runs all the fuzz tests for a project.
  • You can specify various arguments telling CI Sense how to run your fuzz tests. See this page for additional details.
  • You can pass an existing bundle by simply specifying the --bundle flag and the path to the bundle. When passing an existing bundle, CI Fuzz ignores any runtime flags passed as part of the remote-run command (e.g. --timeout, --docker-image).

Run fuzz tests with CI/CD

To create a CI/CD pipeline, you need to install both CI Fuzz and the cictl command line tool. CI Sense must be reachable from the CI/CD server.

The general workflow is:

  1. Build/bundle the artifacts that run on the server
  2. Upload the artifacts to the server and run them
  3. Monitor the results

Here is a script you can use to help write and test your CI/CD pipeline. This script uses cifuzz bundle in conjunction with cictl to create the bundle, upload it, start it, and then monitor it.

#! /usr/bin/bash

# how long to monitor the fuzz test
TIMEOUT=300
# this should match wherever CI Sense is currently reachable
FUZZING_SERVER="127.0.0.1:8080"
# address of the web application
WEB_APP_ADDRESS="http://127.0.0.1:8080"
# project name of the form: projects/c-cpp-demo-0b5fbe28
# this can be obtained by using cictl -s <server> list projects
PROJECT=<project name>
# access token generated in the CI Sense web app
CI_FUZZ_API_TOKEN=
# local directory where the project is located
LOCAL_REPO=

# create bundle from local repo
# you can specify additional bundle options at the command line, or include them in cifuzz.yaml
cd $LOCAL_REPO
cifuzz bundle

CICTL_CMD="cictl -s $FUZZING_SERVER"
# login to the CI Sense server
echo $CI_FUZZ_API_TOKEN | $CICTL_CMD login
# upload the bundle to the CI Sense server
ARTIFACT_NAME=$($CICTL_CMD import artifact fuzz_tests.tar.gz --project-name "$PROJECT")
# start the fuzz tests in the bundle
CAMPAIGN_RUN=$($CICTL_CMD start --application-base-url "${WEB_APP_ADDRESS}" "${ARTIFACT_NAME}")
# monitor the output from the campaign_run
$CICTL_CMD monitor_campaign_run --dashboard_address="${WEB_APP_ADDRESS}" --keep_running --duration="${TIMEOUT}" ${CAMPAIGN_RUN}