Skip to main content Link Menu Expand (external link) Document Search Copy Copied

Running a Fuzz Test with CI App

This page describes how to run a fuzz test on CI App, both manually and as part of a CI/CD pipeline. This primarily applies to non web api fuzz tests. For details on how to run web api fuzz tests, see the relevant sections in web-api-fuzzing.

Table of contents

Overview

Running a fuzz test on CI App consists of the following steps:

  • Create a bundle, which is your software built with instrumentation the fuzzer can use to run the fuzz tests and evaluate coverage.
  • Upload the bundle to CI App.
  • Start the fuzz tests on CI App.
  • Monitor the fuzz test (only needed for CI/CD runs)

A fuzz run on CI App consists of 3 jobs:

  • Fuzz job
  • Coverage job
  • Cleanup job

The Fuzz Job

For fuzzers that support value profiling (libfuzzer and Jazzer), two fuzz jobs will be created for each fuzz test that is part of the bundle. One fuzz job will have value profiling enabled, the other will not. Value profiling can lead to improved code coverage, but will also slow the execution speed of the job. Both runs share the same corpus. Each fuzz job will execute inside of its own Docker container.

The Coverage Job

A coverage job will be started after all fuzz runs have completed. During the coverage job, the coverage binaries that were created as part of the bundle are executed using the generated corpus to determine the total coverage for the run.

Cleanup Job

The cleanup job removes coverage information that was generated during web API fuzzing.

Job Status

In the CI App UI, on the Dashboard or on a specific Project Overview, you can see the Job Status of the most recent fuzz run. You can see the status of previous fuzz runs by selecting the dropdown in the top right of the Project Overview (shown below).

The Job Status can have the following values:

  • FAILED - all jobs for the run failed to start.
  • INCOMPLETE - at least 1 job started, but not all jobs were successful.
  • IN PROGRESS - at least 1 job is still running.
  • SUCCEEDED - all jobs completed.

Manually Uploading Fuzz Tests to CI App

There are a few ways to manually upload fuzz tests to your CI App server. This is useful when you are initially setting up your CI/CD pipeline or simply want to test a new fuzz test.

Create a Bundle with CI Fuzz

If your project is already configured with CI Fuzz, you can create a bundle by running the following command from the root of the project directory:

cifuzz bundle
  • This command will build and bundle the runtime artifacts needed to run your fuzz test(s) on CI App
  • You can specify the name of one or more fuzz tests. If you don’t specify any fuzz tests, then all fuzz tests for the project will be built. If the build-system is other, then the fuzz tests must be specified explicitly.
  • You can specify various arguments that will tell CI App how to run your fuzz tests. See this page for additional details.
  • The output is a tarball of the form fuzz_tests.tar.gz or <name_of_specific_fuzz_test>.tar.gz if you specified one.

After creating the bundle file, you will need to upload it to the CI App server and start the fuzz tests. You can either:

  • Drag and drop it in the Web UI on the project card you want to run it under. You can do this from the Dashboard or from the Overview after selecting your project.
  • Use the cictl command (located in ci-app-install-directory/bin/cictl) to upload (import) the bundle to the CI App server and then start it.

Here is an example bash script that uses cictl to upload and run the fuzz tests (created using cifuzz bundle) to the CI App server.

CI_FUZZ_API_TOKEN=<token>
FUZZING_SERVER="<address of fuzzing server>"
PROJECT="<project name>"
CICTL_CMD="cictl -s $FUZZING_SERVER"
ARTIFACT_NAME=$($CICTL_CMD import artifact fuzz_tests.tar.gz --project-name "$PROJECT")
$CICTL_CMD start ${ARTIFACT_NAME}
  • Most cictl commands require the --server, -s flag to specify the location of the fuzzing server, so be sure to include it
  • The CI_FUZZ_API_TOKEN is either a token you generated in the CI App UI or the password you specified if you configure CI App that way.
  • The project name can also be obtained by running cictl -s <fuzzing server> list projects. The project name should be of the form projects/my_project-391A177B.

CI Fuzz Remote Run

The CI Fuzz remote-run command will bundle, upload, and start your fuzz tests with one command. This method does not require cictl to upload and start the fuzz tests, but if you want to monitor the fuzz test, you will still need cictl.

cifuzz remote-run --server <server address>
  • You can authenticate to the server automatically by passing the environment variable CIFUZZ_API_TOKEN as part of your command (e.g. CIFUZZ_API_TOKEN=<token> cifuzz remote-run --server "https://cifuzz.server.com").
  • You can specify the name of one or more fuzz tests. If you don’t specify any, then remote-run will bundle and run all the fuzz tests for a project.
  • You can specify various arguments that will tell CI App how to run your fuzz tests. See this page for additional details.
  • You can pass an existing bundle by simply specifying the --bundle flag and the path to the bundle. When passing an existing bundle, CI Fuzz will ignore any runtime flags passed as part of the remote-run command (e.g. --timeout, --docker-image).

Running Fuzz Tests with CI/CD

To create a CI/CD pipeline, you will need to install both CI Fuzz and the cictl command line tool. CI App must be reachable from the CI/CD server.

The general workflow is:

  1. Build/bundle the artifacts that will run on the server
  2. Upload the artifacts to the server and run them
  3. Monitor the results

Here is a script you can use to help write and test your CI/CD pipeline. This script uses cifuzz bundle in conjunction with cictl to create the bundle, upload it, start it, and then monitor it.

#! /usr/bin/bash

# how long to monitor the fuzz test
TIMEOUT=300
# this should match wherever CI App is currently reachable
FUZZING_SERVER="127.0.0.1:8080"
# address of the web application
WEB_APP_ADDRESS="http://127.0.0.1:8080"
# project name of the form: projects/c-cpp-demo-0b5fbe28
# this can be obtained by using cictl -s <server> list projects
PROJECT=<project name>
# access token generated in the CI App web app
CI_FUZZ_API_TOKEN=
# local directory where the project is located
LOCAL_REPO=

# create bundle from local repo
# you can specify additional bundle options at the command line, or include them in cifuzz.yaml
cd $LOCAL_REPO
cifuzz bundle 

CICTL_CMD="cictl -s $FUZZING_SERVER"
# login to the CI App server
echo $CI_FUZZ_API_TOKEN | $CICTL_CMD login
# upload the bundle to the CI App server
ARTIFACT_NAME=$($CICTL_CMD import artifact fuzz_tests.tar.gz --project-name "$PROJECT")
# start the fuzz tests in the bundle 
CAMPAIGN_RUN=$($CICTL_CMD start --application-base-url "${WEB_APP_ADDRESS}" "${ARTIFACT_NAME}")
# monitor the output from the campaign_run
$CICTL_CMD monitor_campaign_run --dashboard_address="${WEB_APP_ADDRESS}" --keep_running --duration="${TIMEOUT}" ${CAMPAIGN_RUN}