Write a Java Fuzz Test

Structure of a Fuzz Test

MyFuzzTestCase.java

import com.code_intelligence.jazzer.junit.FuzzTest;

public class MyFuzzTest {
    @FuzzTest
    void myFuzzTest(byte[] data) {
        // Add necessary parameters and call target function with them
    }
}

A Fuzz Test is annotated with @FuzzTest, similar to JUnit's @Test annotation, but can expect/request Fuzzer-generated parameters. The @FuzzTest annotation provides parameters to set the maximum time duration the Fuzz Test should be executed or the maximum number of invocations it should be invoked.

Parameters

Java Fuzz Tests are executed with Jazzer, a coverage-guided, in-process Fuzzer for the JVM platform. Jazzer can work with any number of parameters of primitive and limited object types and can mutate these in a type-specific manner to generate valid input.

The example below shows how to use complex data types in a Fuzz Test. You can use any supported type as a parameter.

Example

record SimpleTypesRecord(boolean bar, int baz) {}

@FuzzTest
public void testSimpleTypeRecord(SimpleTypesRecord record) {
    doSomethingWithRecord(record);
}

Supported types

Mutator	Type	Notes
Boolean	boolean, Boolean
Integral	byte, Byte, short, Short, int, Int, long, Long
Floating point	float, Float, double, Double
String	java.lang.String
Enum	java.lang.Enum
InputStream	java.io.InputStream
Time	java.time.LocalDate, java.time.LocalDateTime, java.time.LocalTime, java.time.ZonedDateTime
Array	Arrays holding any other supported type (for example byte[], Integer[], Map[], String[], etc.)
List	java.util.List
Map	java.util.Map
Record	java.lang.Record	Arbitrary Java Records, if supported by JVM version
Setter-based JavaBean		Any class adhering to the JavaBeans Spec, see JavaBeans Support for details
Constructor-based JavaBean		Any class adhering to the JavaBeans Spec, see JavaBeans Support for details
Constructor-based Java Classes		Any class requiring constructor parameters, but not offering getter methods, see constructor-based classes for details
Builder		See Builder pattern support for details
FuzzedDataProvider	com.code_intelligence.jazzer.api.FuzzedDataProvider
Protobuf	com.google.protobuf.Message, com.google.protobuf.Message.Builder, com.google.protobuf.ByteString	Classes generated by the Protobuf toolchain
Nullable		Any reference type will occasionally be set to null

Annotations

It can be helpful to provide additional information about the Fuzz Test parameters, for example to specify the range of integers, or the maximum length of a string. This is done using annotations directly on the parameters.

Note: Annotations are used on best effort basis, meaning that the Fuzzer tries to honor specified constraints, but can not guarantee it.

All annotations reside in the com.code_intelligence.jazzer.mutation.annotation package.

Annotation	Applies To	Notes
@Ascii	java.lang.String	String should only contain ASCII characters
@InRange	byte, Byte, short, Short, int, Int, long, Long	Specifies min and max values of generated integrals
@InRangeFloat	float, Float	Specifies min and max values of generated floats
@InRangeDouble	double, Double	Specifies min and max values of generated doubles
@NotNull		Specifies that a reference type shouldn't be null
@WithLength	byte[]	Specifies the length of the generated byte array
@WithUtf8Length	java.lang.String	Specifies the length of the generated string in UTF-8 bytes, see annotation Javadoc for further information
@WithSize	java.util.List, java.util.Map	Specifies the size of the generated collection
@UrlSegment	java.lang.String	String should only contain valid URL segment characters

The example below shows how you can annotate Fuzz Test parameters to provide additional information to the mutation framework.

Example

record SimpleTypesRecord(boolean bar, int baz) {}

@FuzzTest
public void testSimpleTypeRecord(@NotNull @WithSize(min = 3, max = 100) List<SimpleTypesRecord> records) {
    doSomethingWithRecord(record);
}

Annotation constraints

Annotations should be applied to a type and all it's nested component types. This use-case is supported by the annotation's constraint property. Setting it to PropertyConstraint.RECURSIVE propagates the annotation down to all subcomponent types. All above-mentioned annotations support this feature.

For example, if a Fuzz Test expects a List of List of Integer as parameter, and both the lists and their values must not be null, you can add the annotation @NotNull(constraint = PropertyConstraint.RECURSIVE) on the root type.

Example

@FuzzTest
public void fuzz(@NotNull(constraint = PropertyConstraint.RECURSIVE) List<List<Integer>> list) {
    // list is not null and does not contain null entries on any level
    assertDeepNotNull(list);
}

JavaBeans support

Jazzer can generate and mutate instances of classes adhering to the JavaBeans Spec.

To serialize and deserialize Java objects to and from corpus entries, Jazzer can use setters, constructors and getters to pass values to a JavaBean and extract them back out from it.

Setter-based approach

The setter-based approach requires a class to provide a default constructor with no arguments. The corresponding methods are looked up by name and must adhere to the JavaBeans Spec naming convention, meaning setXX and getXX/isXX methods for property XX. A JavaBean can have additional getters corresponding to computed properties, but it's required that all setters have a corresponding getter.

Example

public static class FooBean {
    private String foo;

    public String getFoo() {
        return foo;
    }

    public void setFoo(String foo) {
        this.foo = foo;
    }
}

@FuzzTest
public void testFooBean(FooBean fooBean) {
    // ...
}

Constructor-based approach

The constructor-based approach requires a class to provide a constructor with arguments. If multiple constructors are available, the one with the most supported parameters will be preferred.

The lookup of matching getters relies on the Java bean's property names. As a class can have further properties or internal states, this approach relies on the constructor parameter names. Since parameter names are not always available at runtime, you have to explicitly compile them into the class file with the use of the JavaBeans @ConstructorProperties annotation, to specify property names explicitly.

Example

public static class PropertyNamesBean {
    private final String bar;

    public PropertyNamesBean(String bar) {
        this.bar = bar;
    }

    public String getBar() {
        return bar;
    }
}

public static class ConstructorPropertiesBean {
    private final String foo;

    @ConstructorProperties({"bar"})
    public PropertyNamesBean(String foo) {
        this.bar = foo;
    }

    public String getBar() {
        return foo;
    }
}

public static class FallbackTypeBean {
    private final String foo;

    public PropertyNamesBean(String foo) {
        this.bar = foo;
    }

    public String getSomething() {
        return foo;
    }
}

@FuzzTest
public void testBeans(PropertyNamesBean propertyNamesBean, ConstructorPropertiesBean constructorPropertiesBean, FallbackTypeBean fallbackTypeBean) {
    // ...
}

Constructor-based classes

Jazzer can generate and mutate instances of classes that build up their internal state via constructor parameters, and, in contrast to JavaBeans, don't offer getter methods.

The following class would fall into this category:

Constructor-based class

class ImmutableClassTest {

    static class ImmutableClass {
        private final int bar;
        public ImmutableClass(int foo) {
            this.bar = foo * 2;
        }
        String barAsString() {
            return String.valueOf(bar);
        }
    }

    @FuzzTest
    void fuzzImmutableClassFunction(ImmutableClass immutableClass) {
        if (immutableClass != null && "42".equals(immutableClass.barAsString())) {
            throw new RuntimeException("42!");
        }
    }
}

Builder pattern support

The builder pattern is a common design pattern to simplify the construction of complex objects.

• A common implementation gathers all required parameters in the builder and passes them to the constructor of the target class.
• Another approach is used for builders supporting a nested type hierarchy in the target class. In this situation the builder itself is passed into the constructor of the target class.

Note: These pattern are generated by the commonly used Lombok @Builder and @SuperBuilder annotations.

The examples below use Lombok to generate appropriate builder classes:

@Builder pattern support

class SimpleClassFuzzTests {

    @Builder
    static class SimpleClass {
        String foo;
        List<Integer> bar;
        boolean baz;
    }

    @FuzzTest
    void fuzzSimpleClassFunction(@NotNull SimpleClass simpleClass) {
        someFunctionToFuzz(simpleClass);
    }
}

@SuperBuilder pattern support

class SimpleClassFuzzTests {

    @SuperBuilder
    static class ParentClass {
        String foo;
    }

    @SuperBuilder
    static class ChildClass extends ParentClass {
        List<Integer> bar;
    }

    @FuzzTest
    void fuzzChildClassFunction(@NotNull ChildClass childClass) {
        someChildFunctionToFuzz(childClass);
    }
}

FuzzedDataProvider

The FuzzedDataProvider is an alternative approach commonly used in programming languages like C and C++. It provides an intuitive interface to deconstruct Fuzzer input with type-specific functions, for example consumeString, consumeBoolean or consumeInt. Jazzer's Java implementation follows the FuzzedDataProvider of the LLVM Project.

This programmatic approach offers very fine-grained control, but requires much more effort to build up needed data structures.

Below is an example of a simple Fuzz Test using the FuzzedDataProvider:

Example

import com.code_intelligence.jazzer.api.FuzzedDataProvider;
import com.code_intelligence.jazzer.junit.FuzzTest;

class ParserTests {
   @Test
   void unitTest() {
      assertEquals("foobar", SomeScheme.decode(SomeScheme.encode("foobar")));
   }

   @FuzzTest
   void fuzzTest(FuzzedDataProvider data) {
      String input = data.consumeRemainingAsString();
      assertEquals(input, SomeScheme.decode(SomeScheme.encode(input)));
   }
}

Create a Fuzz Test template

You can create a Fuzz Test template in a specified location with following command:

cifuzz create java -o <path to fuzz test>

Create Fuzz Tests close to the code being tested, just as you would for a unit test.