Write a Java Fuzz Test
Structure of a Fuzz Test
import com.code_intelligence.jazzer.junit.FuzzTest;
public class MyFuzzTest {
@FuzzTest
void myFuzzTest() {
// Add necessary parameters and call target function with them
}
}
A Fuzz Test is annotated with @FuzzTest
, similar to JUnit's @Test
annotation. The class needs to import the Jazzer
package to resolve the annotation.
Parameters
Java Fuzz Tests are executed with Jazzer, our coverage-guided, in-process fuzzer for the JVM platform. Jazzer can work with any number of parameters of primitive and limited object types and can mutate these in a type specific manner to generate valid input.
The example below shows how to use complex data types in a Fuzz Test. Any supported type can be used as a parameter.
record SimpleTypesRecord(boolean bar, int baz) {}
@FuzzTest
public void testSimpleTypeRecord(SimpleTypesRecord record) {
doSomethingWithRecord(record);
}
Supported types
Mutator | Type(s) | Notes |
---|---|---|
Boolean | boolean , Boolean | |
Integral | byte , Byte , short , Short , int , Int , long , Long | |
Floating point | float , Float , double , Double | |
String | java.lang.String | |
Enum | java.lang.Enum | |
InputStream | java.io.InputStream | |
Time | java.time.LocalDate , java.time.LocalDateTime , java.time.LocalTime , java.time.ZonedDateTime | |
Array | Arrays holding any other supported type (e.g. byte[] , Integer[] , Map[] , String[] , etc.) | |
List | java.util.List | |
Map | java.util.Map | |
Record | java.lang.Record | Arbitrary Java Records, if supported by JVM version |
Setter-based JavaBean | Any class adhering to the JavaBeans Spec, see [JavaBeans Mutator](#JavaBeans support) for details | |
Constructor-based JavaBean | Any class adhering to the JavaBeans Spec, see [JavaBeans Mutator](#JavaBeans support) for details | |
FuzzedDataProvider | com.code_intelligence.jazzer.api.FuzzedDataProvider | |
Protobuf | com.google.protobuf.Message , com.google.protobuf.Message.Builder , com.google.protobuf.ByteString | Classes generated by the Protobuf toolchain |
Nullable | Any reference type will occasionally be set to null |
Annotations
It is sometimes helpful to provide additional information about the Fuzz Test parameters, e.g. to specify the range of integers, or the maximum length of a string. This is done using annotations directly on the parameters.
Note: Annotations are used on best effort basis, meaning that the fuzzer will try to honor specified constraints, but can not guarantee it.
Annotation | Applies To | Notes |
---|---|---|
@Ascii | java.lang.String | String should only contain ASCII characters |
@InRange | byte , Byte , short , Short , int , Int , long , Long | Specifies min and max values of generated integrals |
@InRangeFloat | float , Float | Specifies min and max values of generated floats |
@InRangeDouble | double , Double | Specifies min and max values of generated doubles |
@NotNull | Specifies that a reference type should not be null | |
@WithLength | byte[] | Specifies the length of the generated byte array |
@WithUtf8Length | java.lang.String | Specifies the length of the generated string in UTF-8 bytes, see annotation Javadoc for further information |
@WithSize | java.util.List , java.util.Map | Specifies the size of the generated collection |
@UrlSegment | java.lang.String | String should only contain valid URL segment characters |
The example below shows how Fuzz Test parameters can be annotated to provide additional information to the mutation framework.
record SimpleTypesRecord(boolean bar, int baz) {}
@FuzzTest
public void testSimpleTypeRecord(@NotNull @WithSize(min = 3, max = 100) List<SimpleTypesRecord> records) {
doSomethingWithRecord(record);
}
Annotation constraints
Often, annotations should be applied to a type and all it's nested component types. This use-case is supported by the
annotation's constraint
property. It can be set to PropertyConstraint.RECURSIVE
so that the annotation is propagated
down to all subcomponent types.
All above-mentioned annotations support this feature.
For example, if a Fuzz Test expects a List
of List
of Integer
as parameter, and both the lists and their values
must not be null
, the annotation @NotNull(constraint = PropertyConstraint.RECURSIVE)
could be added on the root type.
@FuzzTest
public void fuzz(@NotNull(constraint = PropertyConstraint.RECURSIVE) List<List<Integer>> list) {
// list is not null and does not contain null entries on any level
assertDeepNotNull(list);
}
JavaBeans support
Jazzer can generate and mutate instances of classes adhering to the JavaBeans Spec.
To serialize and deserialize Java objects to and from corpus entries, Jazzer can use setters, constructors and getters to pass values to a JavaBean and extract them back out from it.
Setter-based approach
The setter-based approach requires a class to provide a default constructor with no arguments.
The corresponding methods are looked up by name and must adhere to the JavaBeans Spec naming convention, meaning setXX
and getXX
/isXX
methods for property XX
. A JavaBean can have additional getters corresponding to computed
properties, but it is required that all setters have a corresponding getter.
public static class FooBean {
private String foo;
public String getFoo() {
return foo;
}
public void setFoo(String foo) {
this.foo = foo;
}
}
@FuzzTest
public void testFooBean(FooBean fooBean) {
// ...
}
Constructor-based approach
The constructor-based approach requires a class to provide a constructor with arguments. If multiple constructors are available, the one with the most supported parameters will be preferred.
The lookup of matching getters relies on the Java bean's property names. As a class can have further properties or
internal states, this approach relies on the constructor parameter names. Since parameter names are not always available
at runtime, they explicitly have to be compiled into the class file with the use of the JavaBeans @ConstructorProperties
annotation, to specify property names explicitly.
public static class PropertyNamesBean {
private final String bar;
public PropertyNamesBean(String bar) {
this.bar = bar;
}
public String getBar() {
return bar;
}
}
public static class ConstructorPropertiesBean {
private final String foo;
@ConstructorProperties({"bar"})
public PropertyNamesBean(String foo) {
this.bar = foo;
}
public String getBar() {
return foo;
}
}
public static class FallbackTypeBean {
private final String foo;
public PropertyNamesBean(String foo) {
this.bar = foo;
}
public String getSomething() {
return foo;
}
}
@FuzzTest
public void testBeans(PropertyNamesBean propertyNamesBean, ConstructorPropertiesBean constructorPropertiesBean, FallbackTypeBean fallbackTypeBean) {
// ...
}
FuzzedDataProvider
The FuzzedDataProvider
is an alternative approach to using primitive and complex types as parameters and is used to
split the fuzzing input in the data
variable into different data types.
Our Java implementation included in Jazzer is based on the FuzzedDataProvider
of the LLVM Project.
Below is a example of a simple property-based Fuzz Test using the FuzzedDataProvider
:
class ParserTests {
@Test
void unitTest() {
assertEquals("foobar", SomeScheme.decode(SomeScheme.encode("foobar")));
}
@FuzzTest
void fuzzTest(FuzzedDataProvider data) {
String input = data.consumeRemainingAsString();
assertEquals(input, SomeScheme.decode(SomeScheme.encode(input)));
}
}
Create a Fuzz Test template
You can run the following cifuzz
command to create a Fuzz Test template in a specified location:
cifuzz create java -o <path to fuzz test>
We recommend creating your Fuzz Tests close to the code being tested, just as you would for a unit test.