Skip to main content

API testing

Building robust APIs is an important and challenging endeavor that requires thorough testing. While unit tests are indispensable to ensure the functional correctness of APIs, they're not enough to address the security and reliability issues. In unit testing, you test the scenarios that you are aware of. However, there exist unknown scenarios unknown that lead to security vulnerabilities or performance problems.

In this example, you perform automated API testing for security and reliability issues with JUnit. This is enabled by the JUnit integration provided by CI Fuzz.

Demo Spring Boot app​

The demo app offers a single REST API endpoint hello that can be called a name parameter, and it replies with "Hello <name>!". The endpoint fails when the provided name is equal to attacker ignoring case. This tutorial guides you through the concepts and steps of using CI Fuzz with the JUnit 5 integration to test this app. Further, it demonstrates how you can automatically find the failing test case.

Run the app as follows

mvn spring-boot:run

You can reach the endpoint at http://localhost:8080/hello and call it via http://localhost:8080/hello?name=foo

Unit tests​

The project contains two unit tests that test the endpoint with two names Developer and Contributor. This demonstrates how you would test your code using specific inputs triggering specific behavior. You can run the unit tests

mvn test

Fuzz tests​

While unit tests provide a great value to make sure that your code is functionally correct. However, there might be corner cases and interactions that you aren't aware of that can cause security and reliability issues. To address these cases, you create a fuzz test, which is a method annotated with @FuzzTest and at least one parameter. Using a single parameter of type FuzzedDataProvider, which provides utility functions to produce commonly used Java values, or byte[]. CI Fuzz then executes with method in a loop and in each iteration provide new inputs that maximize code coverage and trigger interesting behavior in your app.

In this example, you create a fuzz test that uses the fuzzer input as the name parameter for the hello API. This way, CI Fuzz can explore the different possibilities of the parameter that trigger interesting behaviors, and thus quickly finds the prepared issue by generating the "attacker" name.

@WebMvcTest()
public class GreeterApplicationTests {
    @Autowired private MockMvc mockMvc;

    @FuzzTest
    public void fuzzTestHello(FuzzedDataProvider data) throws Exception {
        // Initialization code
        String name = data.consumeRemainingAsString();
        mockMvc.perform(get("/hello").param("name", name));
    }
}

Run your fuzz test​

  1. (Once) Install the CI Fuzz CLI named cifuzz. You can get the latest release from GitHub or by running the install script:

    sh -c "$(curl -fsSL https://raw.githubusercontent.com/CodeIntelligenceTesting/cifuzz/main/install.sh)"

    If you are using Windows you can download the latest release and execute it.

  2. Login to CI App

    cifuzz login

    This creates an API access token that cifuzz uses to communicate with the CI App. When logged in, the cifuzz can provide more details about the findings including severity. You are also able to run your tests at scale with the Code Intelligences SaaS solution.

  3. Run the fuzz test with CI Fuzz. For that you just need to provide the test class containing the fuzz test.

    > cifuzz run com.example.GreeterApplicationTests
    ▄ Build in progress... Done.

    Running com.example.GreeterApplicationTests
    Storing generated corpus in .cifuzz-corpus/com.example.GreeterApplicationTests
    Starting from an empty corpus

    Use 'cifuzz finding <finding name>' for details on a finding.

    💥 [funny_sparrow] Security Issue: We panic when trying to greet an attacker!
    in processRequest (org.springframework.web.servlet.FrameworkServlet:1014)

    Note: The reproducing inputs have been copied to the seed corpus at:

       src/test/resources/com/example/GreeterApplicationTestsInputs/funny_sparrow

    They will now be used as a seed input for all runs of the fuzz test,
    including remote runs with artifacts created via 'cifuzz bundle' and
    regression tests. For more information on regression tests, see:

        https://github.com/CodeIntelligenceTesting/cifuzz/blob/main/docs/Regression-Testing.md

    Execution time: 19s
    Average exec/s: 1682
    Findings:       1
    Corpus entries: 6 (+6)

    CI Fuzz quickly generates a test case triggering the bug (aka crashing input). This test case is saved as a resource in your project and is automatically picked up when you execute your normal unit tests. That's when you execute mvn test, you execute all your unit tests in addition to the fuzz tests. In this scenario, CI Fuzz only executes the tests with the crashing inputs and inputs from the corpus it has collected during fuzzing. This way you can ensure that you quickly test for regressions.

  4. You can check the finding details as follows

    cifuzz finding funny_sparrow

  5. You can also check the code covered by CI Fuzz

    > cifuzz coverage com.example.GreeterApplicationTests
    Building com.example.GreeterApplicationTests
    ▄  Build in progress... Done.

    ✅ Coverage Report:
                                   File | Functions Hit/Found | Lines Hit/Found | Branches Hit/Found
    com/example/GreeterApplication.java |      2 / 3  (66.7%) |  4 / 6  (66.7%) |     2 / 2 (100.0%)
                                        |                     |                 |
                                        | Functions Hit/Found | Lines Hit/Found | Branches Hit/Found
                                  Total |               2 / 3 |           4 / 6 |              2 / 2

    In addition, you also get a jacoco coverage report that you can observe in your browser. Having a look at coverage report helps understand the testing progress and observe the code areas that CI Fuzz has not yet covered. This is valuable so that you can improve and optimize your tests.

Conclusion

In this short tutorial, you used CI Fuzz to test your API. cifuzz offers many more features, which you can learn more about by running cifuzz help command.