Skip to main content

LLM Setup

Supported LLMs​

The AI Test Agent works with some of the most popular large language models such as GPT-4o or Claude 3.5 Sonnet without any modifications to the model. You can use one of the following ways to grant the AI Test Agent access to these models:

OpenAI​

You can use GPT-4o directly via OpenAI. Please follow the documentation on how to obtain an API key. Set the environment variable CIFUZZ_LLM_API_TOKEN to the value of the API key.

OpenAI on Azure​

You can use GPT-4o hosted on Azure. For more information on how to deploy a model please visit the documentation for AI Services. Once deployed, please set the following environment variables:

CIFUZZ_LLM_API_TYPE=AZURE
CIFUZZ_LLM_API_URL=https://XXX.openai.azure.com
CIFUZZ_LLM_API_TOKEN=<YOUR KEY>
CIFUZZ_LLM_AZURE_DEPLOYMENT_NAME=<YOUR DEPLOYMENT NAME>

In order to obtain these values, please open the deployment in Azure AI Foundry:

Please ensure that the value for CIFUZZ_LLM_API_URL only includes the base URL up until .openai.azure.com

Azure setup

Anthropic on AWS​

You can use Claude 3.5 Sonnet hosted on AWS Bedrock. For more information on how to obtain access to the Claude 3.5 Sonnet foundation model, please visit the documentation for AWS Bedrock. Once made accessible, please set the following environment variables:

CIFUZZ_LLM_API_TYPE=BEDROCK
CIFUZZ_LLM_MODEL=anthropic.claude-3-5-sonnet-20240620-v1:0
CIFUZZ_LLM_BEDROCK_MODEL_VERSION=bedrock-2023-05-31
Make sure to authenticate to AWS, for example via aws configure or by setting the AWS_ACCESS_KEY_ID and

AWS_SECRET_ACCESS_KEY environment variables.

General configuration options​

The interaction with the LLM can be further configured by the following options:

  • CIFUZZ_LLM_TEMPERATURE - Temperature setting for chat completion.
  • CIFUZZ_LLM_MAX_TOKENS - Maximum number of tokens for a single chat completion request.
  • CIFUZZ_LLM_API_HEADER_some_header - Additional headers to add to HTTP requests. Multiple possible. "_" in some-header is replaced by "-" in the header.
  • CIFUZZ_LLM_MAX_ALTERNATIVES - Maximum number of alternatives requested in a single LLM request. Pick a low number to reduce the number of output tokens.