Spring AI: connecting to OpenAI-compatible APIs

Spring AI gives you a portable abstraction over chat models, but the fastest path to a working app is pointing it at a spring ai openai compatible api. This tutorial builds a Spring Boot service that talks to any OpenAI-compatible endpoint, streams responses, and degrades cleanly when a provider fails.

Prerequisites

• JDK 17 or newer installed and on PATH
• Maven 3.9+ (Gradle 8+ works too, but snippets use Maven)
• Spring Boot 3.3.x
• An API key and base URL for an OpenAI-compatible service (OpenAI, a local LLM server, or a gateway)
• Export your key before starting: export OPENAI_API_KEY=sk-...

You should be comfortable with @RestController, @RequestParam, and basic Spring dependency injection.

1. Bootstrap the Spring Boot project

Generate a minimal web app from the command line:

curl -s https://start.spring.io/starter.zip \
  -d type=maven-project \
  -d javaVersion=17 \
  -d bootVersion=3.3.0 \
  -d dependencies=web \
  -d packageName=com.example.aidemo \
  -d artifactId=ai-demo \
  -o ai-demo.zip
unzip ai-demo.zip && cd ai-demo

The generated pom.xml has the web starter only. We add the AI starter manually to pin the milestone version.

2. Add the Spring AI OpenAI starter

Spring AI publishes to a milestone repository because it is pre-GA. Append this to pom.xml:

<repositories>
  <repository>
    <id>spring-milestones</id>
    <name>Spring Milestones</name>
    <url>https://repo.spring.io/milestone</url>
  </repository>
</repositories>

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
    <version>1.0.0-M6</version>
  </dependency>
</dependencies>

The starter auto-configures an OpenAiChatModel and a ChatClient.Builder as soon as spring.ai.openai.api-key is set. No manual @Bean is required for the basic case.

3. Configure the OpenAI-compatible endpoint

In src/main/resources/application.yml, set the base URL, key, and default model:

spring:
  ai:
    openai:
      base-url: https://api.openai.com/v1
      api-key: ${OPENAI_API_KEY}
      chat:
        options:
          model: gpt-3.5-turbo
          temperature: 0.7

The base-url must point at the OpenAI-style /v1 path. To target a different spring ai openai compatible api, change base-url and model. For a local Ollama OpenAI shim, use http://localhost:11434/v1 and model: llama3. For Azure OpenAI, the path includes a deployment segment, so consult the provider docs.

4. Build a chat service with ChatClient

Create a controller that injects the auto-configured ChatClient.Builder. The builder is thread-safe; build one ChatClient per controller and reuse it.

package com.example.aidemo;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.web.bind.annotation.*;
import reactor.core.publisher.Flux;
import org.springframework.http.MediaType;

@RestController
@RequestMapping("/chat")
public class ChatController {

    private final ChatClient chatClient;

    public ChatController(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    @GetMapping("/ask")
    public String ask(@RequestParam String q) {
        return chatClient.prompt()
                .user(q)
                .call()
                .content();
    }

    @GetMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<String> stream(@RequestParam String q) {
        return chatClient.prompt()
                .user(q)
                .stream()
                .content();
    }
}

Non-streaming call

chatClient.prompt().user(q).call().content() blocks the request thread until the model returns the full text. Use it for short, synchronous tasks or internal services where latency is acceptable.

Streaming response

.stream().content() returns a Reactor Flux<String> that emits tokens as they are produced. Spring Boot serializes the Flux as Server-Sent Events with data: prefixes automatically. The client connection stays open until the model finishes or times out.

5. Run and verify

Start the app:

mvn spring-boot:run

Watch the log for Started AiDemoApplication. If the OpenAI starter fails to configure, it will complain about missing api-key at startup.

Expected output

Non-streaming:

$ curl "http://localhost:8080/chat/ask?q=What%20is%20Spring%20AI?"
Spring AI is a Spring ecosystem project that provides a unified API for chat,
text-to-image, and embedding models, letting you swap providers with config changes.

Streaming with -N to disable curl buffering:

$ curl -N "http://localhost:8080/chat/stream?q=Count%20to%203%20with%20a%20space"
data:1
data: 2
data: 3

The exact tokens depend on the model, but the SSE framing confirms the stream works. Add -i to see Content-Type: text/event-stream.

6. Customize model and parameters per request

Global YAML sets defaults; override per call with OpenAiChatOptions. You can also attach a system prompt:

import org.springframework.ai.openai.OpenAiChatOptions;

@GetMapping("/gpt4")
public String gpt4(@RequestParam String q) {
    return chatClient.prompt()
            .system("You are a concise senior engineer.")
            .user(q)
            .options(OpenAiChatOptions.builder()
                    .model("gpt-4o")
                    .temperature(0.2)
                    .maxTokens(256)
                    .build())
            .call()
            .content();
}

This keeps the same ChatClient while targeting a different model or sampling setting for one request. The system() call sets the system message; omit it to use the model default.

7. Production hardening: timeouts and fallback

A single LLM provider will rate-limit or stall. Set HTTP timeouts on the underlying client. For the OpenAI starter, Spring Boot’s spring.http.client applies:

spring:
  http:
    client:
      connect-timeout: 5s
      read-timeout: 30s

If you need fine control, define a RestClient.Builder bean and reference it via spring.ai.openai.rest-client-builder (check the exact property for your milestone).

Swap base-url to a gateway

If you want a single spring ai openai compatible api endpoint that addresses 240+ models with automatic fallback when a provider is degraded and per-token usage metering, point the base-url at a gateway like n4n.ai. The client code stays identical; you only change YAML or environment variables.

spring:
  ai:
    openai:
      base-url: https://api.n4n.ai/v1
      api-key: ${N4N_KEY}

The gateway honors client routing directives and forwards provider cache-control hints, so existing OpenAiChatOptions still apply.

Handle provider errors

Wrap calls to return a clean 503 instead of a stack trace:

@GetMapping("/askSafe")
public ResponseEntity<String> askSafe(@RequestParam String q) {
    try {
        return ResponseEntity.ok(chatClient.prompt().user(q).call().content());
    } catch (Exception e) {
        return ResponseEntity.status(503)
                .body("LLM provider unavailable: " + e.getMessage());
    }
}

Add Resilience4j @Retry or a bulkhead if you need automatic retries with backoff. Do not retry indefinitely; LLM generation is expensive.

8. Where to go next

You now have a runnable Spring Boot service that speaks the OpenAI wire format. Extend it with ChatMemory for multi-turn sessions, or use PromptTemplate to structure system messages from external files. The abstraction means swapping spring-ai-openai for spring-ai-anthropic later requires no controller changes—only dependency and property swaps.

Keep the ChatClient instance as a singleton; building a new one per request adds needless overhead. Monitor token usage via ChatResponse.getMetadata().getUsage() if you need per-call accounting. For browser clients, add a CorsConfiguration bean to allow your origin on /chat/**.

That is the core integration. Everything else is standard Spring.