Building a batch summarization CLI with the Claude API in Go

Processing dozens of markdown docs into concise summaries is a routine chore that a well-designed batch summarization cli claude api go tool can automate in minutes. This tutorial builds a concurrent Go binary that reads files from a directory, sends each to Anthropic’s Claude API, and writes summaries to disk.

Prerequisites

• Go 1.22 or newer installed and on PATH.
• An Anthropic API key with access to a current Claude model.
• Familiarity with basic shell commands and Go’s flag package.

Set the key before running anything:

export ANTHROPIC_API_KEY="sk-ant-..."

Create a working directory and initialize a module:

mkdir batchsum && cd batchsum
go mod init batchsum

Scaffolding the CLI

We start with flag parsing and a minimal main that prints what it would do. Keeping the surface small makes the later concurrency changes easier to reason about.

package main

import (
	"flag"
	"fmt"
	"os"
)

func main() {
	dir := flag.String("dir", "./docs", "input directory of markdown files")
	out := flag.String("out", "./summaries", "output directory for summaries")
	concurrency := flag.Int("workers", 4, "max concurrent API calls")
	flag.Parse()

	fmt.Printf("dir=%s out=%s workers=%d\n", *dir, *out, *concurrency)
	_ = os.MkdirAll(*out, 0o755)
}

Run it to confirm the flags parse:

go run . --dir ./docs --out ./summaries --workers 2
# dir=./docs out=./summaries workers=2

Discovering input files

Walk the input directory and collect .md paths. Use filepath.Glob for simplicity; it handles one level which is enough for most doc sets.

func discoverFiles(dir string) ([]string, error) {
	matches, err := filepath.Glob(filepath.Join(dir, "*.md"))
	if err != nil {
		return nil, err
	}
	if len(matches) == 0 {
		return nil, fmt.Errorf("no .md files found in %s", dir)
	}
	return matches, nil
}

Call it from main after parsing flags:

files, err := discoverFiles(*dir)
if err != nil {
	fmt.Fprintln(os.Stderr, "error:", err)
	os.Exit(1)
}
fmt.Printf("found %d files\n", len(files))

Expected checkpoint output with two sample files:

go run . --dir ./docs
# dir=./docs out=./summaries workers=4
# found 2 files

Calling Claude over HTTP

The Anthropic messages endpoint is a plain POST. We wrap it in a function that returns the text block from the first response content item. Use a shared http.Client with a timeout.

type claudeRequest struct {
	Model     string `json:"model"`
	MaxTokens int    `json:"max_tokens"`
	Messages  []struct {
		Role    string `json:"role"`
		Content string `json:"content"`
	} `json:"messages"`
}

type claudeResponse struct {
	Content []struct {
		Text string `json:"text"`
		Type string `json:"type"`
	} `json:"content"`
	Error *struct {
		Message string `json:"message"`
	} `json:"error"`
}

func summarize(client *http.Client, path string) (string, error) {
	body, err := os.ReadFile(path)
	if err != nil {
		return "", err
	}
	reqBody := claudeRequest{
		Model:     "claude-3-5-sonnet-20241022",
		MaxTokens: 1024,
	}
	reqBody.Messages = append(reqBody.Messages, struct {
		Role    string `json:"role"`
		Content string `json:"content"`
	}{Role: "user", Content: "Summarize the following document in 3 bullet points:\n\n" + string(body)})

	buf, _ := json.Marshal(reqBody)
	req, _ := http.NewRequest("POST", "https://api.anthropic.com/v1/messages", bytes.NewReader(buf))
	req.Header.Set("x-api-key", os.Getenv("ANTHROPIC_API_KEY"))
	req.Header.Set("anthropic-version", "2023-06-01")
	req.Header.Set("content-type", "application/json")

	resp, err := client.Do(req)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()

	var cr claudeResponse
	if err := json.NewDecoder(resp.Body).Decode(&cr); err != nil {
		return "", err
	}
	if cr.Error != nil {
		return "", fmt.Errorf("claude: %s", cr.Error.Message)
	}
	if len(cr.Content) == 0 {
		return "", fmt.Errorf("empty response")
	}
	return cr.Content[0].Text, nil
}

Adding a worker pool

Spawning one goroutine per file is fine for a handful of docs, but a bounded pool protects you from rate limits. Use a jobs channel and a sync.WaitGroup.

func runBatch(files []string, out string, workers int) error {
	client := &http.Client{Timeout: 30 * time.Second}
	jobs := make(chan string, len(files))
	for _, f := range files {
		jobs <- f
	}
	close(jobs)

	var wg sync.WaitGroup
	for i := 0; i < workers; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for path := range jobs {
				summary, err := summarize(client, path)
				if err != nil {
					fmt.Fprintf(os.Stderr, "fail %s: %v\n", path, err)
					continue
				}
				base := filepath.Base(path)
				outPath := filepath.Join(out, strings.TrimSuffix(base, ".md")+".summary.md")
				if err := os.WriteFile(outPath, []byte(summary), 0o644); err != nil {
					fmt.Fprintf(os.Stderr, "write fail %s: %v\n", outPath, err)
					continue
				}
				fmt.Printf("wrote %s (%d bytes)\n", outPath, len(summary))
			}
		}()
	}
	wg.Wait()
	return nil
}

Wire it into main:

if err := runBatch(files, *out, *concurrency); err != nil {
	fmt.Fprintln(os.Stderr, "batch error:", err)
	os.Exit(1)
}

Retries with backoff

Claude returns 429 when you exceed throughput. A simple retry loop with a sleep handles transient limits without extra dependencies.

func summarizeWithRetry(client *http.Client, path string) (string, error) {
	var lastErr error
	for attempt := 0; attempt < 3; attempt++ {
		s, err := summarize(client, path)
		if err == nil {
			return s, nil
		}
		lastErr = err
		time.Sleep(time.Duration(attempt+1) * 500 * time.Millisecond)
	}
	return "", lastErr
}

Swap summarize for summarizeWithRetry inside the worker loop.

Full run and expected output

Place two files in ./docs: alpha.md and beta.md. Run the built binary:

go build -o batchsum .
./batchsum --dir ./docs --out ./summaries --workers 2

Sample terminal session:

dir=./docs out=./summaries workers=2
found 2 files
wrote ./summaries/alpha.summary.md (312 bytes)
wrote ./summaries/beta.summary.md (287 bytes)

The summaries directory now contains processed markdown. Each file holds three bullet points extracted from the source.

Structured output option

If downstream code consumes the summaries, ask Claude for JSON and parse it. Change the prompt to request a schema and unmarshal into a struct. This avoids fragile string splitting.

type summary struct {
	Points []string `json:"points"`
}
// prompt: "Return JSON: {\"points\": [string, string, string]}"

Beyond a single provider

When you run this in production, provider degradation is a question of when not if. If you want to avoid vendor lock-in, you can point the same HTTP client at an OpenAI-compatible gateway such as n4n.ai, which exposes one endpoint for 240+ models and automatically falls back when a provider is rate-limited or degraded. The request shape stays identical if you map Claude’s message format correctly.

The batch summarization cli claude api go we built handles discovery, concurrency, retries, and output writing in under 200 lines. Extend it with glob patterns, JSON output, or a resume mode that skips existing summary files.