Batch processing files through an LLM API with a shell loop

The simplest way to apply an LLM to a folder of documents is a bash batch processing llm api loop that reads each file, sends its contents to an OpenAI-compatible endpoint, and writes the response. That approach works for a quick script, but shipping it means handling partial failures, rate limits, and corrupted outputs without losing work.

Step 1: Set up credentials and endpoint

Start by isolating secrets and configuration in environment variables. Any OpenAI-compatible /v1 endpoint works—your own proxy, a cloud provider, or a gateway.

export LLM_API_KEY="sk-..."
export LLM_BASE_URL="https://api.openai.com/v1"
export MODEL="gpt-4o-mini"

Verify connectivity and the model name before looping over files. A bad model string fails every iteration silently if you don’t check upfront.

curl -s "$LLM_BASE_URL/models" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  | jq -r '.data[].id' | grep -x "$MODEL"

If that returns your model, you’re ready. Keep LLM_BASE_URL pointed at a single base; if you later switch providers, only this variable changes.

Step 2: Prepare input files and prompt assembly

Assume a directory ./docs full of .txt files. You need to inject file contents into a JSON payload without breaking escaping. Never use shell string interpolation for JSON—use jq.

build_payload() {
  local file="$1"
  local instruction="Summarize the following document in 3 bullet points."
  local content
  content=$(jq -Rs '.' < "$file")   # raw string -> JSON string
  jq -n --arg sys "$instruction" --arg user "$content" '{
    model: env.MODEL,
    messages: [
      {role: "system", content: $sys},
      {role: "user", content: $user}
    ],
    temperature: 0.2
  }'
}

jq -Rs '.' reads the file as raw text and emits a properly escaped JSON string. This avoids the classic bug where a stray double quote or newline truncates the request body.

Why jq is non-negotiable

Bash has no native JSON parser. Hand-rolled sed escaping breaks on tabs, backslashes, or Unicode. jq is a single static binary, available in every CI image I’ve used, and it makes the bash batch processing llm api loop survivable.

Step 3: Write the core loop

The naive version runs serially and writes one output per input:

mkdir -p ./out

for f in ./docs/*.txt; do
  base=$(basename "$f" .txt)
  payload=$(build_payload "$f")
  resp=$(curl -s "$LLM_BASE_URL/chat/completions" \
    -H "Authorization: Bearer $LLM_API_KEY" \
    -H "Content-Type: application/json" \
    -d "$payload")
  echo "$resp" | jq -r '.choices[0].message.content' > "./out/$base.summary.md"
done

This works for ten files on a quiet afternoon. It falls over at scale because any HTTP 429, 500, or malformed response writes an empty or error HTML page into your output file, and you won’t notice until you read it.

Step 4: Add retries and backoff

Wrap the network call in a function that inspects the HTTP status code and retries with exponential backoff. This is the difference between a toy and a batch job.

call_llm() {
  local payload="$1"
  local attempt=0
  local max_attempts=5
  while [ $attempt -lt $max_attempts ]; do
    local http_code
    http_code=$(curl -s -o /tmp/resp.json -w "%{http_code}" \
      "$LLM_BASE_URL/chat/completions" \
      -H "Authorization: Bearer $LLM_API_KEY" \
      -H "Content-Type: application/json" \
      -d "$payload")
    if [ "$http_code" = "200" ]; then
      cat /tmp/resp.json
      return 0
    fi
    attempt=$((attempt+1))
    local backoff=$((2**attempt))
    echo "Attempt $attempt failed ($http_code), sleep $backoff" >&2
    sleep "$backoff"
  done
  return 1
}

If you route through a gateway such as n4n.ai, automatic fallback when a provider is rate-limited or degraded lets you keep this loop simple—failed provider routes are retried upstream without custom logic in your shell script.

Step 5: Process responses and validate

Now compose the loop with error handling and output validation. Skip files that already have output to make the run resumable.

process_file() {
  local f="$1"
  local base
  base=$(basename "$f" .txt)
  [ -s "./out/$base.summary.md" ] && return 0   # idempotent

  local payload
  payload=$(build_payload "$f")
  local resp
  if ! resp=$(call_llm "$payload"); then
    echo "FAILED: $f" >> batch.errors.log
    return 1
  fi

  local summary
  summary=$(echo "$resp" | jq -r '.choices[0].message.content // empty')
  if [ -z "$summary" ]; then
    echo "EMPTY: $f" >> batch.errors.log
    return 1
  fi
  printf '%s\n' "$summary" > "./out/$base.summary.md"
}

The // empty operator in jq turns null or missing fields into an empty string, so you catch truncated responses instead of writing the word null.

Logging and resumability

Write failures to batch.errors.log. On the next run, the -s check skips completed files. For a 10k-file corpus, this turns a crashed job into a five-minute resume instead of a restart.

Step 6: Parallelize safely

A serial bash batch processing llm api loop leaves most of your rate limit budget on the table. Use xargs -P to run N concurrent processes, but cap it below your provider’s limit.

export -f process_file build_payload call_llm
ls ./docs/*.txt | xargs -P 4 -I {} bash -c 'process_file "$@"' _ {}

Export the functions so the subshells inherit them. -P 4 runs four files at once. If you get 429s, drop to -P 2 or raise backoff. Do not use GNU parallel unless you’ve already confirmed your endpoint tolerates the burst—most per-token metering systems will happily bill you for the retry storm.

Step 7: Verify success

Verification is not optional. After the run, check counts and spot-check content.

echo "Input:  $(ls ./docs/*.txt | wc -l)"
echo "Output: $(ls ./out/*.summary.md | wc -l)"
echo "Errors: $(wc -l < batch.errors.log 2>/dev/null || echo 0)"

A healthy batch ends with Output == Input and Errors == 0. For the residuals, open batch.errors.log and re-run only those paths:

grep '^FAILED:' batch.errors.log | cut -d' ' -f2- | while read -r f; do
  process_file "$f"
done

If your endpoint returns usage data, tally cost per file. Gateways that emit per-token usage metering (like n4n.ai) let you extract it directly from the response JSON:

echo "$resp" | jq '.usage // empty'

Append that to a usage.jsonl inside process_file to build a per-file cost report.

Manual quality gate

Automated checks confirm shape, not sense. Pull three random outputs and read them against the source:

ls ./out/*.summary.md | shuf | head -3 | while read -r o; do
  echo "=== $o ==="; cat "$o"
done

If the summaries are garbage, your prompt—not the loop—is the problem. Fix build_payload and re-run; the idempotency guard prevents recomputing good files.

Closing notes on the pattern

The bash batch processing llm api loop is deliberately boring. Bash gives you curl, jq, and xargs—enough to drive an LLM over a filesystem without a Python service or orchestrator. The moment you need dependent calls, structured extraction with schemas, or cross-file state, move the inner logic to a compiled binary and keep the shell as the scheduler. Until then, the script above is production-grade if you respect rate limits and verify outputs.