GTE-large vs BGE-large: embedding throughput compared

When you need to embed millions of documents, the choice between GTE-large and BGE-large often comes down to throughput and integration friction. This head-to-head on GTE-large vs BGE-large throughput breaks down the real differences engineers hit in production, not just MTEB leaderboard scores.

Capabilities and embedding quality

Both models are BERT-large sized encoders with 1024-dimensional outputs and 24 transformer layers. GTE-large (from Alibaba) and BGE-large-en (from BAAI) are trained with contrastive objectives on large text corpora, and both handle English strongly with multilingual variants available for other languages.

On public benchmarks like MTEB, BGE-large-en has historically led English retrieval tasks, while GTE-large sits within a narrow margin on most tasks. For semantic search over typical English text, either will produce embeddings that cluster sensibly. If you need Chinese, GTE-large’s multilingual checkpoint and BGE-large-zh are both solid; BGE-zh often edges out on Chinese retrieval due to specialized training data.

Neither model supports long documents natively: max input is 512 wordpiece tokens. If your chunks exceed that, you must truncate or implement a sliding-window pooling strategy. Both output dense vectors that work with cosine similarity after L2 normalization.

Throughput and latency characteristics

The core of GTE-large vs BGE-large throughput is that both share the same transformer backbone. On identical hardware (e.g., an A10G or T4), raw token throughput is nearly identical because the matrix multiplies are the same shape. Differences come from tokenizer speed, pooling implementation, and the serving stack you wrap around the model.

Batch sizing and GPU utilization

BERT-large fits roughly 20–30 sequences of 512 tokens per batch on a 24GB GPU before memory bounds hit. With mixed precision (fp16), both models sustain similar sequences-per-second. Increasing batch size improves GPU utilization linearly until compute-bound.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-large-en")
# vs "Alibaba-NLP/gte-large"
embeddings = model.encode(
    texts,
    batch_size=64,
    convert_to_numpy=True,
    normalize_embeddings=True
)

The encode call hides pooling and normalization. BGE-large-en applies a pooling layer that averages token embeddings; GTE-large uses similar CLS or mean pooling. The overhead is negligible relative to the transformer forward pass.

Quantization and mixed precision

Both models load in fp16 or int8 without architecture changes. Int8 via Hugging Face transformers with load_in_8bit=True cuts memory roughly in half and can increase throughput on memory-bound setups, though some accuracy drop appears on retrieval recall. For GTE-large vs BGE-large throughput, quantization gains are symmetric—you are not penalized for picking one over the other.

from transformers import AutoModel
model = AutoModel.from_pretrained(
    "BAAI/bge-large-en",
    device_map="auto",
    load_in_8bit=True
)

Concurrent requests

When served behind a web server, both models benefit from dynamic batching. A single worker with a batch window of 10ms can absorb bursts. Because the compute profiles match, the saturation point (requests/sec at p99 latency < 50ms) is the same for equal batch sizes.

Cost model

Both are open-weight models. There is no per-call API fee if you self-host. Cost is driven by GPU hours.

• GTE-large: Apache 2.0 license, free to deploy commercially.
• BGE-large-en: MIT license, free to deploy commercially.

If you use a managed endpoint, you pay for throughput. For example, an OpenAI-compatible gateway like n4n.ai exposes both behind one endpoint and meters per token, so cost scales with embedded volume rather than fixed instance cost. That can be cheaper at low QPS where a dedicated GPU would idle.

Self-hosted, the cost equation is pure infrastructure: a single T4 instance handles roughly the same document throughput for either model. There is no price advantage to switching.

Ergonomics and serving

Loading either model is trivial with sentence-transformers:

# GTE-large
from sentence_transformers import SentenceTransformer
gte = SentenceTransformer("Alibaba-NLP/gte-large")

# BGE-large
bge = SentenceTransformer("BAAI/bge-large-en")

BAAI also ships FlagEmbedding, a dedicated library with optimized serving (e.g., BGEModel with matryoshka support for v1.5). GTE-large has no first-party serving lib beyond standard HF ecosystems.

For production serving, both export to ONNX or TensorRT. BGE’s community has more ready-made Docker images for embedding APIs; GTE relies on generic SBERT servers. A minimal FastAPI endpoint for either looks identical:

from fastapi import FastAPI
from sentence_transformers import SentenceTransformer
app = FastAPI()
model = SentenceTransformer("BAAI/bge-large-en")

@app.post("/embed")
def embed(texts: list[str]):
    return model.encode(texts, normalize_embeddings=True).tolist()

Swap the model string and the rest stays the same.

Ecosystem and tooling

LangChain and LlamaIndex treat both as drop-in HuggingFaceEmbeddings. Vector DBs (Weaviate, Qdrant, Milvus) accept the raw vectors without modification.

BGE benefits from the bge-reranker companion models, letting you do two-stage retrieval with the same family. GTE has no official reranker, though cross-encoders from other sources work. For fine-tuning, both support Sentence-Transformers fit method; BGE’s FlagEmbedding includes scripts for hard-negative mining that many teams reuse.

Limits and edge cases

• Sequence length: 512 tokens hard cap. Longer texts need chunking.
• Dimensionality: Both output 1024 floats. BGE v1.5 supports matryoshka truncation to 512/256; base GTE-large does not.
• Normalization: BGE expects L2 normalization for cosine similarity; GTE-large docs recommend the same. Forgetting this hurts recall.
• Multilingual mix: Mixing languages in one index dilutes quality for both; pick the matching checkpoint.
• Padding overhead: Short queries (e.g., 10 tokens) still consume a full forward pass per sequence; batching is mandatory for throughput.

Head-to-head comparison

Which to choose

Use BGE-large-en if: You want the best English retrieval quality and might use the BGE reranker. If you need matryoshka truncation to save vector storage, use BGE-large-en-v1.5. Throughput is identical to GTE, so you lose nothing on speed.

Use GTE-large if: You already standardize on Alibaba’s GTE multilingual checkpoints, or your license policy prefers Apache 2.0 over MIT (though both are permissive). For pure English throughput workloads, there is no measurable win over BGE.

Use either behind a gateway if: You want to A/B test without standing up GPU boxes. Swapping model="BAAI/bge-large-en" to model="Alibaba-NLP/gte-large" on an OpenAI-compatible endpoint is a one-line config change.

For high-volume embedding jobs, benchmark on your own hardware with representative text lengths. The GTE-large vs BGE-large throughput gap will be within noise; pick on ecosystem and quality.