LlamaIndex Google Drive connector setup

This LlamaIndex Google Drive connector tutorial walks you through the full setup: OAuth credentials, service account configuration, incremental loading, and verification. If you’re building a RAG pipeline that needs to ingest Google Docs, Sheets, or PDFs from Drive, this is the practical path from zero to queryable index.

Step 1: Create Google Cloud project and enable APIs

Start in the Google Cloud Console. Create a new project or select an existing one, then enable two APIs:

1. Google Drive API — required for file metadata and content access
2. Google Docs API — required for exporting Google Docs/Sheets/Slides as plain text

gcloud services enable drive.googleapis.com docs.googleapis.com

If you’re using a service account (recommended for production), create one now:

gcloud iam service-accounts create llamaindex-drive-ingest \
    --display-name="LlamaIndex Drive Ingestion"

Grant the service account Viewer role on the project (or narrower: roles/drive.readonly if you prefer least privilege). Then create and download a JSON key:

gcloud iam service-accounts keys create ~/drive-credentials.json \
    --iam-account=llamaindex-drive-ingest@your-project.iam.gserviceaccount.com

Store this file outside your repo. Set GOOGLE_APPLICATION_CREDENTIALS in your environment or pass the path explicitly in code.

Step 2: Install LlamaIndex Google Drive dependencies

The connector lives in a separate package. Install it alongside the core library:

pip install llama-index llama-index-readers-google

Verify the import works:

from llama_index.readers.google import GoogleDriveReader
print("Import successful")

If you hit version conflicts, pin google-api-python-client>=2.100.0 and google-auth>=2.23.0 — older versions break the Drive v3 client.

Step 3: Configure authentication in code

Two paths exist: service account (headless) or OAuth user credentials (interactive). Choose one.

Service account (production)

import os
from llama_index.readers.google import GoogleDriveReader

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/path/to/drive-credentials.json"

reader = GoogleDriveReader()

The reader picks up GOOGLE_APPLICATION_CREDENTIALS automatically. No further config needed.

OAuth user credentials (local development)

If you prefer your own Google account, use the OAuth flow. First, create an OAuth 2.0 Client ID in Cloud Console (type: Desktop app). Download client_secrets.json.

from llama_index.readers.google import GoogleDriveReader

reader = GoogleDriveReader(
    credentials_path="/path/to/client_secrets.json",
    token_path="/path/to/token.json"  # cached after first auth
)

On first run, a browser window opens for consent. The token caches to token_path for subsequent runs.

Step 4: Load documents from a folder or shared drive

The reader accepts either a folder_id (for a specific Drive folder) or shared_drive_id (for a Shared Drive). Find the ID in the URL: https://drive.google.com/drive/folders/<FOLDER_ID>.

documents = reader.load_data(
    folder_id="1aBcDeFgHiJkLmNoPqRsTuVwXyZ",  # replace with your folder ID
    # shared_drive_id="0AABBCCDDEEFF",       # alternative: Shared Drive ID
    file_extensions=[".pdf", ".docx", ".txt", ".md", ".csv"],  # optional filter
)
print(f"Loaded {len(documents)} documents")

Each Document object contains:

• text — extracted content (plain text for Google Docs, OCR’d text for PDFs/images)
• metadata — file_id, file_name, mime_type, created_time, modified_time, web_view_link, parents

Incremental loading (recommended)

Re-processing entire folders on every run wastes quota and time. Track the last successful modified_time and filter:

import json
from pathlib import Path
from datetime import datetime

STATE_FILE = Path("drive_ingest_state.json")

def load_state():
    if STATE_FILE.exists():
        return json.loads(STATE_FILE.read_text())
    return {"last_modified": None}

def save_state(last_modified: str):
    STATE_FILE.write_text(json.dumps({"last_modified": last_modified}))

state = load_state()
last_modified = state["last_modified"]

documents = reader.load_data(
    folder_id="1aBcDeFgHiJkLmNoPqRsTuVwXyZ",
    file_extensions=[".pdf", ".docx", ".txt", ".md"],
    # Only fetch files modified after our checkpoint
    modified_after=last_modified,
)

if documents:
    # Update checkpoint to the newest file we saw
    newest = max(doc.metadata["modified_time"] for doc in documents)
    save_state(newest)
    print(f"Loaded {len(documents)} new/updated documents")
else:
    print("No new documents since last run")

The modified_after parameter accepts RFC3339 timestamps (e.g., "2024-01-15T14:30:00Z"). The reader handles pagination automatically.

Step 5: Parse and chunk for retrieval

Raw Drive documents are often too large for direct embedding. Split them:

from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import Document

parser = SentenceSplitter(
    chunk_size=1024,
    chunk_overlap=128,
    separator=" ",
)

nodes = parser.get_nodes_from_documents(documents)
print(f"Created {len(nodes)} nodes")

For Google Docs with complex structure (headings, tables), consider MarkdownNodeParser after exporting as markdown — but the Drive API exports as plain text, so SentenceSplitter is the pragmatic default.

Step 6: Build and persist the vector index

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.faiss import FaissVectorStore
import faiss

# In-memory FAISS for demo; swap for Pinecone/Weaviate/Qdrant in prod
d = 1536  # text-embedding-3-small dimension
faiss_index = faiss.IndexFlatL2(d)
vector_store = FaissVectorStore(faiss_index=faiss_index)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex(
    nodes,
    storage_context=storage_context,
    show_progress=True,
)

# Persist to disk
index.storage_context.persist(persist_dir="./drive_index")
print("Index persisted to ./drive_index")

Swap FaissVectorStore for your production vector store. The persist_dir contains docstore.json, index_store.json, vector_store.json, and the FAISS binary — copy this directory to your query service.

Step 7: Verify the pipeline end-to-end

Run a test query against the persisted index:

from llama_index.core import load_index_from_storage
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core import Settings

Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

storage_context = StorageContext.from_defaults(persist_dir="./drive_index")
index = load_index_from_storage(storage_context)

query_engine = index.as_query_engine(similarity_top_k=4)
response = query_engine.query("What is the Q3 budget forecast?")
print(response)
print("\n--- Sources ---")
for node in response.source_nodes:
    print(f"  {node.metadata['file_name']} (score: {node.score:.3f})")

Verification checklist:

• Response contains relevant content from your Drive documents
• source_nodes reference actual file names from the folder
• Scores are reasonable (typically > 0.7 for good matches on cosine similarity)
• No KeyError on metadata fields — confirms file_name, file_id, modified_time are present

If the query returns “I don’t know” or empty sources, check:

1. documents list wasn’t empty in Step 4
2. nodes have non-zero text length
3. Embedding model matches the one used at index time

Step 8: Automate with a scheduled job

Wrap Steps 4–6 in a script and schedule it. Cron example (runs daily at 2 AM):

# /etc/cron.d/llamaindex-drive-ingest
0 2 * * * appuser cd /opt/llamaindex-drive && /opt/venv/bin/python ingest.py >> /var/log/llamaindex-drive.log 2>&1

ingest.py should:

1. Load state
2. Fetch incremental documents
3. Parse to nodes
4. Update index (add new nodes, skip existing by file_id + modified_time)
5. Persist
6. Save state

Handling deletions

Drive doesn’t push deletion events via the API. Two strategies:

Option A: Full re-sync weekly

# In your weekly job, skip modified_after and rebuild
documents = reader.load_data(folder_id=FOLDER_ID, file_extensions=EXTENSIONS)
# Rebuild index from scratch

Option B: Track known file IDs

known_ids = set(existing_index.docstore.docs.keys())  # file_ids we have
current_ids = {doc.metadata["file_id"] for doc in documents}
deleted_ids = known_ids - current_ids

if deleted_ids:
    index.delete_ref_doc(list(deleted_ids), delete_from_docstore=True)
    print(f"Removed {len(deleted_ids)} deleted files")

Option B is more surgical but requires keeping the docstore in sync.

Step 9: Monitor quota and errors

Google Drive API has per-project quotas (default: 10,000 queries/100 seconds/user). The reader batches requests, but large folders can hit limits. Add exponential backoff:

from google.api_core import retry
from google.api_core.exceptions import ResourceExhausted

@retry.Retry(
    predicate=retry.if_exception_type(ResourceExhausted),
    initial=2.0,
    maximum=60.0,
    multiplier=2.0,
)
def load_with_backoff(reader, **kwargs):
    return reader.load_data(**kwargs)

documents = load_with_backoff(reader, folder_id=FOLDER_ID, file_extensions=EXTENSIONS)

Log quota errors distinctly so alerts fire:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

try:
    documents = load_with_backoff(reader, folder_id=FOLDER_ID)
except ResourceExhausted as e:
    logger.error("Drive API quota exhausted", extra={"error": str(e)})
    raise

Step 10: Secure the credentials in deployment

Never commit drive-credentials.json or client_secrets.json. In production:

• Kubernetes: Mount as a secret (kubectl create secret generic drive-creds --from-file=credentials.json=drive-credentials.json)
• Cloud Run / Cloud Functions: Use Secret Manager
• ECS / Lambda: Use AWS Secrets Manager or Parameter Store
• VM: Restrict file permissions (chmod 600) and use IAM roles where possible

The reader respects GOOGLE_APPLICATION_CREDENTIALS — point it at the mounted secret path.

Common pitfalls

Production hardening checklist

• Service account with roles/drive.readonly only
• Credentials in secret manager, not env vars
• Incremental loading with persistent state
• Deletion handling (weekly re-sync or ID tracking)
• Quota monitoring + alerting
• Vector store with persistence + backup
• Query latency < 500ms p95 (test with your corpus size)
• Integration test in CI that loads a fixture folder and runs a known query

This LlamaIndex Google Drive connector tutorial gives you a production-ready ingestion pipeline. The same pattern applies to other Google Workspace sources — swap the reader for GoogleDocsReader, GoogleSheetsReader, or GmailReader when those data sources become relevant.