Build Semantic Search for Legal Documents with Pinecone and GPT-4

You can build a semantic search system for legal documents by combining Pinecone (a vector database), GPT-4 (for intelligent retrieval), and LangChain (to orchestrate the pipeline). This setup lets law firms search case law, contracts, and precedents by meaning rather than keyword matching - so a query like "wrongful termination without cause" surfaces relevant statutes even when documents use different phrasing. The result is a retrieval-augmented generation (RAG) system that feeds the most relevant legal context to GPT-4, so it can cite sources and answer with accuracy.

What you need

The tech stack below represents a realistic, minimal setup to ship a working legal document semantic search system. Prices are current as of August 2024; verify each provider's current terms before committing.

Time to build: 4-6 hours for a working prototype on 100-500 documents; 1-2 weeks to production-grade (chunking strategy, fine-tuning relevance, auth, audit logging for legal compliance).

Step 1: Chunk and embed your legal documents

Legal documents are too large to embed as a single vector. A 50-page contract or court opinion needs to be split into semantic chunks - typically 300-800 tokens each - so that GPT-4 can retrieve relevant sections without losing context.

Start by uploading your documents (PDFs, Word files, plaintext) to a staging directory. Then use LangChain's document loaders to parse them:

from langchain_community.document_loaders import PyPDFLoader, UnstructuredPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
import os

# Load all PDFs from a directory
docs = []
for filename in os.listdir('./legal_documents'):
 if filename.endswith('.pdf'):
 loader = PyPDFLoader(f'./legal_documents/{filename}')
 docs.extend(loader.load())

# Split documents into chunks
splitter = RecursiveCharacterTextSplitter(
 chunk_size=800,
 chunk_overlap=100,
 separators=["\n\n", "\n", " "]
)
chunks = splitter.split_documents(docs)

print(f"Created {len(chunks)} chunks from {len(docs)} documents")

This script loads PDFs, splits them into 800-token chunks with 100-token overlap (so key phrases aren't cut off), and reports how many chunks you've created. The overlap ensures that a search query at a chunk boundary still finds the right content. Adjust chunk_size down to 400 tokens if your documents are dense with cross-references (e.g., statutes with many citations); go up to 1,200 if they're more narrative (e.g., case summaries).

Why this matters: a chunk too large (5,000+ tokens) dilutes semantic relevance across unrelated sections; a chunk too small (<300 tokens) loses enough context that GPT-4 struggles to answer follow-up questions. Most law firms find 500-800 tokens is the sweet spot.

Step 2: Initialize Pinecone and upload embeddings

Pinecone is a managed vector database that indexes embeddings for fast similarity search. You'll create an index, then embed each chunk and upsert it with metadata (document name, page number, date) so you can trace answers back to source.

First, install dependencies and authenticate:

pip install pinecone-client openai langchain langchain-openai langchain-pinecone

Then set your API keys as environment variables:

export OPENAI_API_KEY="your-key-here"
export PINECONE_API_KEY="your-key-here"
export PINECONE_ENVIRONMENT="us-east-1" # check your Pinecone console for your region

Initialize Pinecone and create an index:

from pinecone import Pinecone
from openai import OpenAI

# Initialize Pinecone
pc = Pinecone(api_key=os.environ.get("PINECONE_API_KEY"))

# Create index if it doesn't exist
index_name = "legal-docs"
if index_name not in pc.list_indexes().names():
 pc.create_index(
 name=index_name,
 dimension=1536, # OpenAI's text-embedding-3-small produces 1536-dim vectors
 metric="cosine",
 spec={
 "serverless": {
 "cloud": "aws",
 "region": "us-east-1"
 }
 }
 )

index = pc.Index(index_name)
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

The dimension is 1536 because OpenAI's text-embedding-3-small model outputs 1536-dimensional vectors. The cosine metric measures similarity (0 = completely different, 1 = identical). Serverless means Pinecone auto-scales without you managing pods.

Now embed and upload each chunk:

def embed_and_upsert(chunks, batch_size=100):
 """Embed chunks and upload to Pinecone."""
 vectors_to_upsert = []
 
 for i, chunk in enumerate(chunks):
 # Get embedding from OpenAI
 response = client.embeddings.create(
 input=chunk.page_content,
 model="text-embedding-3-small"
 )
 embedding = response.data[0].embedding
 
 # Prepare metadata
 metadata = {
 "text": chunk.page_content[:500], # first 500 chars for display
 "source": chunk.metadata.get("source", "unknown"),
 "page": chunk.metadata.get("page", 0),
 }
 
 vectors_to_upsert.append((
 f"chunk-{i}",
 embedding,
 metadata
 ))
 
 # Upsert in batches
 if (i + 1) % batch_size == 0:
 index.upsert(vectors=vectors_to_upsert)
 print(f"Uploaded {i + 1} / {len(chunks)} chunks")
 vectors_to_upsert = []
 
 # Upload remaining vectors
 if vectors_to_upsert:
 index.upsert(vectors=vectors_to_upsert)
 
 print(f"Complete: {len(chunks)} chunks indexed")

embed_and_upsert(chunks)

This batches uploads in groups of 100 to avoid timeout errors. Each vector is stored with metadata so you can show the user which document a result came from. This step typically costs $20-$60 for 10,000 chunks (depending on document size), because you pay OpenAI for embeddings.

Step 3: Build the retrieval chain with LangChain

Now that embeddings are in Pinecone, you'll chain retrieval + GPT-4 reasoning together. When a user asks a question, LangChain will: 1. Embed the question. 2. Search Pinecone for the top-k most similar chunks. 3. Feed those chunks as context to GPT-4. 4. Return an answer with citations.

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_pinecone import PineconeVectorStore
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

# Initialize embeddings and vector store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = PineconeVectorStore(
 index=index,
 embedding=embeddings
)

# Create a retriever (k=4 means return top 4 most similar chunks)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

# Define a custom prompt for legal context
legal_prompt = PromptTemplate(
 input_variables=["context", "question"],
 template="""You are a legal research assistant. Use the following context from legal documents to answer the question. 
If the answer is not in the context, say 'Not found in provided documents.'
Always cite the source document.

Context:
{context}

Question: {question}

Answer:"""
)

# Create the QA chain
llm = ChatOpenAI(model="gpt-4", temperature=0.1) # low temperature for legal accuracy
qa_chain = RetrievalQA.from_chain_type(
 llm=llm,
 chain_type="stuff",
 retriever=retriever,
 chain_type_kwargs={"prompt": legal_prompt},
 return_source_documents=True
)

# Test the chain
question = "What are the grounds for wrongful termination in California?"
response = qa_chain.invoke({"query": question})

print(f"Answer: {response['result']}")
print(f"Sources: {[doc.metadata['source'] for doc in response['source_documents']]}")

The temperature=0.1 setting keeps GPT-4 focused and factual - appropriate for legal work where hallucination is costly. The stuff chain type concatenates all retrieved documents and sends them to the LLM at once (fine for 4 chunks; for larger contexts, use map_reduce or refine mode). The return_source_documents=True flag ensures users can verify where answers came from.

Step 4: Add filtering and metadata search for legal precision

Legal document search often needs filtering by jurisdiction, case type, or date. Extend the retriever to support metadata-aware searches:

from langchain.retrievers.self_query.base import SelfQueryRetriever
from langchain.chains.query_constructor.base import AttributeInfo

# Define filterable metadata fields
metadata_field_info = [
 AttributeInfo(
 name="source",
 description="The file name or document title (e.g., 'California_Labor_Code.pdf')",
 type="string",
 ),
 AttributeInfo(
 name="page",
 description="Page number in the original document",
 type="integer",
 ),
 AttributeInfo(
 name="jurisdiction",
 description="State or country (e.g., 'California', 'Federal')",
 type="string",
 ),
]

# Create a self-querying retriever that can filter
retriever = SelfQueryRetriever.from_llm(
 llm=llm,
 vectorstore=vectorstore,
 document_contents="Legal documents: statutes, case law, contracts",
 metadata_field_info=metadata_field_info,
 verbose=True
)

# Query with implicit filtering
question = "What does California labor law say about at-will employment?"
response = qa_chain.invoke({"query": question})

The self-querying retriever uses GPT-4 to parse the user's question and automatically extract filters. So "California labor law" is interpreted as a filter on jurisdiction='California' plus a semantic search for "at-will employment." This saves law firms from writing complex filter syntax.

Step 5: Deploy as an API

Wrap the QA chain in a FastAPI server so law firm teams can query via web or integration:

from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn

app = FastAPI(title="Legal Document Search")

class QueryRequest(BaseModel):
 question: str
 top_k: int = 4

class QueryResponse(BaseModel):
 answer: str
 sources: list[str]

@app.post("/search")
async def search(request: QueryRequest):
 """Search legal documents and return answer with sources."""
 response = qa_chain.invoke({
 "query": request.question,
 "k": request.top_k
 })
 return QueryResponse(
 answer=response["result"],
 sources=[doc.metadata["source"] for doc in response["source_documents"]]
 )

@app.get("/health")
async def health():
 return {"status": "ok"}

if __name__ == "__main__":
 uvicorn.run(app, host="0.0.0.0", port=8000)

Run it locally with uvicorn app:app --reload, then test with curl:

curl -X POST http://localhost:8000/search \
 -H "Content-Type: application/json" \
 -d '{"question": "What are non-compete clause limits in California?", "top_k": 5}'

Deploy to a cloud platform (AWS Lambda, Google Cloud Run, or Heroku) for production. Add authentication (API keys or OAuth) if multiple users will access it.

!Step 5: Deploy as an API — build semantic search for legal documents ## Where this breaks

Building semantic search for legal documents introduces several real failure modes. Here's how to handle each:

Rate limits and embedding costs. OpenAI's API has rate limits (3,500 requests per minute on the free tier, higher on paid accounts). If you're embedding thousands of documents at once, you'll hit these limits. Fix: batch your embedding jobs across multiple hours, or use exponential backoff with a sleep counter between API calls. Also monitor your embedding costs - at ~$0.02 per 1M tokens, 100,000 legal documents (50 pages average) can cost $100-$500 to embed. Budget for this upfront.

Stale or irrelevant retrieval. If your top-k retrieved chunks don't actually answer the question, GPT-4 will admit it or make an inference that sounds plausible but is wrong. This happens when your chunk size is too large or too small, or when your documents have inconsistent formatting. Fix: test your retriever on 10-20 real legal questions before going live. If top results are off-topic, lower chunk_size to 500 tokens or adjust your prompt to be more strict ("Answer only from the provided context").

Pinecone costs scaling unexpectedly. Pinecone charges per query for serverless mode. If you have 1,000 law firm users running 10 searches a day, that's 10,000 queries per day. Check Pinecone's current pricing page to estimate monthly cost. Fix: cache frequently asked questions (e.g., "What is California wrongful termination law?") so you don't re-query for identical questions. Use a Redis cache layer in front of your API.

Token limits in context. GPT-4 has an 8,192-token context window (or 128,000 for GPT-4 Turbo). If you retrieve 10 chunks and each is 800 tokens, you're at 8,000 tokens before the user's question even arrives. Fix: limit k to 3-5 chunks, or use a reranker (a smaller, faster model that re-scores retrieved chunks for relevance before sending to GPT-4).

Missing or inconsistent metadata. If PDFs don't have consistent metadata (page numbers, dates, author), your citations will be vague. Fix: add a preprocessing step that extracts and validates metadata from PDF headers, or manually tag documents before uploading.

Cold-start latency. The first query after deployment can take 5-10 seconds because Pinecone needs to warm up. Users expect <1 second. Fix: warm up Pinecone with a dummy query on startup, or accept the cold-start and communicate it to users.

How to choose between Pinecone, Chroma, Weaviate, and FAISS

If you're building semantic search for legal documents, you have multiple vector database options. Here's how they compare:

Pinecone (serverless, managed) is easiest for production because it's hosted and scales automatically. You pay per query and storage, with no infrastructure overhead. Pinecone is ideal if you want to ship fast and don't mind a monthly bill ($0.04-$0.10 per 100 queries, roughly).

Chroma (open-source, self-hosted) is free and runs in-memory or on disk. It's great for prototypes and small deployments (<10,000 documents), but you manage scaling and backups. No monthly cost, but you run the server yourself.

Weaviate (open-source + managed) is a middle ground: free self-hosted version, or a managed cloud plan. It supports filtering and multi-modal search (text + images). If you need production-grade features without Pinecone's cost, Weaviate is worth evaluating.

FAISS (open-source, Facebook) is a bare-metal vector library, not a database. It's extremely fast for similarity search over millions of vectors, but it doesn't handle persistence, filtering, or distributed queries. Use FAISS if you're building a custom search engine with very large-scale data and you have an engineering team to maintain it.

For a law firm, Pinecone is the most practical choice because it handles auth, audit logging, and uptime guarantees - all non-negotiable in regulated industries. If cost is the primary concern and you have in-house DevOps, Weaviate is competitive.

How to build semantic search for legal documents step-by-step: a real example

Let's walk through a concrete example: embedding a corpus of California labor law statutes and case summaries, then querying them.

Assume you have three PDFs: - California_Labor_Code.pdf (100 pages) - Wrongful_Termination_Cases_2023.pdf (50 pages) - Non_Compete_Agreement_Guide.pdf (30 pages)

After chunking, you'll have roughly 600-1,000 chunks (at 800 tokens per chunk). Uploading to Pinecone costs ~$1 in OpenAI embeddings (1M tokens * $0.02 / 1M). Then, every search query costs ~$0.05 (embedding the question + a small LLM inference). At 100 queries per month, that's ~$5 in query costs.

A user asks: "Can a California company enforce a non-compete clause?"

The LangChain chain will: 1. Embed the question (1 API call, ~$0.00001). 2. Search Pinecone for top 4 chunks with "non-compete" and "California" (retrieved from metadata and semantic similarity). 3. Pass the 4 chunks (~3,200 tokens) plus the question to GPT-4. 4. GPT-4 reads the context and replies: "California Business and Professions Code Section 16600 restricts non-compete clauses. Generally, any contract that restrained an individual from engaging in a lawful profession is void. However, there are narrow exceptions for the sale of a business or dissolution of a partnership (see [source: California_Labor_Code.pdf, page 87])."

The end-to-end cost is ~$0.02 per query. Accuracy depends on your chunk quality and metadata tagging.

For a deeper technical reference, see OpenAI's docs.

FAQ

How do I avoid hallucination in legal contexts?

Set temperature=0.1 in your LLM config to keep outputs factual and deterministic. Use a prompt that explicitly says "Do not infer beyond the provided documents." Add a human approval step before any answer is sent to a client - no AI system is 100% accurate in law, and GPT-4's mistakes could be costly.

Can I use open-source models (Llama, Mistral) instead of GPT-4?

Yes. Open-source models like Mistral-7B or Llama-2-70B can work, especially if you fine-tune them on legal documents. However, they typically require more prompting to stay on-topic, and they're less familiar with U.S. law than GPT-4. If you self-host them (via Ollama or vLLM), you save on API costs but add infrastructure complexity. For law firms, GPT-4's accuracy usually justifies the cost.

What's the difference between semantic search and keyword search?

Keyword search looks for exact word matches ("non-compete" returns only documents with that word). Semantic search embeds meaning, so "non-compete clause" and "restriction on competition" return similar results even without exact phrase overlap. For legal documents, semantic search is more powerful because statutes use varied language, but it requires the retriever to be well-trained - hence why you need to test your chunks and prompts carefully.

How do I handle very long legal documents (500+ pages)?

Don't embed the entire document as one chunk. Split into 500-800 token chunks with 100-token overlap, as shown in Step 1. If documents have clear sections (e.g., "Article III", "Section 2"), use those as natural split points instead of raw character counts. This preserves semantic boundaries and makes retrieval more precise.

Can I build semantic search for legal documents without Pinecone?

Yes. Use Weaviate (check the provider's current pricing and self-hosting options) for a free self-hosted option, or Chroma for rapid prototyping. Both work with LangChain and GPT-4. The tradeoff is infrastructure: Weaviate and Chroma require you to run and scale the vector database, whereas Pinecone is fully managed. For a law firm MVP, start with Chroma locally; migrate to a managed solution (Pinecone or Weaviate Cloud) once you have real users and want production guarantees.

How do I ensure legal compliance and audit trails?

Log every query, result, and source document. Store logs in a database with timestamps and user IDs. Ensure your API is authenticated (use OAuth or API keys). If your documents are confidential (attorney-client privilege), run the system on-premise or in a VPC, and encrypt data in transit and at rest. Consult your firm's legal and compliance team on data handling before going live.

Next steps

You now have a working blueprint for semantic search on legal documents. The next stage is tuning for your specific use case: test retrieval accuracy on 20-30 real questions, adjust chunk size and top-k based on results, and add role-based access control if multiple teams will use it.

If you're building AI automation systems like this for clients, explore the AI automations you can sell guide to understand which automation workflows are most in-demand. And grab the free guide for a deeper dive into RAG pipelines, cost optimization, and avoiding common pitfalls.