Large language models (LLMs) turned many features that once needed a research team into an API call: summarization, classification, extraction, question answering and conversational assistants. Adding them to a product still takes architecture. A demo that works in a notebook breaks on cost, latency, accuracy and safety once real users arrive.
Where AI fits in an application
- Assistants: chat interfaces that answer questions about your product, documents or data.
- Automation: reading invoices, tagging support tickets, drafting replies for human review.
- Search and recommendations: semantic search using embeddings.
- Content workflows: first drafts, translations and summaries that staff edit.
Start with a task where mistakes are cheap to catch and the time saved is clear. Customer support triage is a common first project.
Keep the model behind your own service
Call the model provider from your backend, not from the browser, so API keys stay private and you can enforce authentication, rate limits and logging. A thin internal "AI gateway" module also lets you swap providers or models without changing the rest of the app. Providers such as OpenAI, Anthropic, Google and open-source models hosted on your own infrastructure have different strengths in quality, price, latency and data handling.
// Server route: stream a response
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";
export async function POST(req: Request) {
const { messages } = await req.json();
const result = streamText({ model: openai("gpt-4o-mini"), messages });
return result.toUIMessageStreamResponse();
}Stream responses to the user. Seeing words appear within a second feels faster than waiting ten seconds for a full answer.
Ground answers with retrieval
A model does not know your private data and can state false things with confidence. Retrieval-augmented generation (RAG) reduces this by fetching relevant passages and including them in the prompt.
- Split documents into chunks of a few hundred words.
- Convert each chunk to an embedding vector and store it in a vector index: pgvector, Pinecone, Qdrant, Weaviate or OpenSearch.
- At question time, embed the question, retrieve the closest chunks and add them to the prompt.
- Instruct the model to answer only from the provided text and to say when it does not know. Return source links.
Retrieval quality drives answer quality. Combine vector search with keyword search, filter by permissions so users only see what they may read, and refresh the index when documents change.
Let the model call tools
Tool calling lets the model request actions, such as "look up order status" or "create a ticket", which your code executes and returns. Define each tool with a strict schema, validate the arguments, check authorization for the current user, and require human confirmation for actions with side effects like refunds.
Evaluate before and after launch
Build a test set of real questions with expected answers. Run it on every prompt or model change, and track accuracy, refusal rate, latency and cost. Add user thumbs-up and thumbs-down feedback and review a sample of conversations each week. Without evaluation, a prompt tweak that fixes one case breaks five others.
Control cost and latency
- Use a small, cheap model for simple tasks and a larger one only when needed.
- Cap input length, trim chat history and cache repeated prompts and responses.
- Use provider features such as prompt caching for long, stable instructions.
- Set per-user and per-day limits and a monthly budget alert.
- Run slow tasks in the background and notify the user when done.
Guardrails and safety
- Treat all user and document text as untrusted. Prompt injection, where text tells the model to ignore its instructions, is a real risk, especially when the model reads web pages or emails and can call tools.
- Never place secrets in prompts or give the model broader data access than the user has.
- Filter or moderate inputs and outputs for abuse, and log them with access controls.
- Show users that they are talking to an AI and offer a path to a human.
Privacy and compliance
Check what each provider does with your data, whether it trains on prompts and where it stores them. Remove or mask personal data when you can, get consent where required, and document your use for audits under laws such as GDPR and India's DPDP Act.
A rollout path
- Pick one narrow use case and define success numerically.
- Build a prototype with a hosted model and a small test set.
- Add retrieval, logging and guardrails.
- Pilot with a small user group and review the results.
- Expand, and keep a human in the loop where errors cost money.
Common questions
How much does it cost to run an AI feature?
Provider pricing is per token, so cost grows with prompt length and traffic. A support assistant answering a few thousand short questions a month often costs little, while long documents sent with every request add up. Estimate with real prompts and set budget alerts.
We build AI chatbots, document automation and WhatsApp assistants for businesses, including the chat assistant on this site. Discuss your use case or read about our AI solutions.