CVCodeVibers Telegram

Technical branch for developers Assemble manually

How LLM API Works

LLM API is not a “magic chat” but a reliable service with a clear contract. Let’s break down how it works internally, how to use it without pain, and how not to break production.

Discuss implementation All Academy Topics

What is an LLM API?

LLM API is an HTTP interface through which an application sends text (a prompt) and receives generated text from a large language model. It’s not a “model in the cloud,” but a strict contract: POST method, JSON body, tokens, timeouts, and retry policies. At CodeVibers, we use it as the “engine” for agents, RAG, content generation, and automation.

How the request is processed

flowchart TD  
    A[Client: POST /v1/chat/completions] --> B[API Gateway]  
    B --> C[Auth: API key / OAuth]  
    C --> D[Rate Limiter]  
    D --> E[Tokenizer: text → tokens]  
    E --> F[LLM Inference: model + context]  
    F --> G[Detokenizer: tokens → text]  
    G --> H[Post-processing: safety, filtering]  
    H --> I[Response: JSON with content + usage]

Important: Tokenization isn’t just “counting words.” Different models (GPT, Claude, Llama) use different tokenizers. One character may equal 0.3 tokens, and an emoji may count as 2. This affects limits and cost.

Minimal request (Python)

import requests
import os

response = requests.post(
    "https://api.openai.com/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}",
        "Content-Type": "application/json"
    },
    json={
        "model": "gpt-4o-mini",
        "messages": [
            {"role": "system", "content": "You are an assistant for technical documentation."},
            {"role": "user", "content": "Explain what RAG is in 3 sentences."}
        ],
        "temperature": 0.3,
        "max_tokens": 256
    }
)

if response.ok:
    data = response.json()
    print(data["choices"][0]["message"]["content"])
    print("Tokens used:", data["usage"]["total_tokens"])
else:
    print("Error:", response.status_code, response.text)

Common mistakes (and how to avoid them)

  • Token overflow: prompt + response > model limit → error. Solution: count tokens in advance, use `max_tokens` and `truncation`.
  • No retry logic: API fails due to `429` or `503`. Solution: exponential backoff + jitter (e.g., `backoff` in Python).
  • Ignoring `usage`: do not track `prompt_tokens`, `completion_tokens`, `total_tokens` → no cost control.
  • No structure validation: The model returns JSON, but without `type: object` in `response_format` - you can get "`{ "data": "not JSON" }`".

Example: Handling a response with fallback

def safe_llm_call(messages, max_retries=3):
    for attempt in range(max_retries):
        try:
            resp = requests.post(API_URL, json={
                "model": "gpt-4o-mini",
                "messages": messages,
                "response_format": {"type": "json_object"}
            }, timeout=10)
            resp.raise_for_status()
            data = resp.json()
            # Validate structure
            content = data["choices"][0]["message"]["content"]
            return json.loads(content)  # → dict
        except (json.JSONDecodeError, KeyError) as e:
            if attempt == max_retries - 1:
                raise ValueError("Invalid model response") from e
            time.sleep(2 ** attempt + random.random())
        except requests.exceptions.RequestException as e:
            if attempt == max_retries - 1:
                raise RuntimeError("API unavailable") from e
            time.sleep(2 ** attempt + random.random())

At CodeVibers, such wrappers are standard—they turn an “unstable API” into a reliable service.

Production-Ready LLM API Checklist

  • [ ] Uses `response_format` for structured outputs (JSON mode)
  • [ ] Timeouts (no more than 10–15 seconds for chat)
  • [ ] Implemented retry with exponential backoff
  • [ ] Logs `usage` and `model` for auditing and optimization
  • [ ] Check `finish_reason` ("length" = truncated, "stop" = normal)
  • [ ] Handling `content_filter` (OpenAI) or `safety` (Claude)
  • [ ] Caching identical requests (if allowed by business)

Next step

Now that you understand how the LLM API works under the hood, you can build agents on top of it. Proceed to CodeVibers Academies and study “How to Assemble an Agent with Tool-Calling” — There we connect the API, logic, and external tools.

Related Topics