Retry & Fallback

→ Documentation index

Reliable extraction requires two separate retry layers that work independently:

Layer	What it handles	Configuration
Validation retries	LLM returned wrong schema / invalid values	max_primary_retries, max_fallback_retries
Network retries	Connection errors, timeouts, rate limits	RetryConfig

On top of both layers, a fallback model can take over when the primary model exhausts its validation retry budget.

---

Validation retries

Runnable example: examples/03_with_fallback.py

When Pydantic validation fails, the library builds a structured, field-level error report and appends it to the conversation as a HumanMessage:

"The output format is not correct. Please correct it based on the following errors:

MISSING REQUIRED FIELDS:
  - 'revenue_usd_millions': required field is missing

TYPE ERRORS:
  - 'fiscal_year': wrong type. Expected: integer, Received: str = '2025'

REQUIRED CHANGES:
  1. Add all missing required fields.
  2. Correct the data types of the specified fields."

The LLM receives this message and gets another attempt. This loop continues up to max_primary_retries times.

result, stats = await extract_data(
    llm,
    MySchema,
    messages,
    max_primary_retries=5,   # allow 5 correction attempts (default: 3)
)

The same mechanism applies to invalid tool calls (malformed JSON): the library detects them, includes the parse error in the retry message, and tries again.

---

Fallback model

Runnable example: examples/03_with_fallback.py

When the primary model exhausts all its retries, an optional fallback model takes over. It receives a clean context — the original messages plus one sentence noting that a previous attempt failed — so the accumulated retry noise from the primary model does not confuse it.

from langchain_openai import ChatOpenAI
from saidex import extract_data_from_text

primary  = ChatOpenAI(model="gpt-4o-mini", temperature=0)  # fast + cheap
fallback = ChatOpenAI(model="gpt-4o",      temperature=0)  # more capable

result, stats = await extract_data_from_text(
    primary,
    FinancialReport,
    report_text,
    fallback_llm_model=fallback,
    max_primary_retries=2,   # fail fast on primary …
    max_fallback_retries=3,  # … be more patient on fallback
)

print(f"Fallback used:     {stats.fallback_used}")
print(f"Primary retries:   {stats.primary_retries}")
print(f"Fallback retries:  {stats.fallback_retries}")
print(f"Total retries:     {stats.total_retries}")

Why use a fallback model?

• Cost efficiency: The primary model is cheaper. The fallback only runs when actually needed.
• Reliability: Smaller models sometimes fail on complex or deeply-nested schemas. A larger model handles them as a safety net.
• Clean context: The fallback starts fresh — without the failed attempts that could confuse it.

What the fallback model receives

# Original messages (unchanged):
[
    SystemMessage("Extract the financial report data."),
    HumanMessage(report_text),
]
# + one added hint:
HumanMessage(
    "A previous attempt to produce structured output for 'FinancialReport' "
    "failed after 2 retries. Please try again carefully, ensuring your "
    "response strictly follows the required JSON schema."
)

---

Reasoning-effort escalation

Reasoning models can think harder at a higher cost. Start cheap and only spend more effort when an attempt is rejected:

result, stats = await extract_data(
    llm,                                   # e.g. ChatOpenAI(model="gpt-5-mini")
    Invoice,
    messages,
    reasoning_escalation=["low", "medium", "high"],
)
print(stats.reasoning_effort)              # level of the attempt that decided the run

• Attempt n of a model binds reasoning_effort=levels[min(n-1, len-1)]; the last level repeats. Parse errors use up an attempt too.
• reasoning_escalation=True uses the levels from the model's profile (llm.profile["reasoning_effort_levels"]); without them nothing is escalated. True starts at the profile's first (cheapest) level and only climbs as far as the attempt budget allows: with the defaults, gpt-5-mini gets minimal, low, medium and never high. Pass an explicit list to start higher. It does nothing on langchain-core versions without model profiles (< 1.x). If the profile data is stale, override llm.profile to bypass the check.
• The accepted type is exported as saidex.ReasoningEscalation.
• The fallback model only escalates with its own fallback_reasoning_escalation.
• With a profile, unknown level names raise ValueError before any call; a plain string ("high") or an empty list is rejected too.
• In the agent loop, only final-answer attempts move to the next level; helper-tool calls keep the current one.
• The level is bound as reasoning_effort, so the integration must accept it:
• ChatOpenAI works (Chat Completions maps it to reasoning_effort, the Responses API to reasoning.effort).
• Recent ChatAnthropic works: langchain-anthropic maps reasoning_effort to output_config.effort. On adaptive-thinking models this enables thinking, which Anthropic rejects together with a forced tool_choice. SAIDEX then relaxes the tool binding automatically at the cost of one extra call, so pass tool_config=ToolCallConfig.COMPATIBLE. The agent loop is unaffected.
• Integrations that do not accept the kwarg (e.g. ChatOllama), or a ChatOpenAI(reasoning=...) set on the model (conflict), fail on the first attempt with failure_reason="llm_error" and llm_error_kind="invalid_request".
• Validators wrapped with with_context see the level as ctx.reasoning_effort.

Higher levels cost more tokens and latency — escalation keeps the common case cheap and pays more only for hard documents.

---

Network retries (RetryConfig)

Runnable example: examples/04_custom_retry_config.py

Network retries are handled separately from validation retries. They catch transient errors — connection resets, timeouts — and rate-limit responses before the validation loop ever sees the result.

Default configuration

When retry_config is not provided, the library uses DEFAULT_RETRY_CONFIG:

RetryConfig(
    max_retries=3,
    retry_delays=[1.0, 2.0, 4.0],       # exponential back-off
    retryable_exceptions=(               # auto-detected from installed packages:
        ModelConnectionError,            #   langchain-core (if recent enough)
        ModelTimeoutError,
        ModelAPIError,                   #   HTTP 5xx
        openai.APIConnectionError,       #   openai (if installed)
        openai.APITimeoutError,
        openai.InternalServerError,
        httpx.ConnectError,              #   httpx / httpx2 (if installed)
        httpx.TimeoutException,
        httpx.RemoteProtocolError,
    ),
    rate_limit_exceptions=(              # if available
        ModelRateLimitError,
        openai.RateLimitError,
    ),
    rate_limit_retry_interval=10.0,      # seconds between rate-limit retries
    rate_limit_max_duration_seconds=900.0,   # give up after 15 minutes
)

Transient errors are langchain-core's ModelConnectionError, ModelTimeoutError and ModelAPIError (HTTP 5xx), plus openai.APIConnectionError, openai.APITimeoutError, openai.InternalServerError and the httpx / httpx2 ConnectError, TimeoutException and RemoteProtocolError. Rate limits are ModelRateLimitError and openai.RateLimitError.

Server errors are retried by default

Server errors (HTTP 5xx) are retried by default; versions up to 0.7.x did not retry them. These retries stack with the SDK's own retries (for example ChatOpenAI(max_retries=2)): a persistent 500 can take about 12 HTTP calls before the fallback model runs. A custom RetryConfig (without server errors, or with fewer retries) gives faster failover.

A custom RetryConfig(retryable_exceptions=...) keeps full control over what is retried. To react to a failed run in your own handlers, use saidex.classify_model_error(exc) or stats.llm_error_kind (see Failure Reasons).

Custom configuration

from saidex import RetryConfig, extract_data_from_text

config = RetryConfig(
    max_retries=2,
    retry_delays=[2.0, 5.0],             # longer waits
    rate_limit_retry_interval=30.0,      # wait 30 s between rate-limit retries
    rate_limit_max_duration_seconds=300.0,   # give up after 5 minutes
)

result, stats = await extract_data_from_text(
    llm, MySchema, text, retry_config=config
)

The example file 04_custom_retry_config.py shows this pattern alongside a create_instance_safe standalone demo and a note on how to add Langfuse tracing via callbacks.

Disabling network retries (e.g. in tests)

NO_RETRY = RetryConfig(
    max_retries=0,
    retry_delays=[],
    retryable_exceptions=(),
    rate_limit_exceptions=(),
)

result, stats = await extract_data(
    llm, MySchema, messages, retry_config=NO_RETRY
)

Custom exception types

import httpx
import openai

config = RetryConfig(
    retryable_exceptions=(
        httpx.ConnectError,
        httpx.TimeoutException,
        openai.APIConnectionError,
        openai.APITimeoutError,
    ),
    rate_limit_exceptions=(openai.RateLimitError,),
)

RetryConfig reference

Field	Type	Default	Description
max_retries	int	3	Max transient-error retries per LLM call
retry_delays	list[float]	[1.0, 2.0, 4.0]	Seconds to wait before each retry
retryable_exceptions	tuple[type[Exception], ...]	langchain-core, openai and httpx network and 5xx errors	Exceptions that trigger a transient retry
rate_limit_exceptions	tuple[type[Exception], ...]	ModelRateLimitError, openai.RateLimitError	Exceptions that trigger rate-limit wait
rate_limit_retry_interval	float	10.0	Seconds between rate-limit retries
rate_limit_max_duration_seconds	float	900.0	Total seconds before giving up on rate-limit retries

---

Full retry flow

extract_data(primary_llm, schema, messages)
│
│  Phase 1 — Primary model
│  ┌────────────────────────────────────────────────┐
│  │  for attempt in range(max_primary_retries):    │
│  │    try LLM call (with network retries)         │
│  │    validate response with Pydantic             │
│  │    if OK → return (instance, stats)            │
│  │    else   → append error message, retry        │
│  └────────────────────────────────────────────────┘
│         ↓ all attempts failed
│
│  Phase 2 — Fallback model (if configured)
│  ┌────────────────────────────────────────────────┐
│  │  reset to original messages + hint             │
│  │  for attempt in range(max_fallback_retries):   │
│  │    try LLM call (with network retries)         │
│  │    validate response with Pydantic             │
│  │    if OK → return (instance, stats)            │
│  │    else   → append error message, retry        │
│  └────────────────────────────────────────────────┘
│         ↓ all attempts failed
│
└─ return (None, stats)

---

Error handling

Always check whether the result is None before using it:

result, stats = await extract_data_from_text(llm, MySchema, text)

if result is None:
    logger.error(
        "Extraction failed after %d retries (fallback used: %s)",
        stats.total_retries,
        stats.fallback_used,
    )
    # Options:
    # - raise an exception
    # - return a default / partial result
    # - skip this record and continue
    raise RuntimeError("Could not extract structured data from the document.")

# result is guaranteed valid here
process(result)

stats.failure_reason tells you why the run failed and whether retrying later is worthwhile — see Failure reasons.

Tracking retry costs

ExtractDataStats lets you track quality metrics over time:

results = []
for text in documents:
    result, stats = await extract_data_from_text(llm, MySchema, text)
    results.append({
        "result": result,
        "retries": stats.total_retries,
        "fallback": stats.fallback_used,
        "failed": result is None,
    })

failures   = sum(1 for r in results if r["failed"])
avg_retries = sum(r["retries"] for r in results) / len(results)
print(f"Failure rate: {failures}/{len(results)}, avg retries: {avg_retries:.2f}")

---

Related

• Extraction — the two extraction functions in detail
• Schema Design — reducing retries by writing better schemas
• Validators — reducing retries by cleaning LLM output in the schema
• Observability — tracing retries with Langfuse / LangSmith