Quality Analytics¶

Every extraction call returns stats that carry a structured list of [FieldIssue][saidex.FieldIssue] objects — one per field-level validation problem, tagged with the attempt it occurred in. A single run tells you whether that extraction worked. Aggregate many runs and a different question gets answered: which fields does the model keep getting wrong, and why?

That is what [summarize_field_issues][saidex.summarize_field_issues] is for. Feed it the stats from a benchmark sweep or a labelled evaluation set and it groups every issue per schema and per field, so you can find the weak field descriptions and prompts and fix them.

Collecting runs¶

field_issues are recorded even when a run ultimately succeeds — a field that failed on attempt 0 but was corrected on attempt 1 still shows up. That is deliberate: a field the model routinely fumbles (even if it recovers) is a prompt-tuning signal.

from saidex import extract_data_from_text, summarize_field_issues

runs = []
for text in dataset:                      # many varied inputs
    _, stats = await extract_data_from_text(llm, Invoice, text)
    runs.append(stats)

summary = summarize_field_issues(runs)

Runs salvaged by an on_exhausted hook count as successes in success_rate, but are tracked separately: salvaged_runs counts them and clean_success_rate is the share of runs that succeeded without salvage.

summarize_field_issues accepts a heterogeneous iterable: stats objects ([ExtractDataStats][saidex.ExtractDataStats] or [ExtractorRunStats][saidex.ExtractorRunStats]) and/or bare [FieldIssue][saidex.FieldIssue] instances. Stats objects contribute run context (success/failure and run counts); bare issues count only toward the occurrence metrics, because they carry no run outcome.

Reading the summary¶

for schema in summary.schemas:            # most problematic schema first
    print(f"{schema.schema_name}: {schema.success_rate:.0%} success "
          f"({schema.failed_runs}/{schema.total_runs} failed)")
    for fp in schema.field_problems:      # sorted by descending severity
        print(f"  {fp.field_path}: {fp.total_occurrences} issues, "
              f"{fp.failed_runs_with_problem} in failed runs, "
              f"recovery {fp.recovery_rate:.0%}, "
              f"top error {max(fp.by_error_type, key=fp.by_error_type.get)}")

Per-field metrics¶

Field	Meaning
`total_occurrences`	Every issue for this field, across all attempts and runs.
`runs_with_problem`	Runs in which the field failed at least once.
`failed_runs_with_problem`	Of those, how many runs ultimately failed.
`occurrences_on_successful_runs`	Issues that were self-corrected.
`salvaged_runs_with_problem`	Runs that only succeeded through an `on_exhausted` hook.
`recovery_rate`	Share of `runs_with_problem` that succeeded cleanly (salvaged runs do not count).
`by_category` / `by_error_type`	Issue counts per category and raw Pydantic error type.
`first_error_attempt_avg`	Mean earliest attempt the field failed — low means up-front.
`sample_received`	Up to five distinct offending values.
`severity`	Sort score: dominated by failed plus salvaged runs, tie-broken by `total_occurrences`.

A field with a low recovery_rate and a high failed_runs_with_problem is where to spend your prompt-tuning effort first — it causes hard failures. A field with a high recovery_rate is noisier than it is dangerous.

Bare issues and run counts

A bare FieldIssue passed straight in is not a run, so it never increments total_runs, failed_runs, or runs_with_problem — only the occurrence-based fields. A schema seen only through bare issues reports total_runs == 0 and success_rate == 0.0.

Markdown report¶

[FieldIssueSummary.to_markdown()][saidex.FieldIssueSummary.to_markdown] (or the standalone [render_field_issue_report][saidex.render_field_issue_report]) renders the summary as a Markdown document — one section per schema, fields sorted by descending severity. The output is deterministic for the same input, so it is safe to snapshot in tests or commit as a benchmark artifact.

print(summary.to_markdown())

# Field issue report

## Invoice — 33% success (2/3 runs failed)

| Field | Category | Top error | Hits | In failed runs | Recovery | Samples |
| --- | --- | --- | ---: | ---: | ---: | --- |
| total | type | float_parsing | 2 | 2 | 0% | `1.299,00`, `forty-two` |
| currency | enum | enum | 1 | 0 | 100% | `euros` |

Benchmark integration¶

The benchmark suite (benchmarks/run_benchmarks.py) does this for you across every model and scenario it runs. Each run's field_issues are collected and aggregated, then:

a Field Issue Analysis section is appended to the Markdown report,
the structured summary is written under field_issue_summary in the JSON results file, and
the top problem fields per schema are printed to the console.

Because every run is stored with its model, schema_name, and field_issues in the JSON, you can also re-aggregate per model offline.

Runnable example: examples/11_quality_analytics.py