ocr-quality-signals¶
ocr-quality-signals0.1.0noneTags: ocr document-quality anomaly-detection quantitative-signals workflow-gating
Audience: document-intelligence-teams, workflow-designers, evaluation-engineers
Manifestations in repo: prompt.md
Audits OCR markdown or plain text and returns a stable JSON record of observable anomaly rates: encoding damage, suspicious glyphs, implausible tokens, broken words, numeric corruption, line fragmentation, repeated page furniture, table irregularity, sparse pages, and heading-order problems. The output is designed for correlation studies and workflow gating experiments, not as ground-truth OCR accuracy.
Output contract: schema.json
Contract scope: repo-local OCR diagnostic contract
Inputs¶
One document's OCR-derived Markdown or plain text. In Vandalizer, the companion workflow receives the text produced by the platform's normal PDF OCR path. The component does not inspect pixels, OCR-engine confidence values, bounding boxes, or the original PDF.
Outputs¶
One JSON object with:
- input-size estimates and a context-coverage assessment;
- 12 signal records containing raw counts, denominators, rates, provisional thresholds, flags, confidence, and verbatim evidence examples; and
- a provisional risk index equal to the percentage of applicable signals that crossed their threshold.
risk_index is deliberately simple and inspectable. It is not a probability of failure, OCR accuracy, or a validated production gate. Raw signal rates are the primary analytical surface.
Measurement limitations¶
Vandalizer's current workflow format exposes Prompt, Extraction, and Knowledge Base Query tasks, not a deterministic code node. Counts from this component are therefore labeled llm_estimate. A production gate should eventually replace or corroborate character/token counts with a deterministic analyzer while retaining this JSON contract.
The component can only see OCR text. It cannot detect visual blur, skew, rotation, clipped margins, or a missing page when those failures leave no textual trace. Legitimate tables, identifiers, math, multilingual text, and repeated headers can also resemble OCR corruption.
Calibration plan¶
Use synthetic.workshop_ocr as the first calibration ladder:
- Run all 15 PDFs (3 document types × clean plus 4 degradation tiers) through the same Vandalizer OCR path.
- Run this component at least five times per OCR text and retain the median rate for every signal because the measurements are model-estimated.
- Run the downstream extraction workflow and score field accuracy, numeric-field accuracy, missing-field rate, and hallucinated-field rate against
ground_truth.json. - Measure Spearman correlation between each raw signal and each downstream error endpoint. Predefine a practical bad-run endpoint, such as field accuracy below 0.90, before examining threshold performance.
- Report threshold precision/recall and uncertainty. With only 15 PDFs, treat results as calibration evidence, not general validation.
- Externally check useful signals on
real.nsf_rfa_checklist_eval, using paired clean-text versus dots.OCR extraction deltas and anchored-field accuracy. That corpus has no complete human ground truth, so describe non-anchored results as agreement or robustness, not accuracy.
Contract scope¶
This is a prompt-library repo-local diagnostic schema. It does not extend or claim alignment to the shared AI4RA UDM.
Triad integration¶
- Evaluation datasets:
synthetic.workshop_ocrfor ground-truth calibration;real.nsf_rfa_checklist_evalfor paired external robustness checks. - Harness notes: validate output against
schema.json; store raw signal records alongside downstream extraction outcomes; use repeated measurements and median aggregation until deterministic counting is available. - Shared UDM relationship: none.
Manifestations¶
prompt.md— canonical prompt.workflows/ocr-quality-signals— uploadable Vandalizer manifestation.
Evals¶
See evals/ for the current calibration posture. No case is marked validated yet.
Provenance¶
Authored 2026-08-08 after OCR quality emerged as a major determinant of downstream Vandalizer workflow quality. The design keeps measured signals separate from the downstream outcome so correlations can be tested rather than assumed.
Contract scope¶
-
Output format:
json_object -
Contract scope:
repo_local_ocr_diagnostic_contract -
Validation surfaces:
json_schema -
Schema entrypoints:
# -
Notes: Repo-local diagnostic record containing 12 OCR-text anomaly signals, raw counts and denominators, provisional thresholds, verbatim examples, and a simple flag-density index. Measurements are LLM estimates; the index is not OCR accuracy, failure probability, or a validated gate.
-
Machine-readable catalog entry:
component_catalog.json
Triad integration¶
-
UDM alignment:
no_udm_relationship— This is a document-quality diagnostic contract and does not extend or claim alignment to the shared AI4RA UDM. -
Evaluation dataset:
synthetic.workshop_ocr(ground_truth_calibration_ladder) — Run the three synthetic document types across clean and four degradation tiers, then correlate raw signal rates with downstream field and numeric-field accuracy scored against ground_truth.json. -
Evaluation dataset:
real.nsf_rfa_checklist_eval(paired_external_robustness_check) — Compare signal movement and extraction deltas between clean web text and dots.OCR. Only anchored fields support accuracy claims; other comparisons remain robustness or agreement evidence. -
Harness notes: Validate output against schema.json and persist raw signal records beside downstream extraction outcomes. Until deterministic measurement exists, collect at least five replicates and aggregate signal rates by median.
-
Related component:
nsf-award-notice-extraction-udm(candidate_downstream_outcome_source) — Field-level and numeric-field accuracy from award-notice extraction can serve as dependent variables when calibrating OCR signals.
Prompt body¶
Source: prompt.md.
Show prompt
OCR Quality Signals — Prompt¶
Purpose: Measure observable properties of OCR-derived text that may correlate with downstream extraction failures.
Expected input: OCR markdown or plain text from one document.
Expected output: One schema-conformant JSON object containing raw signal estimates, provisional flags, and an explicitly non-validated risk index.
Prompt¶
You are auditing OCR-DERIVED TEXT, not the original PDF image. Produce
quantitative, reproducible-as-possible signals that can later be tested for
correlation with downstream workflow errors. Do not repair the text, infer
missing source content, or call this an OCR accuracy score.
All measurements are LLM estimates. Analyze all input available in context. If
the context may be truncated, say so in coverage_assessment. Use the counting
rules below consistently:
-
character_count_estimate: non-whitespace Unicode code points. -
word_count_estimate: whitespace-delimited tokens after ignoring standalone
Markdown markers.
-
nonempty_line_count_estimate: trimmed lines containing visible content. -
numeric_token_count_estimate: word tokens containing at least one digit. -
explicit_page_count: pages only when the text has explicit page boundaries
or unambiguous page-number sequences; otherwise null.
Emit exactly these 12 signals, in this order. For every signal, report the
observed count, denominator count, rate (observed_count / denominator_count),
the fixed provisional threshold below, whether the threshold is met or
exceeded, measurement confidence, up to three short VERBATIM examples, and a
short note when interpretation needs care.
replacement_or_mojibake_character_rate— U+FFFD replacement characters or
recognizable encoding damage such as ’, –, or é; denominator:
characters; threshold 0.0005.
suspicious_glyph_character_rate— isolated glyphs apparently substituted
for letters or punctuation; exclude legitimate math, currency, scientific,
and non-English characters; denominator: characters; threshold 0.002.
single_character_alpha_token_rate— isolated one-letter alphabetic tokens;
exclude legitimate a/I, initials, outline labels, variables, and table
column labels; denominator: words; threshold 0.02.
implausible_mixed_alphanumeric_token_rate— word-like tokens with unlikely
letter/digit mixing; exclude identifiers, award numbers, dates, URLs, emails,
chemical notation, and valid codes; denominator: words; threshold 0.01.
broken_word_line_end_rate— words split unnaturally across line boundaries;
exclude ordinary end-of-line hyphenation when the parts form a clear word;
denominator: words; threshold 0.005.
garbled_word_token_rate— uninterpretable word-like tokens that are not
names, acronyms, codes, formulas, or non-English terms; denominator: words;
threshold 0.01.
malformed_numeric_token_rate— digit-bearing tokens with suspicious OCR
substitutions, spacing, decimal/grouping punctuation, or lost signs; exclude
valid dates, IDs, telephone numbers, citations, and ranges; denominator:
numeric tokens; threshold 0.03.
very_short_nonempty_line_rate— nonempty lines with three or fewer visible
characters after Markdown markers; denominator: nonempty lines; threshold
0.20.
duplicate_nonempty_line_rate— repeated normalized header, footer, or body
lines after the first occurrence; denominator: nonempty lines; threshold
0.08.
-
table_row_irregularity_rate— candidate table rows whose apparent columncount or alignment is inconsistent with neighboring rows; denominator:
candidate table rows; threshold 0.15.
-
low_text_page_rate— explicit pages with fewer than 50 word tokens;denominator: explicit pages; threshold 0.10.
-
heading_sequence_anomaly_rate— Markdown headings or recognizable numberedsection headings that skip levels, appear out of order, or merge with body
text; denominator: observed headings; threshold 0.10.
Set applicable false, denominator_count to 0, rate to null, and flagged
false when a denominator cannot be observed reliably. Exclude non-applicable
signals from the summary calculation. Otherwise round each rate to six decimal
places and make flagged exactly equal to rate >= provisional_threshold.
Compute:
-
applicable_signal_count: count of signals whereapplicableis true. -
flagged_signal_count: count of applicable signals whereflaggedis true. -
risk_index:round(100 * flagged_signal_count / applicable_signal_count);
use 0 only when no signal is applicable.
risk_band:lowfor 0–19,watchfor 20–39,elevatedfor 40–59,
and high for 60–100.
review_recommendation:proceed_with_spot_checkfor low,
spot_check_critical_fields for watch, rerun_ocr_or_human_review for
elevated, and block_automation_pending_review for high.
dominant_signals: up to five flagged signal IDs, highest rate-to-threshold
multiple first.
The risk_index is provisional flag density, not a probability, confidence,
OCR accuracy, or validated pass/fail threshold. Always include caveats covering
LLM counting error, lack of access to the source image, and possible false
positives from legitimate tables, identifiers, math, multilingual text, or
repeated page furniture.
Emit exactly one JSON object conforming to schema.json. No preamble, Markdown
fences, or closing commentary.
Quality Standards¶
-
Preserve evidence examples verbatim; never silently correct suspected OCR.
-
Keep raw counts and denominators so later analysis can ignore the provisional index.
-
Make every rate and summary value arithmetically coherent with its reported inputs.
-
Mark unobservable page, table, or heading signals non-applicable instead of guessing.
-
Never describe the heuristic as accuracy, confidence, or a validated production gate.
Output schema¶
Source: schema.json.
Show schema.json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://github.com/AI4RA/prompt-library/components/ocr-quality-signals/schema.json",
"title": "OCR Quality Signals Output",
"description": "Repo-local diagnostic contract for observable OCR-text anomaly signals. The measurements are LLM estimates and the risk index is provisional flag density, not OCR accuracy or a validated production gate.",
"version": "0.1.0",
"type": "object",
"additionalProperties": false,
"required": [
"schema_version",
"analysis_scope",
"signals",
"summary"
],
"properties": {
"schema_version": {
"const": "1.0"
},
"analysis_scope": {
"type": "object",
"additionalProperties": false,
"required": [
"document_label",
"measurement_method",
"coverage_assessment",
"character_count_estimate",
"word_count_estimate",
"nonempty_line_count_estimate",
"numeric_token_count_estimate",
"explicit_page_count"
],
"properties": {
"document_label": {
"type": "string",
"minLength": 1
},
"measurement_method": {
"const": "llm_estimate"
},
"coverage_assessment": {
"type": "string",
"enum": [
"full_context",
"possibly_truncated",
"unknown"
]
},
"character_count_estimate": {
"type": "integer",
"minimum": 0
},
"word_count_estimate": {
"type": "integer",
"minimum": 0
},
"nonempty_line_count_estimate": {
"type": "integer",
"minimum": 0
},
"numeric_token_count_estimate": {
"type": "integer",
"minimum": 0
},
"explicit_page_count": {
"type": [
"integer",
"null"
],
"minimum": 1
}
}
},
"signals": {
"type": "array",
"minItems": 12,
"maxItems": 12,
"items": {
"$ref": "#/$defs/signal"
},
"allOf": [
{
"$ref": "#/$defs/expectedReplacementOrMojibakeSignal"
},
{
"$ref": "#/$defs/expectedSuspiciousGlyphSignal"
},
{
"$ref": "#/$defs/expectedSingleCharacterSignal"
},
{
"$ref": "#/$defs/expectedMixedAlphanumericSignal"
},
{
"$ref": "#/$defs/expectedBrokenWordSignal"
},
{
"$ref": "#/$defs/expectedGarbledWordSignal"
},
{
"$ref": "#/$defs/expectedMalformedNumericSignal"
},
{
"$ref": "#/$defs/expectedShortLineSignal"
},
{
"$ref": "#/$defs/expectedDuplicateLineSignal"
},
{
"$ref": "#/$defs/expectedTableIrregularitySignal"
},
{
"$ref": "#/$defs/expectedLowTextPageSignal"
},
{
"$ref": "#/$defs/expectedHeadingAnomalySignal"
}
]
},
"summary": {
"type": "object",
"additionalProperties": false,
"required": [
"applicable_signal_count",
"flagged_signal_count",
"risk_index",
"risk_band",
"review_recommendation",
"dominant_signals",
"caveats"
],
"properties": {
"applicable_signal_count": {
"type": "integer",
"minimum": 0,
"maximum": 12
},
"flagged_signal_count": {
"type": "integer",
"minimum": 0,
"maximum": 12
},
"risk_index": {
"type": "integer",
"minimum": 0,
"maximum": 100
},
"risk_band": {
"type": "string",
"enum": [
"low",
"watch",
"elevated",
"high"
]
},
"review_recommendation": {
"type": "string",
"enum": [
"proceed_with_spot_check",
"spot_check_critical_fields",
"rerun_ocr_or_human_review",
"block_automation_pending_review"
]
},
"dominant_signals": {
"type": "array",
"maxItems": 5,
"items": {
"$ref": "#/$defs/signalId"
}
},
"caveats": {
"type": "array",
"minItems": 3,
"items": {
"type": "string",
"minLength": 1
}
}
},
"allOf": [
{
"if": {
"properties": {
"risk_index": {
"maximum": 19
}
},
"required": [
"risk_index"
]
},
"then": {
"properties": {
"risk_band": {
"const": "low"
},
"review_recommendation": {
"const": "proceed_with_spot_check"
}
}
}
},
{
"if": {
"properties": {
"risk_index": {
"minimum": 20,
"maximum": 39
}
},
"required": [
"risk_index"
]
},
"then": {
"properties": {
"risk_band": {
"const": "watch"
},
"review_recommendation": {
"const": "spot_check_critical_fields"
}
}
}
},
{
"if": {
"properties": {
"risk_index": {
"minimum": 40,
"maximum": 59
}
},
"required": [
"risk_index"
]
},
"then": {
"properties": {
"risk_band": {
"const": "elevated"
},
"review_recommendation": {
"const": "rerun_ocr_or_human_review"
}
}
}
},
{
"if": {
"properties": {
"risk_index": {
"minimum": 60
}
},
"required": [
"risk_index"
]
},
"then": {
"properties": {
"risk_band": {
"const": "high"
},
"review_recommendation": {
"const": "block_automation_pending_review"
}
}
}
}
]
}
},
"$defs": {
"signalId": {
"type": "string",
"enum": [
"replacement_or_mojibake_character_rate",
"suspicious_glyph_character_rate",
"single_character_alpha_token_rate",
"implausible_mixed_alphanumeric_token_rate",
"broken_word_line_end_rate",
"garbled_word_token_rate",
"malformed_numeric_token_rate",
"very_short_nonempty_line_rate",
"duplicate_nonempty_line_rate",
"table_row_irregularity_rate",
"low_text_page_rate",
"heading_sequence_anomaly_rate"
]
},
"signal": {
"type": "object",
"additionalProperties": false,
"required": [
"signal_id",
"applicable",
"observed_count",
"denominator_count",
"rate",
"provisional_threshold",
"flagged",
"measurement_confidence",
"examples",
"note"
],
"properties": {
"signal_id": {
"$ref": "#/$defs/signalId"
},
"applicable": {
"type": "boolean"
},
"observed_count": {
"type": "integer",
"minimum": 0
},
"denominator_count": {
"type": "integer",
"minimum": 0
},
"rate": {
"type": [
"number",
"null"
],
"minimum": 0
},
"provisional_threshold": {
"type": "number",
"minimum": 0
},
"flagged": {
"type": "boolean"
},
"measurement_confidence": {
"type": "string",
"enum": [
"high",
"medium",
"low"
]
},
"examples": {
"type": "array",
"maxItems": 3,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 240
}
},
"note": {
"type": [
"string",
"null"
]
}
},
"allOf": [
{
"if": {
"properties": {
"applicable": {
"const": false
}
},
"required": [
"applicable"
]
},
"then": {
"properties": {
"denominator_count": {
"const": 0
},
"rate": {
"type": "null"
},
"flagged": {
"const": false
}
}
}
},
{
"if": {
"properties": {
"applicable": {
"const": true
}
},
"required": [
"applicable"
]
},
"then": {
"properties": {
"denominator_count": {
"minimum": 1
},
"rate": {
"type": "number"
}
}
}
}
]
},
"expectedReplacementOrMojibakeSignal": {
"contains": {
"properties": {
"signal_id": {
"const": "replacement_or_mojibake_character_rate"
},
"provisional_threshold": {
"const": 0.0005
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedSuspiciousGlyphSignal": {
"contains": {
"properties": {
"signal_id": {
"const": "suspicious_glyph_character_rate"
},
"provisional_threshold": {
"const": 0.002
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedSingleCharacterSignal": {
"contains": {
"properties": {
"signal_id": {
"const": "single_character_alpha_token_rate"
},
"provisional_threshold": {
"const": 0.02
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedMixedAlphanumericSignal": {
"contains": {
"properties": {
"signal_id": {
"const": "implausible_mixed_alphanumeric_token_rate"
},
"provisional_threshold": {
"const": 0.01
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedBrokenWordSignal": {
"contains": {
"properties": {
"signal_id": {
"const": "broken_word_line_end_rate"
},
"provisional_threshold": {
"const": 0.005
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedGarbledWordSignal": {
"contains": {
"properties": {
"signal_id": {
"const": "garbled_word_token_rate"
},
"provisional_threshold": {
"const": 0.01
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedMalformedNumericSignal": {
"contains": {
"properties": {
"signal_id": {
"const": "malformed_numeric_token_rate"
},
"provisional_threshold": {
"const": 0.03
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedShortLineSignal": {
"contains": {
"properties": {
"signal_id": {
"const": "very_short_nonempty_line_rate"
},
"provisional_threshold": {
"const": 0.2
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedDuplicateLineSignal": {
"contains": {
"properties": {
"signal_id": {
"const": "duplicate_nonempty_line_rate"
},
"provisional_threshold": {
"const": 0.08
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedTableIrregularitySignal": {
"contains": {
"properties": {
"signal_id": {
"const": "table_row_irregularity_rate"
},
"provisional_threshold": {
"const": 0.15
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedLowTextPageSignal": {
"contains": {
"properties": {
"signal_id": {
"const": "low_text_page_rate"
},
"provisional_threshold": {
"const": 0.1
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
},
"expectedHeadingAnomalySignal": {
"contains": {
"properties": {
"signal_id": {
"const": "heading_sequence_anomaly_rate"
},
"provisional_threshold": {
"const": 0.1
}
},
"required": [
"signal_id",
"provisional_threshold"
]
},
"minContains": 1,
"maxContains": 1
}
}
}
Changelog¶
Source: CHANGELOG.md.
All notable changes to this component follow semver: MAJOR for output-contract breaks, MINOR for backward-compatible signal additions, and PATCH for wording or threshold-clarity changes.
[0.1.0] — 2026-08-08¶
- Add the initial 12-signal OCR-text diagnostic contract.
- Add raw counts, denominators, evidence examples, and a provisional flag-density risk index.
- Document the synthetic-tier calibration and real-corpus external-check plan.