Structured Output Parsing

Free-text responses from language models are inherently variable. The same prompt may produce 'The sentiment is positive' in one run and 'I'd characterise the overall sentiment as generally positive with some caveats' in another. Any code that parses these responses with regular expressions or string matching must account for every possible phrasing — a losing battle against a model that generates novel phrasings by design.

The fragility compounds when models are updated. A new model version — subject to training data cutoffs and retraining — may rephrase its output slightly — changing 'positive' to 'favourable' or restructuring its response format — and every downstream regex breaks silently, producing null values or misclassifications that may not be detected until they cause visible problems in production.

Structured output eliminates this entire class of failure. When the model is instructed to return `{"sentiment": "positive", "confidence": 0.87}`, the output schema is fixed regardless of how the model's natural language tendencies evolve across versions. The downstream code parses a known structure, and any deviation from that structure triggers an explicit error rather than a silent misparse.

This reliability difference is critical for production systems. A prototype that works in a Jupyter notebook with text parsing may seem fine — until it processes thousands of requests and the one-in-fifty formatting variation corrupts a database record or produces an incorrect customer-facing result.

The simplest approach is prompt-based: include a JSON example in your prompt and instruct the model to respond in the same format. This works reliably for well-known models but offers no formal guarantee that the output will be valid JSON. A malformed response — missing a closing brace, including a trailing comma, or wrapping JSON in markdown code fences — will crash your parser.

API-level enforcement provides stronger guarantees. OpenAI's structured output mode and Anthropic's structured output mode both allow you to define a JSON schema — patterns documented in API integration patterns that the model's output must conform to. The generation is constrained at the token level so that invalid structures are impossible to produce. This is the recommended approach for production systems where parsing failures have operational consequences.

For complex schemas, it can help to decompose the output into multiple calls. Rather than asking the model to produce a deeply nested JSON structure in one pass, generate the top-level structure first, then populate each section in separate calls. This reduces the cognitive load on the model and produces more reliable results for schemas with many fields or complex nesting.

Validation layers provide defence-in-depth regardless of which approach you use. Libraries like Zod (TypeScript), Pydantic (Python), or JSON Schema validators can verify the model's output against your expected schema before your application processes it. If validation fails, the system can retry the request or fall back to a default response rather than propagating malformed data.

Keep schemas as flat as possible. Deep nesting increases the chance of structural errors in prompt-based approaches and makes validation more complex. If a field can be a simple string or number rather than a nested object, prefer the simpler representation.

Use enums for categorical fields. Rather than asking the model to generate a free-text 'priority' field (which might return 'high', 'High', 'HIGH', or 'critical'), define an enum of acceptable values. API-level enforcement will constrain the output to those exact values, eliminating normalisation code downstream.

Include a field for model uncertainty where appropriate. A schema like `{"answer": string, "confidence": number, "reasoning": string}` gives downstream systems the information they need to route low-confidence outputs to human review — a pattern that aligns with human-in-the-loop design principles and prevents the system from acting on uncertain information without oversight.

Design schemas for your consumer, not your generator. The schema should match what the downstream system needs to process, not what is natural for the model to produce. If your database expects ISO date strings, specify that in the schema rather than accepting whatever date format the model defaults to. If your UI needs items sorted by priority, include a sort_order field rather than post-processing the model's arbitrary ordering.

Version your schemas explicitly, following the same discipline as prompt versioning. When downstream systems evolve and the required output format changes, include a schema version field that lets consuming code handle both old and new formats during transition periods. Without versioning, schema changes force simultaneous updates across the model prompt, the validation layer, and every consumer — a coordination burden that grows with system complexity. A version field decouples these updates and makes incremental migration practical.

JSON is the default for most API integrations, but CSV is more practical when the output feeds directly into spreadsheets or data analysis tools. Models can generate well-formed CSV with headers, and the simpler structure reduces parsing errors for tabular data. For batch processing tasks like extracting entities from a list of documents, CSV output can be imported directly into analysis tools without any transformation step.

Markdown with consistent heading structure serves as a useful middle ground for outputs that need to be both human-readable and machine-parseable. A model that consistently uses `## Section Name` headings can be parsed programmatically while still producing output that reads well in documentation contexts or when displayed in a UI.

XML remains relevant in enterprise contexts where existing systems expect it, and models can produce valid XML when given a template. However, for new systems, JSON is almost always preferable — it is lighter, more widely supported in modern toolchains, and easier for both humans and models to read and generate correctly.

The choice of format should be driven by what consumes the output. If it feeds an API, use JSON with schema enforcement. If it feeds a human workflow, use structured markdown. If it feeds a data pipeline, use CSV or JSON Lines. The format is a system design decision that should be made based on the consumer's needs, not the generator's convenience — a principle that applies across the full AI assessment architecture.