Model Cards

A thorough model card includes several key sections: model description (architecture type, parameter count, training procedure, base model if fine-tuned), intended use cases and explicitly stated out-of-scope uses, training data details (sources, composition, preprocessing steps, temporal coverage), evaluation methodology and results (performance on benchmarks with results disaggregated across subgroups), known limitations and biases with concrete examples, and ethical considerations including potential harms the system could cause in deployment.

The most immediately actionable section for practitioners evaluating a model is usually the limitations disclosure. This is where the provider documents what the model does not do well — the languages where performance drops significantly, the input types that produce unreliable or hallucinated results, the demographic groups where accuracy or fairness metrics are lower, the tasks the model was not designed or evaluated for, and the known edge cases that trigger failure modes. These concrete disclosures directly inform whether the model is appropriate for your specific use case and what risk mitigations you need to build.

On open model platforms like Hugging Face, model cards are displayed prominently alongside the model's downloadable files and serve as the primary documentation that prospective users evaluate before downloading. For proprietary API-only models from major providers, equivalent information appears in technical documentation, system cards (the term Anthropic uses), or model reports, though the depth and specificity of disclosure varies significantly between providers.

Marketing materials, blog announcements, and benchmark leaderboards present models in the most favorable possible light — highlighting top-line scores on flattering benchmarks, showcasing impressive demonstrations, and emphasizing capability breakthroughs. The limitations section presents the complementary perspective: where the model struggles, what it cannot reliably do, and where its apparent performance conceals significant weaknesses. This asymmetry in presentation means the limitations section contains the information most likely to change your deployment decision, influence your architecture, or reveal risks that require explicit mitigation.

Consider concrete examples of what limitations sections reveal. A code generation model card might disclose that the model was trained primarily on Python and JavaScript, with accuracy dropping by two-thirds for Go, Rust, or less common languages. A text classification model might note that it has not been evaluated on languages other than English and the five European languages in its training data. A summarization model might acknowledge a tendency toward extractive rather than abstractive summaries, or a bias toward summarizing the beginning of long documents while losing information from later sections.

Reading limitations before committing to a model prevents the most expensive category of AI project failure: discovering a fundamental model-task mismatch after building substantial application infrastructure around the model. A team that discovers their chosen model cannot reliably handle their primary language, their domain vocabulary, or their quality requirements after three months of integration work has wasted those three months entirely. A team that reads the model card carefully first identifies the same mismatch in ten minutes and selects a more appropriate model before writing any code.

The thoroughness, specificity, and honesty of a model card signal the provider's commitment to transparency and responsible deployment. A model card that lists only positive benchmark results, describes training data in vague terms ('trained on a large and diverse corpus of internet text'), and omits the limitations section entirely should be treated with significant skepticism. The absence of limitation disclosures does not mean the model has no limitations — it means the provider chose not to document them, which is itself informative about their approach to transparency.

Strong indicators of a trustworthy and useful model card include: disaggregated evaluation results broken down by demographic group, language, input category, and difficulty level; specific descriptions of training data composition including source domains, temporal range, and any filtering or deduplication applied; named and concrete limitations with examples of the failure modes rather than generic disclaimers; clear statements distinguishing intended use cases from uses the model was not designed or validated for; and documentation of the evaluation methodology itself, allowing independent reproduction. The NIST AI RMF provides structured guidance on what responsible model documentation should include.

For models that lack model cards or have documentation that is clearly insufficient for your decision-making needs, the appropriate response is to build your own empirical evaluation. Apply model benchmarking techniques to test the model systematically on your specific tasks and representative data, paying particular attention to edge cases, subgroup performance, and the specific failure modes that would be most consequential in your application. Treat the absence of documentation not as reassurance but as an active risk factor that raises the evaluation burden on your team.

When evaluating multiple models or AI vendors for an organisational deployment, model cards provide a structured basis for comparison that goes beyond marketing claims and demo performances. Create a checklist of the information your organisation needs — supported languages, evaluated use cases, known demographic performance gaps, training data recency, and documented failure modes — and assess each candidate model's card against this checklist. Gaps in documentation are as informative as the documentation itself.

For regulated industries, model card content may directly support compliance requirements. Regulatory frameworks like the EU AI Act increasingly require documentation of training data provenance, bias evaluation results, and intended use boundaries. A model with a thorough card that addresses these requirements reduces your compliance burden; a model without adequate documentation transfers that burden entirely to your organisation, requiring you to generate the missing evidence through your own evaluation work.

Include model card quality as an explicit criterion in vendor evaluation scorecards. A provider that invests in transparent, detailed documentation is signalling a commitment to responsible deployment practices that extends beyond the document itself. Providers that decline to disclose training data characteristics, avoid disaggregated evaluation, or provide only positive performance claims are presenting an incomplete picture that increases your deployment risk.