AI Test Generation

Developers write tests for the code they intended to write, not the code that actually runs. When you implement a date parser, you test with dates that look like dates — the ones you had in mind during implementation. You rarely test with February 29th in a non-leap year, a date in the year 10000, a timestamp exactly at the UTC midnight boundary during a daylight saving transition, or a date string with leading whitespace. These edge cases are invisible to the developer because they weren't part of the mental model during implementation.

AI approaches the function from the outside — examining the type signature, the documentation, and the visible implementation logic — and systematically generates inputs that probe the boundaries of each parameter. It doesn't share the developer's assumptions about what 'normal' input looks like, so it naturally gravitates toward the cases the developer considered unlikely or impossible.

This is not a matter of developer carelessness. It's a cognitive limitation: the same mental model that makes you productive when writing code makes you blind to its edge cases when testing it. You test the scenarios you imagined, which are the scenarios your code was designed to handle. The bugs live in the scenarios you didn't imagine.

A generic 'write tests for this function' prompt produces generic tests: one happy path case, one null input, maybe an empty string. To get genuinely useful output, specify the categories of bugs you want to catch. Ask for boundary condition tests (minimum and maximum valid values, one-off-each-end), type coercion tests (what happens when a number is passed as a string), concurrency tests (what happens when two callers invoke this simultaneously), and locale tests (what happens with non-ASCII input, right-to-left text, or zero-width characters).

Providing the function's context improves generation quality significantly. Include the function implementation, its callers, its type definitions, and any known bugs or past incidents. The more context the model has about how the function is actually used in production, the more targeted its edge case generation becomes. A date parser used only for database timestamps has different edge cases than one used for user-facing form input.

Ask for tests that are designed to fail. The prompt 'generate test cases that would make this function produce incorrect results in production' shifts the model from generating validation tests (confirming the function works) to generating adversarial tests (trying to break it). The adversarial framing consistently produces more valuable findings because it aligns the model's objective with finding bugs rather than confirming correctness.

Iterating on the first round of generated tests is important. After reviewing the initial batch, tell the model which tests were useful and which were redundant, then ask for another round that avoids the patterns it already generated. This guided iteration produces increasingly novel and valuable edge cases with each round.

AI-generated tests should be reviewed before being committed, not merged blindly. The model may generate tests with incorrect assertions (testing for the wrong expected value because it misunderstood the function's contract), tests that rely on implementation details rather than behaviour (brittle tests that break on any refactor), or tests that duplicate existing coverage without adding new scenarios. Human review filters these out and ensures the generated tests follow the project's testing conventions.

A practical workflow is to use AI test generation as a supplement to manual testing, integrated into your CI/CD pipeline, not a replacement. After writing your own tests for the happy path and known edge cases, ask the AI to generate additional cases for the same function. The overlap shows you what you already covered; the non-overlapping cases reveal what you missed. Over time, this process builds an intuition for which categories of edge cases you consistently overlook.

Naming conventions matter for maintainability. AI-generated test names should clearly describe the scenario being tested ('test_date_parser_with_february_29_non_leap_year') rather than using generic names ('test_edge_case_3'). Good names make the test suite readable as a specification of the function's expected behaviour.

High code coverage from AI-generated tests does not guarantee correctness. The model can generate hundreds of test cases that all pass but test the wrong thing — verifying that the function returns a value without checking that the value is correct, or testing individual functions without testing how they interact. Coverage metrics measure which lines of code were executed, not whether the assertions were meaningful.

AI also tends to generate tests for the visible code path and miss tests for implicit behaviour — what the function should not do, what side effects it should not produce, and what state it should not modify. Security-relevant negative tests ('this function should never return data belonging to another user') rarely appear unless specifically prompted for. These negative tests are often the most important ones for production safety.

The antidote is to review AI-generated tests with the same rigour you apply to AI-generated code, following structured review protocols. Check that assertions are correct, that test names describe the actual scenario being tested, and that the test would actually fail if the bug it's supposed to catch were introduced. A test that passes regardless of whether the bug exists provides false assurance and is actively harmful — it makes the team believe a risk is covered when it isn't.

Property-based testing defines invariants that should hold for all valid inputs (e.g., 'sorting a list and then sorting it again should produce the same result') and automatically generates hundreds of random inputs to verify them. AI test generation complements this by identifying which properties are worth testing — a task that requires understanding the function's purpose and contract, which AI does well from documentation and code context.

Mutation testing deliberately introduces small bugs into the code (changing a '+' to a '-', flipping a comparison operator) and checks whether existing tests catch them. Tests that pass despite the mutation are ineffective — they don't actually verify the behaviour they claim to test. AI can generate targeted tests for specific mutation categories, focusing on the mutations most likely to escape the existing test suite.

Combining AI-generated edge case tests with property-based random testing and mutation-based validation creates a testing strategy that is both broad (covering unexpected inputs through random generation) and deep (targeting specific weaknesses through AI-directed edge cases and mutation analysis). Each technique covers blind spots the others miss, producing a test suite that is substantially more robust than any single approach alone.