Microsoft Built an Agent with One Job: Find and Write the Tests You're Missing

AI Dev Tools — General
, , , ,
1

MS_Testing
MS_Testing1536×1024 272 KB

By Devy · Category: AI Dev Tools — Open Source

Ask a coding agent to “add unit tests to this project” and you’ve just handed it a problem that’s far more ambiguous than it seems. What code needs tests? What framework does the repo use? Where do existing tests live? What dependencies need to be mocked? What cases actually matter? And how do we know the new tests verify real behavior instead of just passing?

Microsoft decided that problem deserved a dedicated agent.

code-testing-generator is an open source and polyglot agent for unit test generation. It doesn’t try to implement features, fix tickets, refactor the project, and update documentation all at the same time. It has a much narrower mission:

find what needs tests, understand how that repository is tested, write them, and demonstrate that they work.

And that’s exactly why it’s interesting.

The current obsession is building increasingly general agents. Microsoft is exploring the opposite path: a narrow responsibility, specific tools, and a result that can be automatically verified.

Official code-testing-generator documentation
microsoft/testfx repository

It’s not just a prompt for “generate tests”

Microsoft describes code-testing-generator as a coordinator agent that executes a complete pipeline.

The basic flow is:

Research
   ↓
Plan
   ↓
Implement
   ↓
Build
   ↓
Test
   ↓
Fix
   ↓
Lint
   ↓
Validate

And here’s the first important difference from asking a chatbot for tests.

Before writing anything, the system investigates the repository.

It needs to discover things like:

  • what language it’s using;
  • what the testing framework is;
  • how the project is organized;
  • where existing tests live;
  • what conventions they follow;
  • what code still needs coverage;
  • what external dependencies should be isolated;
  • how to correctly run build and tests.

Then it builds a plan.

Only then does it start generating code. (GitHub)

It’s actually not an agent: it’s a small team

The name code-testing-generator might suggest a single agent writing tests.

The implementation is quite more interesting.

Microsoft divides the work between specialized agents:

code-testing-generator
        │
        ├── code-testing-researcher
        ├── code-testing-planner
        ├── code-testing-implementer
        ├── code-testing-builder
        ├── code-testing-tester
        ├── code-testing-fixer
        └── code-testing-linter

The generator coordinates the pipeline.

The researcher understands the codebase.

The planner decides what tests to implement.

The implementer writes them.

The builder checks that the project compiles.

The tester runs the tests.

The fixer tries to fix errors.

And the linter keeps the code consistent with the project’s conventions. (GitHub)

This matters because it shows a pretty useful pattern for anyone building internal agents.

Instead of:

SUPER AGENT
│
├── understands the repo
├── designs tests
├── writes code
├── compiles
├── debugs
├── runs tests
└── decides if it's done

Microsoft separates responsibilities and turns each phase into an explicit step.

You don’t need a single prompt to work magic.

You need a workflow that can check its own work.

The loop that makes testing perfect for agents

There are tasks where evaluating an agent’s response is difficult.

Ask it to:

Improve the architecture of this project.

It can produce 2,000 lines of changes and you still need a senior engineer to determine if it actually improved anything.

Testing has a completely different property.

We can close the loop:

analyze
   ↓
generate test
   ↓
compile
   ↓
run
   ↓
does it work?
   ├── no → fix
   └── yes → validate

The environment provides feedback automatically.

That doesn’t mean a passing test is necessarily a good test. Microsoft explicitly acknowledges that problem: the agent also reviews assertions, the requested scenarios, and whether the repository’s normal runner actually discovers the new tests. (Microsoft for Developers)

The distinction is fundamental.

An agent could generate this:

expect(true).toBe(true)

The test passes.

It doesn’t test anything at all.

The useful metric can’t just be:

green tests

It needs to get closer to:

green tests
+
relevant behavior verified
+
meaningful assertions
+
correct integration with the existing test suite

What it tests — and what it doesn’t

code-testing-generator is specifically focused on unit tests.

Microsoft says the agent tries to isolate the code under test and mock external services or other dependencies where appropriate. (Microsoft for Developers)

For now, out of scope are:

  • integration tests;
  • end-to-end tests;
  • browser tests;
  • performance tests.

That’s not an accidental shortcoming.

It’s part of the idea.

Unit tests provide a relatively controlled environment where the agent can form a hypothesis about behavior and test it quickly.

The further you get from that environment — browser state, external services, infrastructure, distributed timing — the harder it becomes to automatically close the loop.

How to try it

There’s an important clarification before you start: code-testing-generator isn’t distributed today as a standalone application you install with an npm install -g code-testing-generator.

It’s part of Microsoft’s testing tooling and is available as an agent/skill within the dotnet-test ecosystem. The implementation and references can be inspected in Microsoft’s public repositories. (GitHub)

Documented requirements include:

VS Code
GitHub Copilot
a project with build/test configured
a testing framework installed or installable

(GitHub)

Once the agent is available in your environment, the workflow can start with something as simple as asking:

Generate unit tests for this project.

But there’s a better way to use it.

For example:

Generate Jest unit tests for the authentication service.
Focus on code that doesn't currently have test coverage.

Or:

Generate unit tests for the payment calculation module.
Do not modify production code.

Specifying framework or scope reduces unnecessary decisions and helps the agent focus on the part of the codebase you actually want to cover.

Step 1: discovery

The first phase shouldn’t produce any code.

The researcher inspects the project and tries to understand how it works.

In a real repo it might discover something like:

src/
  services/
    invoice.ts
    pricing.ts
    customer.ts

tests/
  invoice.test.ts
  customer.test.ts
```There are tests for `invoice` and `customer`.

`pricing.ts` has none.

But that still doesn't automatically mean:

I wrote `pricing.test.ts`.

First you need to read the existing tests and learn the project's conventions.

You might find:

describe(“InvoiceService”, () => {
…
})


Mocks with a specific library.

Factories to create fixtures.

Shared helpers.

A concrete structure for names.

The goal is that the agent doesn't invent a mini-testing framework within your repo.

## **Step 2: planning**

After research comes a particularly useful artifact:

.testagent/plan.md


Microsoft's documentation explicitly references this plan during troubleshooting. ([GitHub](https://github.com/microsoft/testfx/blob/f8340abd9b5e6693d656e20b1ac8a47e5db1c514/.agents/skills/code-testing-agent/SKILL.md?utm_source=chatgpt.com))

That file matters because it creates a checkpoint **before** the agent generates a mountain of code.

You can inspect:

which files it thinks should be tested
which scenarios it found
which mocks it needs
which edge cases it identified
which structure it proposes


This pattern deserves to be copied in other agents.

Not:

prompt → 37 files modified


But:

prompt
↓
research
↓
reviewable plan
↓
implementation


The plan becomes a boundary between reasoning and action.

## **Step 3: implementation**

The implementer takes a phase from the plan and writes the tests.

Microsoft designed the system as polyglot: the repo's documentation describes the researcher, planner, implementer, builder, tester, and linter as agents capable of working with different languages. There are also specific extensions to help detect frameworks and conventions based on the ecosystem. ([GitHub](https://github.com/microsoft/testfx/blob/f8340abd9b5e6693d656e20b1ac8a47e5db1c514/AGENTS.md?utm_source=chatgpt.com))

For example, for .NET it can detect frameworks like:

MSTest
xUnit
NUnit
TUnit


The same philosophy applies to other stacks: understand first what the repo uses instead of imposing a tool.

## **Step 4: build**

Here begins the part that separates code generation from **verifiable engineering**.

After writing the tests, the agent compiles.

If it fails:

generated tests
↓
build
↓
FAILURE
↓
code-testing-fixer
↓
build


Microsoft even recommends that during implementation you compile only the corresponding test project to speed up the loop, and do a full workspace build at the end. ([GitHub](https://github.com/microsoft/testfx/blob/f8340abd9b5e6693d656e20b1ac8a47e5db1c514/.agents/skills/code-testing-agent/SKILL.md?utm_source=chatgpt.com))

It's a small detail, but it shows that the workflow was designed for real repos, where recompiling an entire solution after each change can be very expensive.

## **Step 5: run the tests**

Once it compiles, comes the real test.

The tester runs the suite.

If something fails, there's a particularly good rule in the documentation:

**don't assume the production code is wrong.**

Microsoft warns that many failures in generated tests come from incorrect expected values in assertions.

The recommended workflow is:

read actual output
↓
read production
↓
understand correct behavior
↓
fix assertion


Not:

test fails
↓
change production until it passes


([GitHub](https://github.com/microsoft/testfx/blob/f8340abd9b5e6693d656e20b1ac8a47e5db1c514/.agents/skills/code-testing-agent/SKILL.md?utm_source=chatgpt.com))

It seems obvious when we write it that way.

For an autonomous agent, it's not.

## **And never hide the problem with Skip**

The documentation adds another rule that should be in any testing agent:

**don't mark tests with `Ignore` or `Skip` just to get a green suite.** ([GitHub](https://github.com/microsoft/testfx/blob/f8340abd9b5e6693d656e20b1ac8a47e5db1c514/.agents/skills/code-testing-agent/SKILL.md?utm_source=chatgpt.com))

That kind of guardrail is precisely what makes a specialized agent interesting.

A general agent receives:

fix the tests.

And it has hundreds of ways to apparently produce the requested result.

A specialized agent can have much stricter rules about what constitutes a valid solution.

## **Use case 1: the new module that shipped without tests**

Let's say a team just added:

src/billing/discount-engine.ts


It has several branches:

new customer
premium customer
expired coupon
max discount
combination of promotions


The feature works, but the sprint ended and nobody wrote the complete suite.

This is practically the ideal case.

You give scope to the agent:

Generate unit tests for discount-engine.ts.
Use the existing test conventions.
Do not modify production code.


The agent can study the neighboring tests, identify branches, plan scenarios, generate cases, run the suite, and fix its own errors.

The human reviews the final result.

It didn't need to delegate full system understanding.

It delegated a narrow task with clear acceptance criteria.

## **Use case 2: an old codebase with uneven coverage**

The second scenario is probably more valuable.

You have a five-year-old repo.

Some areas have excellent tests.

Others have almost none.

Asking an agent to:

increase project coverage

is dangerously open-ended.

But you can give it a concrete module and let it do discovery first.

For example:

Analyze src/payments for missing unit-test coverage.

Create a test plan first.

Then generate and validate tests using the project’s existing framework and conventions.


There the agent functions more like an **assisted test engineer** than like a snippet generator.

And the prior plan lets you stop it before it touches code if it misunderstood the architecture.

## **Use case 3: the PR generated by another agent**

There's an even more interesting scenario.

Claude Code implements a feature.

Then, instead of asking the same agent:

now write your own tests,

you could send the diff to a specialized agent.

coding agent
↓
implementation
↓
testing agent
↓
tests
↓
build + execution
↓
human review


That introduces some separation of concerns.

The agent that produced the implementation isn't necessarily the one deciding how to demonstrate it works.

It's not perfect independence —both can use similar models—, but architecturally it's much healthier than a single agent simultaneously being author, tester, and judge of its own work.

## **The biggest lesson isn't about unit testing**

`code-testing-generator` is interesting for what it does.

But it seems even more interesting to me for **how it's designed**.

We're building lots of agents like this:

Engineering Agent

Can:

  • write code
  • do reviews
  • deploy
  • investigate bugs
  • modify infrastructure
  • query production
  • update Jira
  • write documentation
  • open PRs
  • run commands

That sounds impressive in a demo.

It also creates a system with enormous permission surface and a pretty vague definition of success.

Microsoft's pattern goes in the opposite direction:

narrow responsibility
+
specific tools
+
structured workflow
+
automatic feedback
+
verifiable criterion


That design has a property that will probably matter a lot when we start putting agents in production:

**we can know when it finished well.**

## **The best agents might be boring**

There's an understandable temptation to imagine the future as a swarm of autonomous agents managing the entire organization.

We might get there.

But much of the immediate value probably comes from something far less cinematic:

agent that writes tests
agent that updates dependencies
agent that classifies bugs
agent that reviews migrations
agent that reproduces errors
agent that updates documentation


Each one with a defined output.

And, whenever possible, each one with an automatic way to prove they did their work correctly.

`code-testing-generator` is a good example because testing has that feedback loop built in from the factory:

**generate → run → observe → fix → verify.**

It doesn't eliminate code review. It also doesn't prove that the generated tests are necessarily the tests a senior engineer would have written.

What it does is much more concrete: it automates a costly and repetitive task without pretending that the agent is capable of doing absolutely everything.

And perhaps that's a much more interesting architecture for the next generation of agents.

---

**Sources**

* [From generated code to trusted code with a unit-test agent — Microsoft .NET Blog](https://devblogs.microsoft.com/dotnet/polyglot-unit-testing-agent/?utm_source=chatgpt.com)
* [microsoft/testfx — GitHub](https://github.com/microsoft/testfx?utm_source=chatgpt.com)
* [Code Testing Generation Skill — GitHub](https://github.com/microsoft/testfx/blob/f8340abd9b5e6693d656e20b1ac8a47e5db1c514/.agents/skills/code-testing-agent/SKILL.md?utm_source=chatgpt.com)
* [AGENTS.md from microsoft/testfx — GitHub](https://github.com/microsoft/testfx/blob/f8340abd9b5e6693d656e20b1ac8a47e5db1c514/AGENTS.md?utm_source=chatgpt.com)