[Continuation of the previous article on detecting hallucinations in AI-generated code.]( Tu linter no ve esto: alucinaciones de API y paquetes que no existen )
In the previous article I told how the idea was born: a `requests.get_json()` that never existed, which passed tests, linter and type checker, and would only have failed in production. The tool I built then relied on a handcrafted registry of valid APIs: a map of library → known functions, and a simple rule: if the library is in the registry and the function is not, it’s a hallucination. It worked for `requests`, `json` and `os`, and detected nonexistent packages by querying PyPI.
But the registry had an obvious limitation: it only judged what it knew. And as soon as it went beyond `requests`, `json` and `os`, the tool fell silent. Not because there were no hallucinations, but because it had nothing to compare against. On that ceiling is where I’ve been thinking these past weeks. And this is what came next.
As is evident, this work does not start from scratch. Recent research confirms that library hallucinations are systematic (5.2% of the packages mentioned by commercial models don’t exist, and up to 21.7% in open-source models; likewise, static analysis has a ceiling of 48.5% to 77% of what it can detect, and that verification against real documentation significantly reduces false positives. Our project addresses some of those limitations (invented packages, APIs of real packages verified against wheels) and leaves others documented (APIs of nonexistent packages, advanced slopsquatting). This continuation tells what we’ve done and what remains.
How to verify more rigorously?
The question that changed everything was: what if instead of manually maintaining a list of valid functions, we read what the package actually contains?
PyPI doesn’t just serve packages: it serves wheels, which are ZIP files with the code and, very importantly, the type stubs (`.pyi`) when the package includes them. If you download the wheel and open it, you can search for whether `class DataFrame` exists in `pandas` without installing anything and without executing any of the user’s code.
That’s exactly what we did. And it changes the game: you no longer depend on someone maintaining a list of functions by hand. You depend on what the package actually publishes on PyPI. Coverage stops being handcrafted and becomes that of the ecosystem.
The cache, or how not to punish PyPI on each analysis.
The first practical problem was obvious: if each analysis queries PyPI for each import, on a slow network like mine, the analysis takes forever. And if PyPI goes down, the detector gets stuck waiting. So we set up a two-level cache (memory + disk) with TTL:
- `exists` (does the package exist on PyPI?): 7 days.
- Reputation metadata (first release, maintainers): 30 days.
- Network errors: 10 minutes (to avoid retrying in a loop if the network is bad).
And a design decision that already came from the previous article but now becomes central: fail-open . If we can’t verify (timeout, PyPI down), we don’t report the package as nonexistent . Better a one-off false negative than a false positive that erodes trust.
The result is noticeable: the second run on the same file goes from ~50 seconds to 17 milliseconds . And all that without touching the normal flow: if you don’t configure the cache, the detector works as before.
Canonicalization, stdlib and the false positive that taught us
With the cache came the fine details. PyPI normalizes names (`PIL` is installed as `Pillow`, `sklearn` as `scikit-learn`, `cv2` as `opencv-python`), and imports bring submodules (`tensorflow.keras`, `os.path`). If you don’t canonicalize, the cache doesn’t hit and weird false positives appear.
One of the most instructive: `imp` (a stdlib module, deprecated) was marked as typosquatting of `pip` . The edit distance was 1, and the heuristic didn’t know that `imp` is stdlib. The lesson: the lists of stdlib and the lists of PyPI packages must be kept in sync , or the detector starts inventing problems where there are none. Today `imp`, `distutils` and their ilk are recognized as stdlib and not flagged.
Verifying attributes for real: `pandas.DataFrame` is no longer a false positive
Here’s the biggest change from the previous article. Before, `api-hallucination` was heuristic : it looked at suspicious names, it didn’t verify. And that generated false positives like `pandas.DataFrame`, which does exist but wasn’t in the registry.
Now there’s a deep mode (`deep`) that, when you activate it:
1. Queries PyPI for the package.
2. Downloads the wheel (once, cached by version).
3. Opens the ZIP and searches for the attribute in all the `.pyi` and `.py` files of the package.
4. Returns one of three states: confirmed , not found or not verifiable .
The result, measured against the same test file: 15 findings without deep mode, 14 with deep mode . The difference is exactly the false positive of `pandas.DataFrame`, which disappears. And what’s more important: `pandas.funcion_inventada` is indeed detected, with evidence that it’s not in the real wheel.
That’s what the handcrafted registry couldn’t give. And we did it without installing Python on the server, without executing user code, and without adding third-party dependencies : just HTTP, ZIP and a bit of regex.
IP-based rate limiting, or how to open it without getting your server knocked down
Deep mode downloads wheels. That costs bandwidth and time. Before opening it to the public, there had to be a limit. So we added IP-based rate limiting : 3 deep analyses per day per IP, with daily reset.
Three details that seemed important to us:
- Localhost is exempt. So development tests don’t stop by hitting the limit. Otherwise, you block yourself on your own machine.
- IPs can be simulated in development with `X-Forwarded-For` and a `DETECTOR_DEV=1` variable. This way you can test the limit without deploying.
- In production, `DETECTOR_TRUST_PROXY=1` to read the real IP behind a proxy. Otherwise, all users would share the proxy’s IP.
The limit is simple, but it’s the difference between opening deep mode with network and opening it without network.
The four test files, and what they actually say
To measure without self-deception, we wrote four files:
| File | What it measures | Result |
|---|---|---|
test_debe_detectar.py |
Recall (detecting the bad) | 14–15 findings, almost all correct |
test_no_debe_detectar.py |
Precision (not flagging the good) | 0 findings  |
test_limitaciones.py |
Known false negatives/positives | 1 finding (a TP by heuristic) |
test_syntax_error.py |
Broken syntax | 0, “file is clean”  |
The good news: precision is perfect on clean code. The honest truth: there’s a UX bug on broken syntax (the detector says “clean” when the file doesn’t even compile) and there are structural false negatives (dynamic imports, `from-import` of names, nonexistent submodules) that require deeper analysis. We have them documented, not hidden.
So the project is no longer “an API registry”. It’s a detector that verifies against PyPI for real , with caching, opt-in deep mode and usage control. From this milestone, what remains to review:
- Wheel download events in the progress bar (today, with deep mode, the bar freezes while downloading the wheel).
- `–deep` in the CLI , so the terminal has the same power as the web.
- Report `syntax-error` as a finding, instead of saying “file is clean”.
- Evidence and statuses in each finding (`confirmed`, `probable`, `not verifiable`).
- pyright/mypy as a local option only , never on the server (slow, requires installation, and without stubs it flags nothing).
And there’s one more thing we’d like to tell you about. When we have hosting (complicated from my country under siege) we’ll upload a web that already allows you to run analyses online. It’s a tool that’s already built and working locally: you upload a .py, see the progress in real time (with progress bar via SSE) and get the findings, including deep mode with verification against PyPI wheels. But for that web to be sustainable and not crash at the first sign of trouble, we need a contribution to the tool’s development. It’s not a payment to use the service: it’s a way for the project to be able to pay for hosting, bandwidth for wheel downloads, and maintenance time. The idea is that whoever wants to use deep mode intensively can contribute, and whoever just wants to try basic analysis can do so without barriers. The code will remain open; the web with hosting will be how the project sustains itself. We’ll tell you more about it when it’s ready.
The previous article ended with an idea: “the linter sees what it knows; the type checker sees what’s in the signature; none know if the method actually exists”. This is the attempt to close that gap. Not with a bigger list, but by reading what the package publishes . It’s still not perfect: there are false negatives we acknowledge and a pending syntax bug. But the difference between “there’s no hallucination” and “we couldn’t verify” is no longer hidden: we measure it, document it, and try to reduce it.
If you try deep mode and find a false positive or one that slips through, I’d like to know. This project gets corrected by what others find.
References
1. Spracklen, J., Wijewickrama, R., Sakib, A.H.M.N., Maiti, A., & Viswanath, B. (2025). *We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs*. 34th USENIX Security Symposium. arXiv:2406.10279 .
2. Miranda-Pena, C., Reeson, A., Paris, C., Poon, J., & Kummerfeld, J.K. (2026). *An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations*. arXiv:2604.07755 .
3. Tileria, M., et al. (2026). *Hallucination Inspector: A Fact-Checking Judge for API Migration*. arXiv:2604.20202 .
4. Raj, A., & (coauthor). (2026). *Names Can Hurt: Spotting Slopsquatting Risks Caused by Package Name Hallucinations in Local Coding LLMs*. arXiv:2608.23897 .
5. Churilov, A. (2026). *The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort*. arXiv:2605.17062 .