Continuation of the previous article on detecting hallucinations in code generated by AI.
In the previous article we explained how we moved from manual API logging to actually verifying against PyPI wheels. It was a big leap: we stopped relying on hand-written lists and started reading what packages publish. But having the tool is one thing, and testing it with real code, generated by real models, is another. And that’s where the surprises come.
This article is about that: the tests, the shortcomings we found, and the five bugs we had to fix to make deep mode reliable. Spoiler: the detector worked, but thought it was smarter than it was. And in a detector, that’s dangerous.
The idea: test with code that wasn’t mine
Up to that point, tests were with files I had written myself (test_debe_detectar.py, test_no_debe_detectar.py, etc.). That’s fine for validation, but it’s not the same as code generated by an AI. So I asked three models — ChatGPT, Z.ai, and ChatGPT again — to generate executable Python code with subtle hallucinations: invented packages, typos, nonexistent APIs, fake dunders, strange decorators. Real code, not a test harness with strings.
The prompt was explicit: “executable code, not a test harness, not strings containing code”. And it worked. The three files were real code, with imports and actual calls, with hallucinations hidden inside.
The results, without deep mode
First we ran them through the detector without deep (the usual heuristic mode). The results:
test_debe_detectar.py (corpus): 15 findings, 11 true positives, 3 false positives, 17 ms.
testChatGPT.py: 19 findings, 11 true positives, 8 false positives, 6.6 s.
stress_test_zia.py: 13 findings, 9 true positives, 4 false positives, 0.8 s.
codigo_generado.py: 13 findings, 11 true positives, 1 false positive, 17 ms.
The good: the detector caught invented packages, typosquatting, invented APIs, dunders and decorators. Recall was high across all four files.
The bad: false positives spiked in some. The most egregious: config.url, config.timeout, config.output_file. Why? Because the detector treated any x.y as if x were a PyPI package. And config was a local variable, not a module. But since a package called config exists on PyPI, the detector downloaded it, didn’t find url or timeout, and flagged them as hallucinations. Four false positives at once.
And another: pandas.DataFrame._name_. DataFrame exists, but _name_ is an attribute of the class, not of the pandas module. The detector looked for DataFrame._name_ in the pandas wheel, didn’t find it, and flagged it. False positive.
The results, with deep mode
This is where deep mode should shine. And it did… after fixing the bugs. The final results:
test_debe_detectar.py: 15 without deep, 14 with deep, 0 false positives, 21 ms.
testChatGPT.py: 19 without deep, 11 with deep, 0 false positives, 35 ms.
stress_test_zia.py: 13 without deep, 9 with deep, 0 false positives, 16 ms.
codigo_generado.py: 13 without deep, 12 with deep, 0 false positives, 16 ms.
Zero false positives across all four files. Deep mode eliminated FPs from real attributes (pandas.DataFrame, requests.Session, numpy.ndarray) and introduced no new ones. And with a warm cache, analyses drop to 16–35 ms.
The five shortcomings (and their fixes)
Here comes the important part: what was wrong and how we fixed it. Five bugs, all found by testing with real code.
1. The results cache wasn’t being written
The problem: deep mode analyzed the pandas wheel (11 MB, ~1000 files) on every run, because the results cache (wheel_results.json) wasn’t being written. The CLI wasn’t calling ConfigureWheelResults or FlushWheelResults. Result: each analysis took 6–11 minutes.
The fix: call ConfigureWheelResults on startup and FlushWheelResults on exit. From 11 minutes to 31 ms (with warm cache).
2. config was treated as a PyPI package
The problem: four false positives (config.url, config.timeout, config.output_file ×2). The detector verified any x.y as if x were an import.
The fix: isAttributeHallucinationDeep now receives the list of imported modules. If the first segment (config) is not in the imports, it returns CANT_VERIFY (no flag). From 4 FPs to 0.
3. The numpy wheel wasn’t being saved (or took forever)
The problem: numpy is 17 MB. The first download took 4.5 minutes on a slow connection. And sometimes the .tmp stayed at 0 B (download hung).
The fix: the version cache (see point 5) avoids querying PyPI on every analysis. And the wheel is saved once; the second time is instant. From 4.5 min to 32 ms.
4. numpy.isfinite was flagged as a hallucination (false positive)
The problem: np.isfinite exists in numpy. But the detector flagged it because isAPIHallucination was heuristic (didn’t use the wheel) and isfinite wasn’t in the list of known functions. In deep mode, only attributes were verified, not calls. So isfinite stayed with the heuristic and gave an FP.
The fix: in deep mode, also verify calls against the wheel. And if we can’t verify (stdlib with no wheel, like json), fall back to the heuristic (mark as “possible”). isfinite disappears, json.serialize_object stays.
5. The version cache: 38 seconds per PyPI query
The most serious problem: FetchWheel queried PyPI JSON on every analysis, even if the wheel was cached. And on a slow connection, PyPI JSON took 38 seconds. Multiplied by each attribute, it was unfeasible.
The fix: a version cache (version_cache.json, 1-hour TTL, configurable). Only queries PyPI once per package per hour. From 38 s to 31 ms.
What we learned
Code generated by AI is an excellent testing ground. The three files (ChatGPT, Z.ai, and the third one) uncovered bugs that our own corpus didn’t catch. Testing with external code is mandatory.
A detector that thinks it’s smarter than it is, is dangerous. The false positives from config and numpy.isfinite eroded trust. Better an honest CANT_VERIFY than a lying error.
Caching isn’t a luxury, it’s a necessity. Without version and results caching, deep mode is unfeasible on slow connections. With caching, it’s instant.
Verifying against the real wheel works… if the wheel is cached. The first download of numpy (17 MB) is slow, but only once. After that, 32 ms.
Suspicious is not the same as proven (and why the message says “possible”)
There’s a nuance worth clarifying, because it’s the difference between a reliable tool and one that gives false reassurance: detecting suspicious code is not the same as proving an AI hallucinated. And our detector, after the fixes, distinguishes three states:
NOT_FOUND: the symbol doesn’t exist in the PyPI wheel. It’s a hallucination (or an error), with objective evidence. It’s marked as an error.
CANT_VERIFY: couldn’t verify (stdlib with no wheel, local variable, nested string). We don’t know. It’s marked as “possible” (heuristic) or not marked.
CONFIRMED: the symbol exists. Not marked.
That’s why the message says “possible hallucination”, not “hallucination”. We don’t promise more than we deliver. And detection is of the error, not the author: it works the same for AI-generated code as for human code. If requests.get_json doesn’t exist, it doesn’t exist, whoever wrote it.
The challenge of division by zero (and why it’s not our goal)
Someone posed this code as a challenge:
import numpy as np
def calcular_rendimiento(datos):
media = np.mean(datos)
desviacion = np.std(datos)
return media / desviacion
The code is syntactically valid, uses real APIs, but if deviation is zero, the result is infinite. Should our detector catch that?
No. And it’s important to understand why. That’s not an AI hallucination: it’s a logic bug (division by zero). The code doesn’t invent anything; it makes a conditional error. Detecting that requires semantic analysis (data flow, conditions, types), which is the domain of pyright, mypy, pylint, or ruff, not a detector for API hallucinations.
Our goal is different: detect invented symbols (packages, functions, attributes that don’t exist). If the AI had written np.std_inventado(datos), we’d catch that (the function doesn’t exist in the wheel). But media / desviacion is correct code with a logic bug. That’s not our field. And it’s fine to say so: an honest tool recognizes its limits.
What the state of the art says (and what tools exist)
We’re not the first to do this. Recent literature confirms that code hallucinations are systematic: LLMs generate code using nonexistent library features in 8.1% to 40% of cases in NL-to-code benchmarks. And static analysis can detect between 14% and 85% of those hallucinations, with an estimated ceiling of 48.5% to 77% of what’s statically detectable. We don’t solve the problem 100%, but we attack an important and cheap part.
There are several approaches and tools:
depverify (PyPI) is the tool closest to what we do: it verifies packages and, with --symbols, attributes of already-installed packages locally. Its advantage: it’s deterministic (no LLM), and with --symbols it verifies attribute chains with getattr without executing the analyzed code. Its limits: it doesn’t download wheels (if the package isn’t installed, CANT_VERIFY), it doesn’t detect dynamic imports (importlib.import_module(“…”)), and the import-to-name alias table is hand-written and incomplete. Our approach downloads the wheel from PyPI, which lets us verify without installing anything and without depending on the server’s environment. They verify the environment; we verify PyPI.
Hallucination Inspector (2026) is a fact-checking judge for Android API migration: it uses a deterministic oracle derived from the official SDK documentation, and distinguishes “Phantom Symbols” (atomic invented symbols) from “Phantom Members” (invented methods on an object). Its architecture is static, lightweight, and language-agnostic. It’s the approach most similar to ours, but for Android and with official documentation as the oracle. We use the PyPI wheel as the oracle.
LettuceDetect (Hugging Face) is a generative detector (Qwen 2B model) that localizes hallucinated spans in text and code, with categories and subcategories. Its advantage: multi-domain coverage (code, prose, 14 languages) and fine-grained localization (spans). Its limit: it’s a probabilistic model, not deterministic; and it’s trained against a reference context (RAG), not against PyPI. It detects “this isn’t supported by the context”, not “this doesn’t exist in the package”. These are complementary approaches, not competitors.
CodeHalu (AAAI 2025) proposes a taxonomy of code hallucinations (mapping, named, resource, logic) and a benchmark of 8,883 samples to evaluate them via execution-based verification. Its strength: broad coverage and systematic evaluation. Its limit: requires executing the code (or test suites), which isn’t viable on a server for security and cost reasons. We don’t execute; we read the wheel.
In summary: the state of the art divides between deterministic static verification (like depverify, Hallucination Inspector, and our detector) and probabilistic/generative detection (LettuceDetect, uncertainty models). Neither is perfect. Static ones are cheap, deterministic, and honest, but don’t catch logic bugs or semantic errors. Generative ones cover more, but can hallucinate too (a detector that hallucinates is ironic) and don’t give objective evidence. Our detector is in the first group, and proudly so: we’d rather say “this doesn’t exist in the wheel” (evidence) than “this seems odd” (opinion).
What’s next
The detector is solid: 0 FPs across the four files, 16–35 ms with a warm cache, and verifies for real against PyPI. But there’s more to do:
Wheel download events in the progress bar (today, the first time with deep, the bar freezes while the wheel downloads).
Group related findings (today reqeusts shows up twice: as typosquatting and as package-exists; they’re the same problem).
Version-specific detection (verify against the version the project uses, not just the latest on PyPI).
Structural false negatives: session.get_json (heuristic) and create_parents (class method) slip through. Phase 3.
Pre-cache popular wheels (numpy, pandas, requests) on the server, so the first user doesn’t wait.
The previous article ended with an idea: “the linter sees what it knows; the type checker sees what’s in the signature; none knows if the method really exists”. This is the attempt to close that gap for real, with verification against PyPI, caching, and testing with AI-generated code. It’s not perfect (there are false negatives we acknowledge and the first download is slow), but it’s honest: we measure, document, and fix. And the four test files are there, with their results, so anyone can repeat the experiment.
If I can get the web on a server, it would be great to subject it to tests. It might be useful.
Reproducible materials
The .py files used in the tests are available in a public Google Drive folder:
test-files — Reproducible materials: https://drive.google.com/drive/folders/12gBRD-Ut7Gae-2_JO6baqG-VepiQvjYZ?usp=drive_link
Includes:
Own corpus: test_debe_detectar.py (recall), test_no_debe_detectar.py (precision), test_limitaciones.py (limitations), test_syntax_error.py (broken syntax).
AI-generated code: testChatGPT.py, stress_test_zia.py, codigo_generado.py.
Minimal files: test_attr_pandas.py, test_attr_requests.py, test_attr_numpy.py, test_config_bug.py, test_pandas_only.py.
Miscellaneous: robust_check.py, limpio.py.
README.md with a description of each file, expected results, and what the detector detects (and doesn’t).
How to reproduce the results: when the frontend and server are deployed on a public server, you’ll just upload each .py from the web and click “Deep Analysis”. In the meantime, the files are there to inspect, and the README.md describes what to expect from each one. The idea is that, once hosting is ready, anyone can repeat the experiment without installing anything.
Note: the first time with deep analysis, the server downloads wheels for the packages (pandas, numpy, requests…). On a slow connection, that can take minutes. Subsequent times are instant (cache).
References
Spracklen, J., Wijewickrama, R., Sakib, A.H.M.N., Maiti, A., & Viswanath, B. (2025). We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. 34th USENIX Security Symposium. arXiv:2406.10279.
Miranda-Pena, C., Reeson, A., Paris, C., Poon, J., & Kummerfeld, J.K. (2026). An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations. arXiv:2604.07755.
Tileria, M., et al. (2026). Hallucination Inspector: A Fact-Checking Judge for API Migration. arXiv:2604.20202.
Raj, A., & (coauthor). (2026). Names Can Hurt: Spotting Slopsquatting Risks Caused by Package Name Hallucinations in Local Coding LLMs. arXiv:2608.23897.
Churilov, A. (2026). The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort. arXiv:2605.17062.
Tian, Y., Yan, W., Yang, Q., et al. (2025). CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification. AAAI 2025.
KRLabsOrg. (2026). LettuceDetect-v2-Qwen-2B. Hugging Face.
depverify. (2026). depverify: Deterministic dependency and symbol verification for Python. PyPI.