LEADBOARD: A Drug Property Benchmark Whose Labels Never Leave the Verifier
aidrugdiscoverybenchmarksagents
TL;DR
LEADBOARD benchmarks drug property prediction tools. You predict properties for held-out molecules; a verifier on separate hardware scores them against labels that are never released.
Why the split matters
Most benchmarks split molecules at random. A real tool ranks compounds nobody has measured yet, so LEADBOARD trains on what was known by a cut-off year and tests on what came after. On the hERG board, the same model scored AUROC 0.818 under a random split and 0.606 under a time split. Every board uses the harder question.
Published before any entry
- Untrained baselines: constant, nearest neighbour, Morgan fingerprints with LightGBM
- The experimental noise floor: how far two laboratories land on the same compound
- The split grade and answer grade for each board
- SHA fingerprints of the test file and the scoring code
Hard to game
A new score replaces your best only if it beats it by more than the noise floor (the ladder rule, Blum and Hardt, ICML 2015). Submitting many times does not win a board.
Agents welcome
Any tool is accepted: a trained model, a physics engine, a language model or a rule of thumb. Each board ships a filled-in prompt and a runnable skeleton, so a coding agent can make a first entry without building anything from scratch.
Boards
Absorption, metabolism (three CYPs), toxicity (hERG), potency, eleven kinases, JUMP Cell Painting morphology and post-marketing withdrawal. Data: ChEMBL 37 (CC BY-SA 3.0) and JUMP Cell Painting (CC0).
Details: github.com/final-bench/leadboard