Strategies for robust, accurate, and generalizable benchmarking of drug discovery platforms

Motivation Benchmarking is essential for the improvement and comparison of drug discovery platforms. We revised the protocols used to benchmark our Computational Analysis of Novel Drug Opportunities (CANDO) multiscale therapeutic discovery platform to bring them into strong alignment with best practices. Results CANDO ranked 7.4% and 12.1% of known drugs in the top 10 compounds for their respective diseases/indications using drug-indication mappings from the Comparative Toxicogenomics Database (CTD) and Therapeutic Targets Database (TTD), respectively. Performance was weakly positively correlated (Spearman correlation coefficient > 0.3) with the number of drugs associated with an indication and moderately correlated (coefficient > 0.5) with intra-indication chemical similarity. There was also a moderate correlation between performance on our original and new benchmarking protocols. Better performance was observed when using TTD instead of CTD when drug-indication associations appearing in both mappings were assessed. Availability and implementation CANDO is available at https://github.com/ram-compbio/CANDO. The version used in this article is available at http://compbio.buffalo.edu/data/mc_cando_benchmarking2.