• Julio Saez-Rodriguez

The AI promise of powerful solutions to solve biomedical and healthcare related problems needs to be accompanied by a transparent evaluation and proof of reproducibility of the corresponding algorithms. The evaluation for algorithms in machine learning (ML) is typically done by assessing their performance on prediction tasks. The application of this benchmarking paradigm to foundation models (FM) is not straightforward. FMs are typically trained using self-supervised methods that don’t need labeled ground truth data, and therefore the models are embodiments of the phenomena that gave rise to the data. Usually, the training of FMs is followed by finetuning the models for specific tasks that are trained using more traditional ML methodologies, and can be benchmarked as such ML models. However, such an assessment would fail to elucidate if possible failures of the model reside in the FM or in the refined model. In this era of FMs, there is a need to rethink rigorous evaluation, both in what it means to validate and in how we validate. One strategy could be a concerted effort of the communities developing and using them, whereby crowdsourcing microtasks that test the limits of these models from every possible perspective within the domain of competence of the models, and in a continuous manner. The aim of this DREAM session at ISMB is to explore together this strategy and define as a community a roadmap to move forward with such a critically needed benchmark of foundation models.

• John Moult, University of Maryland, US

Current AI methods have transformed aspects computational structural biology,
delivering protein structures with accuracy in many cases rivalling that of experiment
and huge improvements in the accuracy of protein assemblies. But so far results from
CASP show AI has not overtaken the traditional (and quite poor) methods for modeling
RNA structure, are only beginning to rival traditional (and inadequate) ligand docking
methods, and struggle with alternative conformations. What is now limiting progress and
what might be done about it? One obvious explanation is lack of training data. Can
approaches developed for other AI areas, such as bootstrapped training data, help here
too? Can alternative structure representations provide a more effective training regime?
Can AI approaches be productively combined with traditional methods, as recent CASP
results suggest? What does that landscape of possibilities look like?
The success of AI in structural biology has led to an avalanche of papers proposing new
AI approaches. These papers contain a wealth of interesting and stimulating ideas, but
the amount of information is overwhelming and its usually unclear how reliably methods
have been benchmarked. Consequently, it is very difficult for investigators to assimilate
the results and to know how much weight to give each, so slowing progress. CASP and
other rigorous benchmarking will eventually sort all this out, but in the current hyper-
output situation that’s too slow to be fully effective. Initial experiments suggest AI
methods can critically assess these papers, in terms of both evaluating the results and
potentially in constructing a dynamic ontology of methods, how they have been applied,
and with what success. But this AI approach is in its infancy. How can this new area of
critical assessment be most effectively advanced? Can AI-driven critical assessment be
extended to other areas of biological research?

• Maria Brbic

TBA

• Pablo Meyer-Rojas

The AI Alliance is focused on fostering an open community and enabling developers and researchers to accelerate responsible innovation in AI while ensuring scientific rigor, trust, safety, security, diversity and economic competitiveness. We bring together a critical mass of compute, data, tools, and talent to accelerate and advocate for open innovation in AI. Together with DREAM challenges we aim to create a world-class research community that harnesses the potential of AI foundation models, transforms the field of drug discovery, and accelerates scientific progress by driving interdisciplinary collaboration on AI-powered drug discovery projects in the open. IBM Research biomedical foundation model (BMFM) technologies leverage multi-modal data of different types, including drug-like small molecules and proteins (covering a total of more than a billion molecules), as well as DNA and single-cell RNA sequence.

• Elizabeth Fahsbender, Chan Zuckerberg Initiative
• Katrina Kalantar, Chan Zuckerberg Initiative

Realizing the vision of AI-powered Virtual Cells demands robust and biologically meaningful benchmarks that ensure models are reliable, reproducible, and relevant. This talk will present key insights from a recent community workshop convened by the Chan Zuckerberg Initiative, which identified critical challenges to benchmarking in this space—ranging from data heterogeneity and systemic bias to evaluation metric gaps and ecosystem fragmentation. We will highlight a set of community-driven recommendations and describe how CZI is beginning to address these through targeted investments in infrastructure, high-quality data generation, and community coordination. These efforts aim to catalyze progress toward a trustworthy benchmarking ecosystem that can accelerate foundational model development for cell biology.

• Anshul Kundaje

Keynote: Gene expression is tightly regulated by complexes of proteins that interpret complex sequence syntax encoded in regulatory DNA. Genetic variants influencing traits and diseases often disrupt this syntax. Several deep learning models have been developed to decipher regulatory DNA and identify functional variants. Most models use supervised learning to map sequences to cell-specific regulatory activity measured by genome-wide molecular profiling experiments. The general trend in model design is towards larger, multi-task, supervised models with expansive receptive fields. Further, emerging self-supervised DNA language models (DNALMs) promise foundational representations for probing and fine tuning on limited datasets. However, rigorous  evaluation of these models against lightweight alternatives on biologically relevant tasks have been lacking. In this talk, I will demonstrate that light-weight, single-task CNNs are competitive with or significantly outperform massive supervised transformer models and fine-tuned DNALMs on critical prediction tasks. Additionally, I will show that the multi-task, supervised models learn causally inconsistent features, impairing counterfactual prediction, interpretation, and design. In contrast, our lightweight, single task models are causally consistent and provide robust, interpretable insights into regulatory syntax and genetic variation, enabling scalable novel discoveries.

• Justin Guinney

Inputs into cancer prognostic models are primarily structured data such as demographic and clinicopathological features, and lack richer and temporal context often found in unstructured clinical notes. We hypothesize that creating a temporal clinical patient note from structured data that preserves longitudinal and clinical contextual information, and coupling it with a large language models (LLM) that is trained to prognosticate overall survival (OS), may improve model accuracy with an interpretable embedding space. In this study, we benchmark different LLMs and fine-tuning strategies to develop optimal models for predicting overall survival from time of metastasis in a large cohort of de-identified patients with metastatic breast cancer.

• Gustavo Stolovitzky

We will collectively conduct a small scale community experiment to simulate a large scale crowdsourcing initiative to benchmark foundation models.

• Bo Wang, Anshul Kundaje, Justin Guinney, Katrina Kalantar, Maria Brbic, Luca Foschini

Speakers will give their opinions and about best practices to evaluate foundation models in biomedicine, and engage in conversation with attendees.