DREAM ABSTRACTS


DREAM Prostate Cancer Challenge Session


Project Data Sphere® initiative overview
Liz Zhou, Director, US Medical Affairs, Sanofi

The Project Data Sphere® initiative (PDS) is an independent not-for-profit initiative of the Life Sciences Consortium of the CEO Roundtable on Cancer, with the vision to broadly share, integrate, and analyze historical comparator-arm cancer trial data sets from any source (academic, government, industry, etc.) with protocols, CRFs, and data descriptors, freely online at www.projectdatasphere.org. The goal is to accelerate innovation to improve cancer care.

Launched in April 2014, the PDS website provides registered users access to de-identified patient level raw data from the control arms of Phase 3 oncology clinical trials. At the time of the launch, there were 4,000 patient lives across 9 datasets from 7 industry and academic data providers. As of August 2015, the number of datasets had increased from 9 to 52, with nearly 30,000 patient lives across multiple tumor types.

Data from the PDS platform was used for the Prostate Cancer DREAM Challenge in the summer of 2015 – this challenge had the greatest level of registration of any DREAM Challenge to date. Laborious effort was put behind preparing data for the Challenge, which resulted in relatively smooth Challenge process for solvers working with clinical trial data.

PDS provides a model to share clinical trial data from oncology; the rapid growth of the number of users, downloads, and publications demonstrates a viable approach to accelerating research, including exploring innovative ways to analyze the data (e.g. crowdsourcing) without exposing patients to new clinical trials.

...............................................................................................................................

Introduction to the Prostate Cancer DREAM Challenge
James C Costello,1

1 Department of Pharmacology, University of Colorado Anschutz Medical Campus, Colorado, USA

Prostate cancer is the most common cancer among men in developed countries and ranks third in terms of mortality after lung cancer and colorectal cancer. Nearly 15% of prostate cancer patients have metastatic disease (Stage IV) at the time of diagnosis. The mainstay of treatment for metastatic disease is androgen deprivation therapy (ADT), though inevitably many patients develop resistance resulting in metastatic castrate-resistant prostate cancer (mCRPC). To gain a better understanding of mCRPC, the Prostate Cancer DREAM Challenge was developed to address two sub-challenges: 1) Predict overall survival for mCRPC patients based on clinical variables, and 2) Predict treatment discontinuation for mCRPC patients treated with docetaxel due to adverse events at early time points. The underlying data was collected from 4 separate clinical trials and annotated by Project Data Sphere, LLC. The prognostic calculators for sub-challenge 1 were scored using the integrated Area Under the Curve (iAUC) and sub-challenge 2 was scored using the Area Under the Precision Recall Curve (AUPRC). The Challenge had over 50 teams with 180 individual researchers actively participating and more than half of the teams outperformed the standard model in the field for sub-challenge 1. Sub-challenge 2 represents a novel set of results as the amount of data needed to address discontinuation due of adverse events has not been compiled until the Prostate Cancer DREAM Challenge. The results of the Prostate Cancer DREAM Challenge, specifically the prognostic calculators developed by participating teams, will be made available to clinicians to aide in patient treatment decisions. The methods developed to identify patients likely to discontinue can be used for patient selection in future clinical trial design.

...............................................................................................................................

Prostate Cancer Challenge - Best Performer SC1

Predicting patient survival in the DREAM 9.5 mCRPC challenge
Teemu D. Laajala1,2, Suleiman Khan2, Antti Airola3, Tuomas Mirtti2,4, Tapio Pahikkala3, Peddinti Gopalacharyulu2, Tero Aittokallio1,2

1 Department of Mathematics and Statistics, University of Turku, Finland
2 Institute for Molecular Medicine Finland, University of Helsinki, Finland
3 Department of Information Technology, University of Turku, Finland
4 Department of Pathology, HUSLAB, Helsinki University Hospital, Finland

We present the top performing ensemble of models for predicting patient survival in the context of metastatic castration-resistant prostate cancer (mCRPC) patients, originating from several clinical trials (subchallenge 1a of the DREAM 9.5 Prostate Cancer Challenge). By coupling unsupervised learning with survival-analysis-based supervised learning, we constructed an ensemble of batch-wise optimized penalized regression coxnet-models. The final ensemble models were simultaneously optimized for the penalized regression through L1/L2-norm parameter α along with the penalization coefficient λ. Model-based imputation of missing values as well as incorporating clinical á priori knowledge of variables is discussed, along with practical lessons learned from processing such challenging clinical data that required wide multidisciplinary expertise. Lastly, we offer our personal view of the clinical novelty of the model coefficients and its ensemble structure including interactions among the clinical variables

...............................................................................................................................

Prostate Cancer Challenge -Best Performer SC2

Docetaxel adverse event prediction: a boosting method application
Fatemeh Seyednasrollah1, 2, Mehrad Mahmoudian1, Outi Hirvonen3, 4, Sirkku Jyrkkiö3 and Laura L. Elo1, 2

1Turku Centre for Biotechnology, University of Turku and Åbo Akademi University, Turku, Finland
2 Department of Mathematics and Statistics, University of Turku, Turku, Finland
3 The Department of Oncology and Radiotherapy, Turku University Central Hospital, Turku, Finland
4 The Department of Clinical Oncology, University of Turku, Turku, Finland

The aim of this study was to predict the adverse event of docetaxel at early stage in patients with metastatic castration-resistant prostate cancer (mCRPC). More specifically, our objective was to provide more precise insights for clinicians on deciding whether to continue or discontinue docetaxel within three months of starting the treatment using baseline clinical features. To address this question, boosting method, a class of supervised machine learning techniques was utilized.

The analysis was commenced with feature selection and data preprocessing. Preliminary predictors were selected after filtering out features with insufficient clinical relevance to the study question (suggested by our clinical members) and features with high rates of missing values and/or collinearity. For selected feature categories, including lesions, prior medications and diseases, we used an arithmetic sum of presence of features in the corresponding categories. For the selected laboratory values, they were transformed, scaled or truncated based on their reference ranges and distributions. Finally, these preprocessed features were used to develop the final predictive model.

In the model building step, we focused on the R package: “gbm” (Generalized Boosted Regression Modeling). The package utilizes gradient boosting approach which iteratively tests the importance of features and aims to combine several weak models into a powerful high performance ensemble prediction. Also, the gbm package contains capacity of handling missing values and requires lower run time. In addition to this DREAM challenge, they have been shown to be successful in various practical applications.

In conclusion, the model can assist clinicians to capture suitable candidates to continue/discontinue docetaxel treatment.

...............................................................................................................................

Prostate Cancer Challenge -Best Performer SC2

Predicting discontinuation due to adverse effect in mCRPC
Yuanfang Guan

1Department of Computational Medicine and Bioinformatics, Ann Arbor, MI, 48103
2Department of Internal Medicine, Ann Arbor, MI, 48103

In the second sub-challenge of the DREAM 2015 prostate cancer challenge, participants were asked to predict which patients cannot tolerate docetaxel therapy, i.e. early adverse effect (AE) defined as termination within 91.5 days. Multi-task comparison showed that early AE is tightly connected to early death, while the patients that experienced early AE but not early death showed heterogeneous characteristics, preventing them from used as gold standard. Thus predicting adverse effect was transformed into the problem of predicting early death. I used 3 months as the cutoff, where deaths prior to 3 months were used as the gold standard positives versus the rest as negatives. This method turned out to the best performing method of this sub-challenge.

...............................................................................................................................

DREAM Olfaction Challenge Session


Introduction to the DREAM olfaction challenge
Andreas Keller1

1 Laboratory of Neurogenetics and Behavior, The Rockefeller University, New York City, USA

The complexity of olfactory stimuli as well as of the olfactory perceptual space makes the question of what determines a molecule's smell an ideal topic for the collaborative approach provided by DREAM challenges. I will briefly discuss previous attempts to predict how a molecule smells based on its physical properties and then present the psychophysical data set that was collected at Rockefeller University as the basis for the DREAM olfaction challenge. I will point out how this dataset differs from data that has been traditionally used for this type of project. I will also discuss how the influence of genetic variability between individuals and of previous experiences on odor perception complicates the stimulus-percept-correlation in olfaction.

...............................................................................................................................

DREAM Olfaction Challenge -Best Performer SC1

Predicting olfaction response for each individual
Yuanfang Guan

Department of Computational Medicine and Bioinformatics, University of Michigan, Ann Arbor

This abstract describes the method I wrote for the 2015 DREAM Olfaction Challenge- sub-challenge 1: building models to predict olfactory response for each individual. I used decision tree as a base-learner of this chemical structural data. There are two reasons that I chose decision tree: 1) the dimension of the structure data is high, which contained over 4000 parameters; decision tree helped to reduce the dimension. 2) The data matrix is sparse; a decision boundary can be put between zeros and the rest of the values. Olfactory responses reported by individuals were noisy. Thus I used the global response across all to balance the individual response, in order to 1) capture the personalized features, and b) stabilize the predictions. For example, a chemical reported to be ‘sweet’ by individual A is trusted more when other peoples also report this chemical to be ‘sweet’. Finally, I used 0.2*individual score + 0.8* global average for each chemical as predictions. But the above parameter can be rather flexible to achieve decent performance. Similar technique also resulted in one of the best performing algorithms in 2014 DREAM Broad Institute Gene Essentiality Challenge. No external data was used. This technique turns out to be the best-performing method for this sub-challenge.

...............................................................................................................................

DREAM Olfaction Challenge -Best Performer SC2

From Shape to Smell: Predicting Olfactory Perceptual Descriptors using Molecular Structural Information
Richard C. Gerkin1

School of Life Sciences, Arizona State University

The DREAM Olfaction Prediction Challenge asked participants to predict 21 different olfactory perceptual features (i.e. smell descriptors) of single molecules using a library of several thousands structural features of those molecules. One sub-challenge asked participants to make predictions for individual human subjects, and one for the mean and variance of responses across subjects. Here I describe the winning submission to the latter sub-challenge.

After using missing value imputation to fill out the structural feature dataset, I trained Random Forest Regression models (using Python's scikit-learn package) to predict mean (across subjects) responses for each of the perceptual features. I made extensive use of cross-validation to optimize these models. While I also constructed similar models to predict the variance (across subjects) of the responses, I found that prediction was improved by exploiting the relationship between the variance and the mean that was guaranteed by basic psychometric considerations. Consequently, I obtained decisive improvements in my prediction of the variance by pooling results from models trained only on the variance with a theoretically motivated non-linear transformation of results from models trained only on the mean. This technique proved decisive in constructing the winning submission for the sub-challenge.

Collaboration with other challenge participants further improved olfactory prediction by utilizing additional molecular features; this provided scientific insight into the categories and origins of features that provide useful olfactory information about molecules.

...............................................................................................................................

DREAM Olfaction Challenge: Lessons Learned

Amit Dhurandhar, Pablo Meyer, Guillermo Cecchi

IBM TJ Watson Research, NY, USA

We present the insights gained from the DREAM Olfaction challenge run earlier this year. Our sense of smell critically affects our emotions and hence decisions, making it an important part in human cognition. We report analysis of results from the challenge where only molecular structure was used to design learning algorithms that were trained to predict individual and average judgment of smell for 49 commoners (not experts) based on 21 descriptors. The results were promising in the sense that the best models achieved greater than 0.75 correlation, with linear models being competitive with the best. The challenge showed that in terms of predictability not only generalization across odors for specific individuals is possible, but that generalization across individuals for the same odors is also possible, which is highly encouraging from a science and application point of view.

...............................................................................................................................

DREAM ALS Stratification Challenge Session


The ALS Stratification Prize-Using the Power of Big Data and Crowdsourcing for Catalyzing Breakthroughs in Amyotrophic Lateral Sclerosis (ALS)

Neta Zach,1, Robert Kueffner,2, Nazem Atassi,3, Venkat Balagurusamy4, Barbara di Camillo,5, Merit Cudkowicz,3, Donna Dillenberger4, Javier Garcia-Garcia,6, Orla Hardiman,7, Bruce Hoff,8, Joshua Knight4, Melanie Leitner,9, Guang Li10, Lara Mangravite,8, Raquel Norel4, Thea Norman,8, Liuxia Wang10, Gustavo Stolovitzky4

,1 Prize4Life, Israel, ,2 Ludwig-Maximilian-University, Germany, ,3 Massachusetts General Hospital, MA, USA, 4 IBM Research, NY, USA, ,5 University of Padova, Italy, ,6 Pompeu Fabra University, Spain, ,7 Trinity College Institute of Neuroscience, Ireland, ,8 Sage Bionetworks, ,9 Biogen Idec, MA, USA, and ,10 Origent, VA, USA

Amyotrophic lateral sclerosis (ALS) is a fatal neurodegenerative disease with significant heterogeneity in its progression. In order to address this heterogeneity and spur ALS research, clinical care and drug development we need sufficient clinical data and suitable analysis approaches. To address this heterogeneity for the first time we launched the DREAM ALS Stratification Prize4Life Challenge in summer 2015, using clinical Data from the PRO-ACT database of ALS clinical trials, as well as data from National ALS registries from Italy and Ireland.

In the challenge, we asked participants to derive meaningful subgroups of ALS patients relative to disease progression and survival. The challenge drew in 75+ submissions from 31 teams. We will discuss the different approaches used by participants, as well as the baseline algorithms, and how patient classification affected performance in predicting disease outcomes. We will also discuss the different predictive features and patient subgroups that the challenge helped unveil.

...............................................................................................................................

DREAM ALS Stratification Challenge Best Performers

A Boosting Approach to Predicting ALSFRS Slope for the PRO-ACT Database
Wen-Chieh Fang1, Chen Yang1, Huan-Jui Chang2, Hsih-Te Yang1,3, Jung-Hsien Chiang1,3


1Department of Computer Science and Information Engineering, National Cheng Kung University, Taiwan
2Department of Economics, National Cheng Kung University, Taiwan
3Institute of medical informatics, National Cheng Kung University, Taiwan

Amyotrophic lateral sclerosis (ALS) is a progressive neurological disease that leads muscle weakness and gradually impacts on the functioning of the body, leading to eventual death. It greatly reduces an individual's life expectancy. Currently, experts do not know precisely what causes ALS. There is no known cure for ALS. The DREAM ALS Stratification Prize4Life Challenge is held for the purpose of enabling better understanding of patient profiles and application of personalized ALS treatments.

In our approach to DREAM ALS Challenge, we first neglect those features with high percent of missing values. For the remaining, we replace any missing value with the mean of that feature for all other cases. Meanwhile, we apply equal frequency binning that divides the response variable into three groups such that each group contains approximately same number of values. There are two kinds of features in the data set: static features and 'time-resolved' features (those with different values when time varies). For the latter, we try two designated measurements, the minimum and the maximum as additional features. Then for both two kinds of features, we apply feature selection based on information gain to select the top-six features. In order to select optimal features, we run cross validation on the feature candidates. In prediction, we apply Gradient Boosted Regression Trees (GBRT) to predict the ALSFRS slopes. GBRT computes a sequence of simple decision trees, where each successive tree is built for the prediction residuals of the preceding tree.

In this challenge, we think that the feature selection is one of the most important steps and we believe that the most appropriate features dominate the performance of the model. In the final submission round, our team attained the best performance, outperforming the methods of all other teams.

...............................................................................................................................

Predicting ALS survival through complete ranking of censored data
Yuanfang Guan1


1Department of Computational Medicine and Bioinformatics, University of Michigan, Ann Arbor, USA, 48109

Prize4life ALS sub-challenge 2 and 4 asked participants to predict survival for two separate cohorts, both as typical censored data problems. In this talk, I will review existing methods (cox-related, survival-random-forest, etc.) and their potential limitation as an incomplete ranking of training patients, and the resulting limitation in the choices of base-learners. Then, I will describe the method I wrote for this challenge, which provides a probabilistic comparison between two censored data points derived from the K-M curves (in addition to non-censored data point pairs, and censored-non-censored pairs). A complete ranking of all patients is thus given, which allows incorporation of any base-learners. This generalizes a censored data prediction problem to a standard regression problem. Finally, I will talk about one (GPR) of the many base-learners that are capable of achieving similar performance under the above modeling framework. This method turns out to be the best-performing one for both sub-challenges.

...............................................................................................................................

DREAM SMC Challenge Session


The ICGC-TCGA DREAM Somatic Mutation Calling Challenge: Combining accurate tumour genome simulation with crowd-sourcing to benchmark somatic variant detection
Joshua M. Stuart2,11,, Anna Y. Lee1,10, Kathleen E. Houlahan1,10, Adam D. Ewing2,3,10, Kyle Ellrott2,10, Yin Hu4, J. Christopher Bare4, Shadrielle Espiritu1, Vincent Huang1, Kristen Dang4, Cristian Caloian1, Takafumi N. Yamaguchi1, ICGC-TCGA DREAM Somatic Mutation Calling Challenge Participants, Michael R. Kellen4, Thea C. Norman4, Stephen H. Friend4, Justin Guinney4, Gustavo Stolovitzky5, David Haussler2, Adam A. Margolin4,7,11, Paul C. Boutros1,8,9,11

1 Informatics and Biocomputing Program; Ontario Institute for Cancer Research; Toronto, Ontario, Canada
2 Department of Biomolecular Engineering; University of California, Santa Cruz; Santa Cruz, CA, USA
3 Mater Research Institute; University of Queensland; Woolloongabba, QLD, Australia
4 Sage Bionetworks; Seattle, WA, USA
5 IBM Computational Biology Center; T.J.Watson Research Center; Yorktown Heights, NY, USA
6 Computational Biology Program; Oregon Health & Science University; Portland, OR, USA
7 Department of Biomedical Engineering; Oregon Health & Science University; Portland, OR, USA
8 Department of Medical Biophysics; University of Toronto; Toronto, Ontario, Canada
9 Department of Pharmacology & Toxicology; University of Toronto; Toronto, Ontario, Canada
10 These authors contributed equally
11 Corresponding authors

The identification of somatic mutations in cancer genomes via next-generation sequencing will transform our understanding and treatment of cancer. Unfortunately, accurate identification of somatic mutations of all types – point-mutations and structural variants – remains challenging, with many anecdotal, small-scale reports of discordance across methods. One underlying reason for this problem is the lack of robust, impartial benchmarking studies and widely-accepted gold-standards. The Cancer Genome Atlas (TCGA) and the International Cancer Genomics Consortium (ICGC) launched the ICGC-TCGA DREAM Somatic Mutation Calling Challenge: a crowd-sourcing effort to identify the best pipelines for detecting mutations in the high-throughput sequencing reads of cancer genomes (www.synapse.org/#!Synapse:syn312572).

To benchmark variant calling approaches, a novel simulator called BAMSurgeon was developed to synthesize cancer genomes in silico. The results of 248 single nucleotide variant and 204 structural variant analyses run on five synthetic tumors will be presented. Different algorithms exhibit characteristic error profiles, and, intriguingly, false positives show a trinucleotide mutation signature often reported in human tumors. Although the three simulated tumors differ in sequence contamination (deviation from normal cell sequence) and in sub-clonality, an ensemble of pipelines outperforms the best individual pipeline in all cases. We will discuss several findings from the analysis of these methods including the ability of methods to improve performance without overfitting, that SNV but not SV callers benefit from a “wisdom of the crowds” ensemble, and the first clear picture of the origins of methodological errors in SNV and SV calling. The leaderboard for this Challenge remains open and is continually attracting new entries, serving as a living-benchmark for comparing new algorithms.

...............................................................................................................................

DREAM SMC Challenge Best Performer SC3

Strategies for SNV and Indel Detection in the DREAM Challenges
Team WashU
R. Jay Mashl, Daniel C. Koboldt, Kai Ye, Li Ding

McDonnell Genome Institute, Washington University, St. Louis, Missouri 63108, USA

Identifying genomic variants is a fundamental step in the understanding of mutations associated with cancer. To this end, the ICGC-TCGA DREAM series of in silico calling challenges was designed to improve standard methods for identifying somatic mutations and rearrangements in whole-genome sequencing (WGS) data. Here, we present an assessment of metrics and other observations for distinguishing true single-nucleotide variants from false positives in the third DREAM sub-challenge. Additional filtering of putative calls using a panel of normal samples was found to reduce further the false-positive rate. A filtering suite was also developed for calling insertion and deletion events. The resulting parameters are being incorporated into the GenomeVIP cloud-enabled variant identification platform.

...............................................................................................................................

DREAM SMC Challenge Best Performer SC2, 3, & 4

novoBreak: a k-mer targeted assembly algorithm for breakpoint detection in cancer genomes
Zechen Chong1 and Ken Chen1

1Department of Bioinformatics and Computational Biology, the University of Texas MD Anderson Cancer Center

Somatic structural variations (SVs) are major driving forces for tumor development and progression. Sporadic and recurrent chromosomal aberrations have been observed in most cancer types, including breast, lung, brain, leukemia, pancreatic and prostate cancers. The advent of high-throughput next generation sequencing (NGS) technologies has made it possible to perform genome-wide detection of SVs at base pair resolution. However, current sequencing-based computational methods are limited in sensitivity and comprehensiveness due to the challenges of acquiring sufficient information to characterize different types of SVs. Here, we present novoBreak, a novel k-mer targeted local assembly algorithm that discovers somatic and germline structural variation breakpoints in whole genome sequencing data. NovoBreak can directly identify breakpoints from clusters of reads that share a set of k-mers uniquely present in a subject genome (e.g., a tumor genome) but not in the human reference genome or any control data (e.g., a matched normal genome). In synthetic data from the ICGC-TCGA DREAM 8.5 Somatic Mutation Calling Challenge and real data from a cancer cell-line, novoBreak consistently outperformed existing algorithms due mainly to more effective utilization of reads spanning breakpoints. NovoBreak also demonstrated great sensitivity in identifying short INDELs and gene fusions. The wider application of novoBreak is expected to reveal comprehensive structural landscape that can be linked to novel mechanistic signatures in cancer genomes.

...............................................................................................................................

Lessons from the SMC-DNA IS Challenges & Looking Forward
Paul C. Boutros1,2, Adam A. Margolin3, Kyle Ellrott3, Quaid D. Morris2, Paul Spellman3, David Wedge4, Peter Van Loo5, Gustavo Stolovitzky6, Joshua D. Stuart7

1 Ontario Institute for Cancer Research, Toronto, Canada
2 University of Toronto, Toronto, Canada
3 Oregon Health & Sciences University, Portland, Oregon, USA
4 Wellcome Trust Sanger Institute, Hinxton, UK
5 Crick Research Institute, London, UK
6 IBM Research, NY, NY, USA
7 University of California, Santa Cruz, California USA

We present a summary of the key lessons learnt through the in silico challenges of the ICGC-TCGA DREAM Somatic Mutation Calling Challenge (SMC-DNA), with a particular focus on the challenges in simulating, scoring and integrating structural variant (SV) predictions. We discuss the remarkable failure of ensemble models to improve upon SV prediction, and note the significant differences to other types of genomic data. Finally we outline the future of the SMC series of DREAM Challenges, giving the official launch of a tumour subclonality reconstruction challenge (SMC-Het) and talking about the progress towards an RNA-Seq Challenge (SMC-RNA).

...............................................................................................................................

DREAM Drug Combination Challenge Session


Crowdsourcing combinatorial therapies: The AZ-Sanger DREAM synergy prediction challenge
Menden MP1,*, Wang D2,*, Chaibub Neto E3, Ghazoui Z2, Jang IS3, Giovanni Di Veroli4, Gustavo Stolovitzky5, Dry JR2,#, Guinney J3,#, Saez-Rodriguez J1,6,#

1European Molecular Biology Laboratory – European Bioinformatics Institute
2Oncology Innovative Medicines, AstraZeneca
3Sage Bionetworks
4Early Clinical Development – Innovative Medicine, AstraZeneca
5IBM Research
6RWTH Aachen University Hospital, Joint Research Center for Computational Biomedicine (JRC-Combine)
* First authors; # Corresponding authors (alphabetically ordered)

In the last 20 years targeted therapies and personalized treatments have been the most promising assets to treat cancer. However, the success is often lessened by secondary resistance. As a solution to increase the therapeutic search space and to overcome resistance, combinations of drugs are currently extensively investigated. A major current limitation is the lack of effective strategies to search the virtually intractable combinatorial space. To tackle this issue and advance the development of combinatorial therapies DREAM, Sage Bionetworks and AstraZeneca are jointly hosting a drug combination challenge, which is open to the scientific community to participate.

The challenge aims to address two specific questions: (i) to predict synergies in cancer cell lines from molecular data, and (ii) to identify biomarkers that discriminate between synergistic and non-synergistic behaviors. Towards this goal, AstraZeneca provides a combinational cell line screening, which comprehends a total of ~11.5k experimental tested drug combinations, as well as all mono therapies as baselines. The previous DREAM drug combination challenge focused on ~100 experimental tested combinations and explored synergy signatures based on before and after treatment gene expression. Besides the smaller scale, those signatures are not practical/ethical to retrieve from patients. In addition our proposed challenge focus on revealing mechanistic insights from any biomarker selection and model build. By providing a framework where any group in the world can participate, and whereby methods can be evaluated in an unbiased and double-blinded way, we aim to move forward in this complex problem of discovering novel and effective drug combinations.

Here we present results from the first challenge round and will show glimpses of knowledge gained in this crowed-sourcing drug combination challenge.

...............................................................................................................................

Preventing data-leakage in leaderboard evaluations: the Ladder and LadderBoot algorithms
Elias Chaibub Neto1

1Sage Bionetworks

Over-fitting is a common issue in machine learning challenges. Because participants rely on the public leaderboard to evaluate and refine their models, there is always the danger they might start to over-fit their models to the holdout data supporting the leaderboard. Standard remedies to this problem include limiting the number of allowed submissions per participant and rounding the released public scores. Recently, Hardt and Blum (2015) proposed the Ladder algorithm, which reduces over-fitting by preventing the participant from exploiting minor fluctuations in public leaderboard scores during their model refinement activities. Mechanistically, the Ladder only releases the actual (rounded) score of a new submission if the score presents a statistically significant improvement over the previously best submission of the participant. If not, the Ladder releases the score of the best submission so far.

In this talk, we present evaluations of the Ladder algorithm under two adversarial attacks. Both are inspired by Freedman's paradox, where the selection of the features entering a multiple regression model is guided by the public leaderboard. In the first attack, we simply select the top features according to the public leaderboard scores of the univariate regression models. Our experiments show the effectiveness of the Ladder algorithm in this context. Our second attack, on the other hand, is based on a more aggressive step-forward variation of this first attack, and can lead to severe over-fitting. This attack explores the fact that the Ladder leaks too much information about the holdout data when it releases the public leaderboard score of the best model so far. To circumvent this problem, we propose a variation of the Ladder mechanism, called LadderBoot algorithm, which releases a bootstrapped estimate of the public leaderboard score, instead of the actual rounded score. In our experiments, the LadderBoot mechanism tended to compare favorably to the Ladder.

Reference. Hardt and Blum (2015) The Ladder: a reliable leaderboard for machine learning competitions. arXiv:1502.04585, 2015.

...............................................................................................................................

OTHERS


The DREAM Challenges channel: open science publishing for all participating DREAMers
Michael Markie1

1F1000Research, London, UK

F1000Research is an Open Science publishing platform that offers the immediate publication of posters, slides and articles with no editorial bias. All published articles benefit from a collaborative, transparent peer review process and the inclusion of all source data and code. F1000Research partners with the International Society for Computational Biology through the publication of the ISCB Community Journal, and have recently built the DREAM Challenges channel, a central venue to publish peer-reviewed method articles from participants of DREAM challenges. Developments in science most often build upon previous findings, insights, and data, so it is important to make this information accessible to enable easy reuse. Both DREAM challenges and F1000Research have core values that are underpinned by open, reproducible science and this collaboration aspires to advance the questions posed in DREAM Challenges through the open sharing of data-analyses and computational methods. Participants of the recently concluded Prostate Cancer challenge will be the first DREAMers to make use of the channel. Participants will take advantage of a dynamic, post publication peer review model and have the opportunity to work with expert reviewers from the biomedical research community to help improve and refine their methods after the challenge has completed. In addition to encouraging participants to publish their findings, the DREAM Challenges channel is also open to the wider DREAM community to publish research around challenge topics and the theory of how the challenges work.

The aim of the collaboration is to make further progress beyond the challenges, help answer important biomedical and biological questions and pave the way to improving clinical practice, such as the prognostic calculators developed for the recent Prostate Cancer DREAM Challenge.

...............................................................................................................................

DREAM Hackathon


The DREAM Challenges Hackathon: mining big data from a Parkinson’s Disease mobile research study
Brian Bot1, Chris Bare1, and DREAM Challenges Hackathon Team

Sage Bionetworks 1, University of Rochester, DREAM Challenges

The DREAM Challenges will be sponsoring a Hackathon on Sunday and Monday evenings, based on data from the mPower study. mPower, is an App-based clinical study, focused on Parkinson Disease (PD), developed by Sage Bionetworks and the University of Rochester. As of August 2015, mPower enrolled 18,000 participants – the largest PD study ever. Typically symptoms in PD patients are evaluated and recorded twice a year, when the patient goes to the doctor. Between these doctor’s visits, a patient’s disease status is left unmonitored, and important decisions about interventions are often not made in a timely manner. mPower is built to allow patients to continuously track their PD signs and symptoms, generating data on the patient’s voice (pitch and tremor), balance, walking and finger speed and dexterity. The data being collected from this study will be made available for this year’s DREAM Conference Hackathon’s participants.

The goal of this hackathon is to use the mPower data to generate insights about PD. The data is collected using Sage Bionetworks Bridge Server, and Sage scientists have already proved that valuable information can be extracted from the mPower study. However there is a lot more work to be done. In this hackathon participants will bring their expertise and creative energies to develop ideas, models or proposals on how best to glean insights from this powerful study.


top