Keynote: Gene expression is tightly regulated by complexes of proteins that interpret complex sequence syntax encoded in regulatory DNA. Genetic variants influencing traits and diseases often disrupt this syntax. Several deep learning models have been developed to decipher regulatory DNA and identify functional variants. Most models use supervised learning to map sequences to cell-specific regulatory activity measured by genome-wide molecular profiling experiments. The general trend in model design is towards larger, multi-task, supervised models with expansive receptive fields. Further, emerging self-supervised DNA language models (DNALMs) promise foundational representations for probing and fine tuning on limited datasets. However, rigorous evaluation of these models against lightweight alternatives on biologically relevant tasks have been lacking. In this talk, I will demonstrate that light-weight, single-task CNNs are competitive with or significantly outperform massive supervised transformer models and fine-tuned DNALMs on critical prediction tasks. Additionally, I will show that the multi-task, supervised models learn causally inconsistent features, impairing counterfactual prediction, interpretation, and design. In contrast, our lightweight, single task models are causally consistent and provide robust, interpretable insights into regulatory syntax and genetic variation, enabling scalable novel discoveries.