Pangenomics studies intra-species genetic diversity by analyzing collections of genomes from the same species. As pangenomics scales to millions of sequences, efficient data formats become crucial to enabling future applications and ensuring efficient computational and memory performance for pangenomic analysis. Current pangenomic formats primarily store variation across genomes but fail to capture shared evolutionary and mutational histories, limiting their applicability. They also face scalability issues due to storage and computational inefficiencies. To address these limitations, we present PanMAN (Pangenome Mutation-Annotated Network), a novel pangenomic format that is the most compact, scalable, and information-rich among all variation-preserving formats. PanMAN encodes not only genome alignments and variations but also shared mutational and evolutionary histories inferred across genomes, making it the first format to unify multiple whole-genome alignment, phylogeny, and mutational histories into a single unified framework. By leveraging "evolutionary compression," PanMAN achieves 3.5X to 1391X compression over other formats (GFA, VG, GBZ, PanGraph, AGC, and tskit) across microbial datasets. To demonstrate scalability, we built the largest pangenome in terms of number of sequences —a PanMAN with 8 million SARS-CoV-2 genomes—requiring just 366MB of disk space. Using SARS-CoV-2 as a case study, we show that PanMAN offers a detailed and accurate portrayal of the pathogen's evolutionary and mutational history, facilitating the discovery of new biological insights. We also present panmanUtils, a software toolkit for constructing, analyzing, and integrating PanMANs with existing pangenomic workflows. PanMANs are poised to enhance the scale, speed, resolution, and overall scope of pangenomic analyses and data sharing.