Data-driven whispers from the dark side of the Universe

Data-driven whispers from the dark side of the Universe: a data-driven Bayesian reconstruction of how dark energy evolves from the latest data.
Illustration of supernovae, baryon acoustic oscillations and the cosmic microwave background probing dark energy across cosmic time
My original artwork created for the journal cover.
Bayesian inferenceMCMCData-driven reconstruction

Data-driven whispers from the dark side of the Universe

Dark energy is the most elusive and mysterious component of the Universe. Over the past decades, theorists have proposed dozens of models to describe it, yet none has managed to beat the seemingly unbreakable standard cosmological model. Recently, the community began to ask a different question: what if we let the data decide how dark energy evolves in time? At the same time, the first data release of the Dark Energy Spectroscopic Instrument (DESI) brought unexpected hope for evidence of a non-standard dark energy. So I asked myself: how do these new results change the non-parametric reconstruction of dark energy's history? And what do they tell us about the underlying theory?

The problem

Since the accelerated expansion of the Universe was discovered in 1998, the standard model of cosmology has assumed that dark energy is a constant. In 2024 the DESI survey published new measurements that hinted otherwise, and the question became one of the most debated in the field.

Most analyses test this by choosing a specific formula for how dark energy could change, and fitting its parameters. The answer then depends on the formula. I wanted an approach closer to how a data scientist would treat an unknown function: let the data determine the shape, and measure how confident we can be in every part of it.

What I built

A flexible, non-parametric model. I described the dark energy density as a smooth curve defined by a small number of control points (a cubic spline). The value at each control point is a free parameter learned from the data. I chose to model the quantity that the measurements are most directly sensitive to, which made the model both easier to constrain and easier to interpret.

Data fusion across heterogeneous sources. I combined three very different datasets in a single likelihood: the cosmic microwave background (the early Universe), distance measurements from DESI galaxy clustering, and supernovae, using two alternative catalogues one at a time. I wrote the likelihood for the DESI data and integrated everything into an existing scientific modelling pipeline.

Bayesian inference at scale. The full model has 14 parameters with strong correlations between them. I explored the posterior with an ensemble MCMC sampler, and cross-checked it with a second, independent algorithm (Metropolis–Hastings) plus a direct optimiser for the best fit. Every result comes with a full uncertainty band, not just a point estimate.

Validation and robustness. Before trusting the results, I compared the new pipeline with published results obtained with independent, widely tested codes, and found consistent answers. I then stress-tested the conclusions in two ways:

  • Overfitting: a flexible model can fit noise. I compared an 8-point and a 4-point version of the model. The simpler one gave smoother curves with the same overall features, but the strength of the signal depends on how flexible the model is, so I report both.
  • Data quality: an open question was whether a small subset of nearby supernovae carried a calibration offset. I applied the proposed correction to only those points and measured how much the conclusions moved.

Results

Reconstructed deviation of dark energy from a constant, and the derived equation of state, for two supernova catalogues
The reconstructed history of dark energy. Each panel shows the median of the model's estimates (solid line), the single best fit (dashed line) and the 68% and 95% uncertainty bands, using one of the two supernova catalogues (left and right). On the right, the blue curves in the zoomed-in boxes show the result after correcting the nearby supernovae (see below). The flat black line is what a constant dark energy would look like. Top: how far dark energy deviates from a constant at each epoch (later times on the left). Bottom: the derived quantity w, explained below.
  • A mild preference for change. The data favour a dark energy that evolves over time, but the evidence is moderate: about 2.4σ with one supernova catalogue and 1.3σ with the other. That is weaker than claimed with the simpler, fixed-formula approach. Choosing a more flexible model gives a more honest picture of the uncertainty.
  • Localising the signal. Because the model estimates each epoch separately, I could see where the deviation comes from. For the stronger result, the evidence is driven almost entirely by a single point in the recent Universe. That is the kind of insight a single summary number would hide.
  • A hint of unusual behaviour. From the same curve I derived how dark energy’s pressure relates to its density, a number called w. For a constant dark energy, w is exactly −1. In every case I studied, the data suggest that w moves from one side of −1 to the other over cosmic time, although the uncertainties are still large. The simplest theories, built on a single ingredient, cannot do this. If confirmed, it would point to richer physics, such as several interacting components or a change to the law of gravity.
Corner plot of joint parameter constraints before and after correcting the low-redshift supernova offset
Testing a suspected data issue. Joint and marginal posterior distributions for key parameters, before (green) and after (blue) correcting a small group of nearby supernovae for a suspected calibration offset, with the DESI data (filled contours) and without them (dashed lines). The bottom-right panel is the parameter that drives the signal; the dotted line is the "constant dark energy" value.
  • A small data issue can create a big signal. When combined with the DESI data, correcting the suspected offset in only the nearest supernovae was enough to remove the tension completely: the key parameter moved from 3σ away from the standard value to fully consistent with it. The correction also shifted other parameters, showing the trade-off it introduces. Without the DESI data, the same correction did not clearly help. This is a concrete example of why auditing input data matters as much as the model.

Techniques

Modelling

Non-parametric regression with splines, choosing an interpretable target variable, controlling model complexity.

Inference

Bayesian analysis, MCMC in high dimensions, convergence and sampler cross-checks, uncertainty quantification.

Data

Combining heterogeneous datasets in one likelihood, testing a suspected systematic error.

Engineering

Extending a large scientific codebase, writing custom likelihoods, validating against reference tools.

Read the full paper
← All projects