Skip to content

Shipped checkpoint re-evaluated on the held-out validation split (this code)

MEIDNet verified · ●○○○○ generated
Dataset Perov-5
Modalities structure, property:heat_all, property:dir_gap
Model MEIDNet early fusion + curriculum (production model) (MEIDNet 2.0.0, perovskite_abx3)
Category Representation quality
Submitted by Anand Babu (UCLouvain) on 2026-10-03
Configuration examples/perov5/meidnet.yaml
Checkpoint dual_autoencoder_clip_earlyfusion_propertyaware_2k.pth
Resources laptop CPU (AMD64)
Notes on the model the checkpoint shipped with the repository and used by the live Studio; 128-dimensional shared latent space

Results

metric value split meaning note
cosine_matched 0.771 val mean cosine similarity between the structure and property latents of the same material (1 = aligned) mean over the 3,787 validation materials
l2_matched 0.676 val mean L2 distance between those two latents (0 = identical) unit latents: sqrt(2 - 2 cos)
retrieval_top1 0.0253 val fraction of validation materials whose property latent is nearest to its own structure latent
retrieval_top5 0.0908 val the same within the five nearest
mae_heat_all 0.392 eV/atom val mean absolute error of heat_all predicted from the structure alone (physical units) predicted from the structure latent alone
r2_heat_all 0.497 val coefficient of determination of that prediction; 0 = no better than the mean, 1 = perfect
mae_dir_gap 2.79 eV val mean absolute error of dir_gap predicted from the structure alone (physical units) 96% of the rows have a direct gap of 0 eV; the structure-only predictor does not capture that
r2_dir_gap -35.8 val coefficient of determination of that prediction; 0 = no better than the mean, 1 = perfect
n_evaluated 3,787 materials val materials in the evaluation split

Validation

Evidence level: ●○○○○ generated.

Evidence:

  • checkpoints/dual_autoencoder_clip_earlyfusion_propertyaware_2k.pth
  • benchmarks/verified/shipped-checkpoint-val-split.predictions.csv

Reproduce

Configuration examples/perov5/meidnet.yaml, split val:

python scripts/benchmarks.py reproduce shipped-checkpoint-val-split

Verified here

Reproduced on 2026-10-03 with MEIDNet 2.0.0 by Anand Babu on AMD64 CPU:

metric submitted obtained here
mae_heat_all 0.392 0.392
r2_heat_all 0.497 0.497
mae_dir_gap 2.79 2.79
r2_dir_gap -35.8 -35.8
retrieval_top1 0.0253 0.0253
retrieval_top5 0.0908 0.0908
cosine_matched 0.771 0.771
n_evaluated 3,787 3,787
l2_matched 0.676 0.676

Per-material analysis

Every validation material, predicted from its structure alone; the dashed line is perfect agreement.

dir_gap: predicted from the structure vs. true (3,787 validation materials)02460246true dir_gap (eV)predicted dir_gap
heat_all: predicted from the structure vs. true (3,787 validation materials)024024true heat_all (eV/atom)predicted heat_all
cosine similarity of the two latents, per material0.650.70.750.80.850200400600800cosine(structure latent, property latent)count0.636–0.642: 10.672–0.678: 10.689–0.695: 10.695–0.701: 10.701–0.707: 20.707–0.713: 30.713–0.719: 30.719–0.725: 60.725–0.731: 50.731–0.737: 160.737–0.743: 280.743–0.748: 520.748–0.754: 1080.754–0.76: 2970.76–0.766: 7360.766–0.772: 7540.772–0.778: 5820.778–0.784: 7710.784–0.79: 2510.79–0.796: 640.796–0.802: 290.802–0.808: 290.808–0.813: 180.813–0.819: 170.819–0.825: 50.825–0.831: 30.831–0.837: 30.855–0.861: 1mean

Notes

What this code measures on the shipped checkpoint, with every prediction saved (parity plots below). The paper's representation row quotes a cosine of about 0.97; this re-run does not reach it, see the comparison on that row's page.