2022
Benchmarking & evaluation
When is progress real?
Systematic comparisons of unsupervised network representations.
The research question
Do new graph embedding methods consistently improve downstream performance when compared under a common evaluation protocol?
The work
Our TMLR paper, “Benchmarking and Analysing Unsupervised Network Representation Learning and the Illusion of Progress,” systematically compares network representation learning methods. It examines how evaluation choices, dataset properties, and baselines affect conclusions about progress.
Research artifacts
The open-source NRL_Benchmark repository accompanies this research. The KDD 2021 Deep Learning Day presentation is noted beneath the TMLR paper. The earlier consolidation preprint is part of the same work and is not listed separately.
Why this matters
For me, rigorous evaluation is part of representation learning itself: it helps separate useful modeling ideas from effects of experimental choices. This perspective also motivates my emerging interest in evaluating generalization in language models.