r/optimization • • 2d ago

Quality profiles: benchmarking optimization solvers by the quality of their solutions (paper accepted in Math. Programming Computation + Python package)

I’m one of the authors of a paper just accepted in Mathematical Programming Computation, and we’ve released a Python package that implements the method.

The problem. Performance and data profiles are the standard tools for comparing solvers on computational effort. But they implicitly assume solvers end up at comparable solutions. What if they converge to different minimizers, or some fail?

The idea. Quality profiles compare algorithms on the quality of the final objective value, so problems where solvers reach different solutions (or fail) don’t have to be discarded. They need no a priori accuracy thresholds, let you “zoom” into any accuracy range, and work for both smooth and derivative-free optimization. We also establish some theoretical properties (monotonicity, invariance under affine transformations of the objective) and analyze when each solver’s curve is independent of the others in the comparison.

The package.

pip install quality-profiles

It reproduces the results of the reference MATLAB implementation, and includes publication-ready plots, interactive figures and the data needed to reproduce the paper’s experiments.

The paper will be online after Springer’s production phase. Happy to hear feedback, questions, or ideas for what would make this more useful in practice. Contributions are welcome too.

8 Upvotes

0 comments sorted by