Robustness Reports: An Initiative to Prevent Model Myopia by Consilience of Inductions

In most scientific articles, the primary research question is addressed using a single statistical analysis conducted by a single analyst. However, recent many-analysts projects have demonstrated that even straightforward statistical scenarios allow a broad choice of defensible modeling techniques that may well lead to qualitatively different conclusions.

In other words, current reporting practice ignores an important aspect of uncertainty (i.e., the choice of modeling technique), and this in turn leads to overconfidence and model myopia. In order to probe the fragility of the conclusions and explore the extent to which different analysts reach the same conclusion, standard reporting practice needs to change. One concrete proposal is to introduce "Robustness Reports": short, methods-oriented comments that are published alongside the original article.

I will elaborate on this proposal, with an emphasis on its advantages and its feasibility.