Fidelity of Statistical Reporting in 10 Years of Cyber Security User Studies

Abstract

Studies in socio-technical aspects of security often rely on user studies and statistical inferences on investigated relations to make their case. They, thereby, enable practitioners and scientists alike to judge on the validity and reliability of the research undertaken.

To ascertain this capacity, we investigated the reporting fidelity of security user studies.

Based on a systematic literature review of 114 user studies in cyber security from selected venues in the 10 years 2006–2016, we evaluated fidelity of the reporting of 1775 statistical inferences using the R package statcheck. We conducted a systematic classification of incomplete reporting, reporting inconsistencies and decision errors, leading to multinomial logistic regression (MLR) on the impact of publication venue/year as well as a comparison to a compatible field of psychology.

We found that half the cyber security user studies considered reported incomplete results, in stark difference to comparable results in a field of psychology. Our MLR on analysis outcomes yielded a slight increase of likelihood of incomplete tests over time, while SOUPS yielded a few percent greater likelihood to report statistics correctly than other venues.

In this study, we offer the first fully quantitative analysis of the state-of-play of socio-technical studies in security. While we highlight the impact and prevalence of incomplete reporting, we also offer fine-grained diagnostics and recommendations on how to respond to the situation.

Preregistered at the Open Science Framework: osf.io/549qn/.

Appendices

A Details on Qualitative Analysis

1.1 A.1 Errors Committed by statcheck

Parsing Accuracy. In all 34 error cases, statcheck parsed the PDF file correctly, and its raw test representation corresponded to the PDF. In all but two tests, statcheck recognized the test correctly. In said two cases, it mistook a non-standard-reported Shapiro-Wilk test as $\chi ^2$ test, creating two false positives. There was one case in which the statcheck computed p-value for an independent-samples t-test differed slightly from our own calculation, yet only marginally so, presumably because of a unreported Welch correction.

One-Tailed Tests. In seven cases, statcheck recognized one-tailed tests correctly. For three of those tests, the authors framed the hypotheses as one-tailed. In three other tests, the authors used one-tailed test results without declaring their use. There was one additional case in which the authors seemed to have used a one-tailed test, yet the rounding was so far off the one-tailed result that statcheck did not accept it as “valid if one-tailed” any longer. There was one test marked as “one-tail” which statcheck did not recognize as one-tailed, yet that test also suffered from rounding errors.

Dependent-Samples Tests. There were 7 papers using dependent-samples methods (such as matched-pair tests or mixed-methods regressions). We found that statcheck treated the corresponding dependent-samples statistics correctly.

Multiple Comparison Corrections. In three cases, statcheck did not recognize p-values that were correctly Bonferroni-corrected, counting as three false positives. It is an open point, however, how many paper should have employed multiple-comparison corrections, but have not done so, an analysis statcheck does not perform.

1.2 A.2 Errors Committed by Authors

Typos. We considered 6 to be typos or transcription errors ($18\%$). Another 1 error seemed to be a copy-paste error ($3\%$)

Rounding Errors. Of all 34 reported errors, we found 8 to be rounding errors ($24\%$).

Miscalculations. We found 13 cases to be erroneous calculations ($38\%$).

1.3 A.3 Composition of Incomplete p-Values

Of 1523 incomplete cases, 134 were declared “non-significant” without giving the actual p-value ($8.8\%$). Further, 6 were shown as $p > .05$. ($0.394\%$).

Of the incomplete cases, 102 were reported statistically significant at a .05 significance level ($6.7\%$).

Of the incomplete cases, 477 were reported statistically significant at a lower significance level of .01, .001, or .0001 ($31.3\%$).

Of 1523 incomplete p-values, 680 gave an exact p-value ($44.6\%$). Of those exactly reported p-values, half (367) were claimed statistically significant at a significance level of $\alpha = .05$ ($54\%$). Of those exatly reported p-values, 19 claimed an impossible p-value of $p = 0$ ($2.79\%$).

Online Supplementary Materials

We made the materials of the study (specification of the inputted SLR, included sample, contingency tables) publicly available at its Open Science Framework Repository (see Footnote 1).