Current browse context:
math.ST
Change to browse by:
References & Citations
Mathematics > Statistics Theory
Title: Safe Testing
(Submitted on 18 Jun 2019 (this version), latest version 10 Mar 2023 (v5))
Abstract: We present a new theory of hypothesis testing. The main concept is the S-value, a notion of evidence which, unlike p-values, allows for effortlessly combining evidence from several tests, even in the common scenario where the decision to perform a new test depends on the previous test outcome: safe tests based on S-values generally preserve Type-I error guarantees under such "optional continuation". S-values exist for completely general testing problems with composite null and alternatives. Their prime interpretation is in terms of gambling or investing, each S-value corresponding to a particular investment. Surprisingly, optimal "GROW" S-values, which lead to fastest capital growth, are fully characterized by the joint information projection (JIPr) between the set of all Bayes marginal distributions on H0 and H1. Thus, optimal S-values also have an interpretation as Bayes factors, with priors given by the JIPr. We illustrate the theory using two classical testing scenarios: the one-sample t-test and the 2x2 contingency table. In the t-test setting, GROW s-values correspond to adopting the right Haar prior on the variance, like in Jeffreys' Bayesian t-test. However, unlike Jeffreys', the "default" safe t-test puts a discrete 2-point prior on the effect size, leading to better behavior in terms of statistical power. Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, S-values and safe tests may provide a methodology acceptable to adherents of all three schools.
Submission history
From: Rianne de Heide [view email][v1] Tue, 18 Jun 2019 20:39:27 GMT (85kb,D)
[v2] Wed, 10 Jun 2020 08:38:35 GMT (96kb,D)
[v3] Mon, 6 Dec 2021 20:41:47 GMT (75kb,D)
[v4] Tue, 7 Mar 2023 15:17:24 GMT (165kb,D)
[v5] Fri, 10 Mar 2023 13:14:45 GMT (165kb,D)
Link back to: arXiv, form interface, contact.