Power Analysis for a-priori determination of sample size

Power Analysis

Power Analysis is actually just the conversion of an equation in which the statistical test power is a central quantity. In general, this is understood to mean the conversion according to a sample size n, which is at least necessary to achieve a certain test power. The statistical test power is the complementary probability to the $$\beta$$-error. The $$\beta$$-error describes the situation in which the null hypothesis (= “there is no effect”) actually does not apply, but the null hypothesis is (incorrectly) accepted due to the data and the test results.

$$\beta$$-Error and statistical test power

In other words, we make the mistake of $$\beta$$-error when we interpret a statistical test in such a way that there is no effect - even though it is actually there. The smaller the effect, the greater the probability of this error: smaller effects are more likely to be overlooked than larger effects. Insofar as the statistical test power corresponds to 1-$$\beta$$: the greater the power, the lower the risk of overlooking an existing effect.

Power and n

In order to determine the power, assumptions must be made about the values of certain parameters as well as certain parameters must be known. A required quantity to determine the power is n. For the conversion to n a (minimum) power must be set as well as the value of the test statistic $$\tilde{\theta}$$ and the significance level $$\alpha$$. The determination of $$\tilde{\theta}$$ is done by setting a maximum effect size, which is in turn a function of $$\tilde{\theta}$$ and n [Depending on the specific approach, sometimes the degrees of freedom df of the test instead of n used.].
The power is

1-$$\beta=CDF_{\theta}\left(\left|\tilde{\theta}\right|-\theta_{1-\alpha}\right)$$

where $$CDF_{\theta}$$ is the cumulative density function of the random variable $$\theta$$.

Example

Let $$\theta$$ a standard-normal distributed test statistic. What is needed is the minimal n to discover an effect of strength $$\rho$$=0.1 at a significance level of $$\alpha$$=0.05 with a power of 1-$$\beta$$=0.8.

0.8 =$$CDF_{N\left(0,1\right)}\left(\left|\tilde{\theta}\right|-1.6449\right)$$
0.8 =$$CDF\left(0.8416\right)\Rightarrow\left|\tilde{\theta}\right|$$=0.8416+1.6449=2.4865

DASMod (Förster 2022) calculated $$\rho=\sqrt{\frac{\theta^{2}}{\theta^{2}+n}}$$. Accordingly

0.1=$$\sqrt{\frac{2.4865^{2}}{2.4865^{2}+\tilde{n}}}$$

$$\tilde{n}=\frac{\tilde{\theta}^{2}}{\tilde{\rho}^{2}}-\tilde{\theta}^{2}=\frac{2.4865^{2}}{0.1^{2}}-2.4865^{2}\approx612$$

The iterative method according to Cohen (2013) and G*Power (Faul et al. 2009), which possibly also is working with this method, both come to a similar result.
Finally, two suggestions will be given to facilitate proper interpretation. First: In this example, with $$\rho$$=0.1 (in a notation according to Cohen 2013: $$f^{2}$$=0.01) that smallest still acceptable effect size according to conventions was chosen. In order to discover larger effects with the same power, a significantly lower $$\tilde{n}$$ is required. Second, n<$$\tilde{n}$$ does not mean that effects of strength $$\rho\leq$$0.1 cannot be detected. It simply means that the probability of the $$\beta$$-error is at least 20%.

References

Cohen, Jacob (2013). Statistical Power Analysis for the Behavioral Sciences. Routledge. DOI: 10.4324/9780203771587.
Faul, Franz et al. (2009). "Statistical power analyses using G*Power 3.1: Tests for correlation and regression analses". In: Behavior Research Methods 41.4, pp. 1149-1160. DOI: 10.3758/brm.41.4.1149.
Förster, Martin (2022). DASMod. (2022.08.01.175). DOI: 10.17605/OSF.IO/JPFYG.

Seastar DASMod
Version: 2022.08.01.175