In this homework we compare null hypothesis significance testing (NHST) with Bayesian analysis. The domain of application is coin flipping, wherein we measure a two category outcome (heads or tails). We flip the coin N=20 times and get r=6 heads. We are interested in two hypotheses: One hypothesis is that the coin is fair with the probability of heads being p=.50; the other hypothesis is that the coin is biased to come up heads only p=.25. These hypotheses are formalized in more detail below.

1. NHST.

   A. Test the null hypothesis. Can the null hypothesis (p=.50) be rejected? Assume a non-directional null hypothesis, i.e., a two-tailed test, with Type I error rate set at 5%. In other words, is the probability of getting just 6 or fewer heads less than 2.5%?

   B. Determine the confidence interval. What is the range of values for p that could not be rejected? (Again, using a two-tailed test with Type I error rate of 5%.)

   C. Compute the power. For an alternative hypothesis of p=.25, what is the power of the test? That is, if the alternative hypothesis is true, what is the probability of rejecting the null hypothesis?

2. Bayesian analysis.

   A. Determine the credible interval. Suppose the prior on p is Beta(1,1). Notice that the prior is uniform, with mean p=.50, so this is one possible expression of a "null" hypothesis.
   * What is the 95% posterior credible interval? (Define the credible interval as on the bottom of p.38 of our textbook; i.e., the region of highest posterior density. One way to find this is with the "rising tide method" used in class.)
   * How does the credible interval compare with the confidence interval found in NHST (larger, smaller, overlapping)?
   * Does the credible interval contain p=.50?
   Now answer these three questions again, using a prior of Beta(10,10), which also is a reasonable expression of a "null" hypothesis.

   B. Compare null and alternative hypotheses. Consider the following two hypotheses. The null hypothesis, H0, supposes a prior of Beta(10,10). The alternative, H1, supposes a prior of Beta(5,15).
   * What is the mean of the H0 prior? What is the mean of the H1 prior?
   * With r=6 and N=20, determine the Bayes factor p(r|H1)/p(r|H0).
   * Suppose that H1 is true; i.e., coins are minted with flipping probabilities as described by H1. Generate random values of p from H1, and for each p generate a random value of r and compute the Bayes factor p(r|H1)/p(r|H0). Do that enough times so that a histogram of the Bayes factors is fairly stable. Show the histogram. What is the probability that the Bayes factor is greater than 1.0? What is the probability that the Bayes factor is greater than 20.0?
   Answer these three questions again, using H0: Beta(1,1) and H1: Beta(50,150).

Post Script: We will not pursue the topic of sample size determination in this homework, but it follows from the questions above. In NHST, sample size is determined by power: We determine the minimal N that yields the desired power. In Bayesian analysis, sample size is determined in various ways, but one approach is to determine the minimal N that yields a desired probability of the Bayes factor being greater than, say, 20.0. See: Weiss, R. (1997). Bayesian sample size calculations for hypothesis testing. The Statistician, v.46, n.2, pp.185-191.