P747 Intro. to Bayesian Statistics, Prof. Kruschke

P747
Introduction to Bayesian Statistics.

Prof. John K. Kruschke

The page below describes the 2005 version of the seminar. The 2006 version will use different materials. Much of the 2006 materials will be created by Prof. Kruschke. Here are some textbooks that you might find useful as alternative sources:

  • Albert and Rossman. Workshop Statistics: Discovery with Data, a Bayesian Approach.
    The last few chapters of this book give a wonderfully clear introduction to the basics of Bayesian statistics --- but only the basics.
  • Bolstad. Introduction to Bayesian Statistics.
    Terrific tutorial that uses only basic calculus; highly recommended. Only down side is that it does not cover numerical approximation methods.
  • Gill. Bayesian Methods: A Social and Behavioral Sciences Approach.
    and
  • Gelman, Carlin, Stern and Rubin: Bayesian Data Analysis.
    These two books offer more advanced examples of Bayesian methods, including Monte Carlo techniques.

    • This web page is at URL = http://www.indiana.edu/~jkkteach/P747_2005/

    Brief Description: This course is an introduction to Bayesian data analysis. We'll try to cover the first 11 chapters of the textbook (see below). We will do a lot of computer simulation, not just mathematical theory.
    Format: This is a one-time seminar course, which means that class periods will be devoted to impromptu lectures, discussions of assignments, and computer examples. There will be regular homeworks. Peer grading of homework is an integral part of the course, because studying the key (generated by the instructor) and grading a peer's paper will enhance learning.

    Pre-requisites. Students should have already taken a thorough course in "hypothesis testing" statistics (such as Psych P553, and preferably also P554, or equivalent). Familiarity with basic integral calculus and linear algebra will help. We will also be programming statistical analyses, so previous experience with a language such as Matlab will help.

    Time and Place: Fall semester, 2005. Wednesdays 9:00-11:00am. Psychology Building, room 111.

    Required Textbook:
    Gelman, A., Carlin, J. B., Stern, H. S., and Rubin, D. B. (2004). Bayesian Data Analysis, 2nd Edition. Boca Raton, FL: CRC Press.

    We will not be using the books listed below, but if you would like supplementary material, you might consider this less mathematical and lower level introduction:
  • Berry, D. A. (1996). Statistics: A Bayesian Perspective. Belmont, CA: Duxbury Press.
    Or, this excellent but very elementary introduction to statistics with a few final chapters on Bayesian methods:
  • Albert, J. H. and Rossman, A. J. (2001). Workshop Statistics: Discovery with Data, a Bayesian Approach. Emeryville, CA: Key College Publishing.

    • Homework: Regular homework will be assigned. Some exercises will be from the textbook. The authors have selected answers available at their website.

  • Textbook author's web page: http://www.stat.columbia.edu/~gelman/

    Software: We will do some introductory analyses in Excel or Matlab, but most analyses will be done in the language "R".

  • R software web page: http://www.r-project.org/
  • Other documents about R: http://cran.r-project.org/other-docs.html

    Oncourse: Announcements and homework keys will be posted on Oncourse (new window). Please check often.

    Schedule. As this is a one-time (and first-time) course, the schedule will emerge as we go. The following table is continually under construction.

    Week #, Date Topic, Assignment, Etc.
    1: Aug. 31
  • Ch. 1: The basics. Bayes' theorem. Goals of Bayesian modeling. Algebra of probability distributions.
  • Ch. 2: Example of Bayesian parameter estimation: Coin flip.
  • Excel demo of coin flip example.
  • Homework 1. (For this and any other assignment, only enrolled students should turn in the homework and have it graded. Others may, of course, do the homework, but please do not turn it in.)
    • 2: Sep. 7
  • Excel demo of coin flip example. (Probability density on a continuum versus probability of an interval.)
  • R version of coin flip example.
  • Ch. 2: Doing it analytically: Conjugate priors. (Binomial=>beta; Normal_Mean=>normal; Normal_Variance=>inverse_gamma; Poisson=>gamma. Prior hyperparameter values from prior knowledge.)
  • Ch. 2: Noninformative priors. (Really wide conjugate priors. Jeffrey's parameter invariance principle. Location parameter => uniform prior; scale parameter => 1/θ prior. But is "noninformative prior" an oxymoron?)
  • Homework 2.
    • 3: Sep. 14
  • Ch. 3: Models with more than one parameter. Probability distribution over joint parameter space versus marginal probability distributions over single parameters. Example: Estimating mean and variance of a normal model, using (i) noninformative prior, (ii) conjugate prior, (iii) semi-conjugte prior.
  • Example in R of Bayesian updating of mean and standard deviation in normal model.
  • Homework 3.
    • 4: Sep. 21
  • Consolidating what we've learned. Three goals of Bayesian statistics: Estimation, Prediction, Model Comparison.
  • We will pass over Ch. 4 for now, but we might return to it later if needed. (Ch. 4 and Appendix B briefly consider normal approximations to posterior distributions and convergence to normal when N is large.) Next week we will be pushing ahead into the important concepts of Ch. 5.
  • Homework 4.
    • 5: Sep. 28
  • Ch. 5: Hierarchical models.
  • Homework 5. The figure you create in the homework might look something like the one at the right (click it to enlarge).
    • 6: Oct. 5
  • Continued discussion of Ch. 5 and HW 5.
    • 7: Oct. 12
  • Ch. 6: Does the best fitting model fit the data well? Random samples from the posterior distribution of parameter values, and random samples from the model given those parameter values.
  • Homework 6. The figure at right shows what your output might look like.
    • 8: Oct. 19
  • Ch. 6, continued: Does the best fitting model fit the data well? Random samples from the posterior distribution of parameter values, and random samples from the model given those parameter values.
  • Homework 7.
    • 9: Oct. 26
  • Ch. 11: Rejection sampling.
  • Grading of HW 6 due. HW 7 due. Homework 8 assigned.
    • 10: Nov. 2
  • Ch. 11: Metropolis algorithm for sampling.
  • Grading of HW 7 due. HW 8 due. Homework 9 assigned.
    • 11: Nov. 9
  • Sampling applied to the three main goals of Bayesian modeling: Estimation, prediction and model comparison.
  • Grading of HW 8 due. HW 9 due. Homework 10 assigned.
    • 12: Nov. 16
  • Ch. 12: EM algorithm for finding modes of (marginal) distributions.
  • Ch. 18: Mixture models and application of EM algorithm.
  • Grading of HW 9 due. HW 10 due.
    • 13: Nov. 23
  • No class this week because of Thanksgiving break.
    • 14: Nov. 30
  • Bayesian versus NHST approaches to data analysis: Tests of proportions (binomial) and differences of means (t-test).
  • Grading of HW 10 due. Homework 11 assigned.
    • 15: Dec. 7
  • HW 11 due. Class will discuss key to that HW. No new HW assigned.
  • Concluding remarks and discussion.
    • Finals Week: Wednesday Dec. 14
  • Grading of HW 11 due. No final exam.