A Guide to Bayesian Statistics
Bayesian statistics is one of my favorite topics on this blog. I love the topic so much I wrote a book on Bayesian Statistics to help anyone learn: Bayesian Statistics the Fun Way! The following post is the original guide to Bayesian Statistics that eventually became a the book! If you enjoy the resources below, please consider ordering a copy of “Bayesian Statistics the Fun Way!” It includes reworked versions of these posts and tons of original content.
Getting Started with Bayes' Theorem and Prior Probability
This guide only assumes you have a basic familiarity with probability. If you can work out the chance of flipping a coin and getting 3 heads in a row, then you should be all set. If you do need a refresher the Khan Academy has some great lectures on the topic. There is no need for a background in Statistics. The beauty of Bayesian Statistics is it can be built up from basic probability.
What to read to get you excited?
This is a great book to get you excited about Bayesian Stats!
Nate Silver's The Signal and the Noise is a fun read and will get you very excited about Bayesian statistics. There are no strong math requirements for this book. If equations still make you a bit nervous this is a great place to build up your enthusiasm to dive in!
Recommended posts for getting started!
Bayes' Theorem is much easier to understand visually
The best place to start learning is with Bayes' Theorem. In this popular post, we'll cover the basics using Lego bricks. This allows you to use visual learning to build a deep intuition for the theorem. We won't be ignoring the mathematical detail either. By the end of this post, you'll be able to derive this formula from scratch!
"Never tell me the Odds!"
The next key topic is Bayesian Priors. Prior Probabilities allow us to model our beliefs about the world. This post focuses on C3POs classic underestimation of Han Solo's ability to navigate an asteroid field. Rather than simply dismissing C3PO, we'll see how using a Prior could help him.
Let's put Bayes' theorem to use!
Ever wonder if your friend is really allergic to gluten? We can use Bayes' Theorem and prior information to find out! We'll also discuss how being more certain than not is not all that certain.
Thinking like a Bayesian
With the basics down, it's time to start really thinking like a Bayesian! In these posts, we focus on how we can model our beliefs about the world, learn about the beliefs of others and compare hypotheses the Bayesian way!
Recommended Posts to get you thinking like a Bayesian!
That glimmer in his eye is from Bayesian Statistics!
Are you superstitious? Do you believe that a dinner novelty could have mystic powers? How much would it take to convince you? Using a classic episode of the Twilight Zone we'll learn how to answer these questions mathematically. We'll learn about Bayes' Factor and how we can use it to understand everyday reasoning.
Just say no to p-values
We further explore Bayes' Factor by building the Voight-Kampff Test from Blade Runner! In this post, we explore how to make Bayes' Factor easier to interpret using decibel-based 'evidence'. Evidence allows us to very easily interpret the results of our test. No more confusing p-values!
Recommend Reading to really understand Bayesian thinking
You could easily spend a lifetime pondering this Jaynes' masterpiece.
Probability Theory: The Logic of Science is the book to read if you're fascinated with Bayesian thinking. Though there is plenty of math, this book is about philosophy. Jaynes is probably the most radical Bayesian there is. A few pages often provide me with food for thought for months. Also, if you need insults to sling at Frequentists this is the place to get them!
The catch is that this is a very challenging book. To help guide you through it Aubrey Clayton has put together a very nice set of lectures.
Bayesian Analysis
Bayesian analysis is where we put what we've learned to practical use. In my experience, there are two major benefits to Bayesian statistics over classical statistics:
The first is that you very easy model existing information. This can very often lead to better results since the model has more to work with. Additionally, you can account for inherent bias in your analysis. By modeling the subjective parts of analysis, you can better control them.
The second is that is no "mystery meat" in your tools. All of Bayesian statistics is built from the basics of probability. If you understand the posts so far, you have the tools to build nearly everything you'll need for practical analysis.
Recommend Posts to Learn Bayesian Analysis
Understanding CDF plots is very important
Most probability books tell us how likely events are. We'll be told that a biased coin has a 0.6 probability of heads. In practice, we often don't know these probabilities. Determining whether or not the data comes from a biased coin is the real challenge. Parameter estimation deals with how we fill in these missing probabilities.
Three priors from the same data
We've seen that priors can be used for subjective information. But quite often prior beliefs are based on hard data. In this post, we see how we can use existing data to develop priors. With data backed priors we can develop much better parameter estimates.
Rejection sampling looks pretty fun!
In all of the examples we've seen so far we've had very nice probability distributions to work with. Quite often we aren't so lucky. In this post, we learn one of the simpler techniques for working difficult prior probabilities and still getting useful results.
The beauty of two competing posteriors!
Can you derive a classical t-test from scratch? My guess is not without some difficulty. One of the benefits of Bayesian statistics is everything can be built from the basics. In this post, we combine everything covered so far to build an A/B test from scratch.
Recommend Books for Bayesian Analysis
The next logic step after the posts here
If you've followed the post up to this point Doing Bayesian Data Analysis is an excellent next step. It will cover much of the material we have so far and then take you much further. The approach is mathematical, but never too challenging.
This book is full of wonderful, practical examples
For practical Bayesian statistics, nobody gets me more excited than Andrew Gelman! This is not an easy book to work through but it is an absolute gem. The text is filled with wonderful, real world example that will alway renew your love of Bayesian Statistics.
Here's a great video that shows off Gelman's enthusiasm for Bayesian Analysis:
It's never the end! Look out for updates!
There's always more to explore in Bayesian Statistics! I plan to continually update this guide as I write relevant posts or come across other amazing books. Please subscribe the to email list or follow me on twitter for updates. Feel free to leave a comment on any topics you'd like to see covered!