# Accelerating and Improving Approximate Bayesian Inference

Probabilistic models are a powerful way to reason about uncertainty in order to support prediction, decision making, and discovery.  Probabilistic models are specified by coming up with a joint probability distribution that connects unknown parameters, unobserved latent variables, and observed data.  We often describe this approach as "generative modeling" because it constructs a distribution over data from which we can sample.  Not only does this let us "tell a story" about what latent factors might've contributed to the structure in the data, but it also lets us apply mathematical tools such as graph theory in the form of probabilistic graphical models.

The challenge of the approach, however, is that learning (fitting parameters) and inference (fitting latent variables) both correspond to manipulating a conditional distribution --- the Bayesian posterior distribution --- that may have annoying structure.  To make predictions and decisions, we need to be able to take expectations under this distribution.  The goal of "approximate inference" in this context is to develop computational algorithms that allow one to compute such expectations.

There are two main approaches to approximate inference: 1) drawing samples from the posterior using a technique such as Markov chain Monte Carlo, and 2) approximating the intractable posterior with a simpler family, i.e., variational inference.  In the LIPS group, we study both approaches and also look for ways to apply these methods to real problems in, e.g., astronomy.

Zoltowski, David M.; Cai, Diana; Adams, Ryan P.

Advances in Neural Information Processing Systems 34 (NeurIPS), 2021.

BibTeX

Luo, Yucen; Beatson, Alex; Norouzi, Mohammad; Zhu, Jun; Duvenaud, David; Adams, Ryan P.; Chen, Ricky T. Q.

Proceedings of the Eighth International Conference on Learning Representations (ICLR), 2020.

Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.

Regier, Jeffrey; Miller, Andrew C.; Schlegel, David; Adams, Ryan P.; McAuliffe, Jon D.; Prabhat,

In: Annals of Applied Statistics, vol. 13, no. 3, pp. 1884-1926, 2019.

Miller, Andrew C.; Foti, Nicholas J.; Adams, Ryan P.

Proceedings of the 34th International Conference on Machine Learning (ICML), 2017, (arXiv:1611.06585 [stat.ML]).

Miller, Andrew C.; Foti, Nicholas J.; d'Amour, Alexander; Adams, Ryan P.

Advances in Neural Information Processing Systems (NIPS) 30, 2017, (arXiv:1705.07880 [stat.ML]).

Angelino, Elaine; Johnson, Matthew J.; Adams, Ryan P.

Patterns of Scalable Bayesian Inference Journal Article

In: Foundations and Trends in Machine Learning, vol. 9, no. 2-3, pp. 119–247, 2016, (arXiv:1602.05221 [stat.ML]).

Duvenaud, David; Maclaurin, Dougal; Adams, Ryan P.

Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2016, (arXiv:1504.01344 [stat.ML]).

Johnson, Matthew J.; Duvenaud, David; Wiltschko, Alexander B.; Datta, Sandeep Robert; Adams, Ryan P.

Advances in Neural Information Processing Systems (NIPS) 29, 2016, (arXiv:1603.06277 [stat.ML]).

Grosse, Roger B.; Ghahramani, Zoubin; Adams, Ryan P.

2016, (arXiv:1511.02543 [stat.ML]).

Rao, Vinayak; Adams, Ryan P.; Dunson, David B.

Bayesian Inference for Matérn Repulsive Processes Journal Article

In: Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 79, no. 3, pp. 877–897, 2016, (arXiv:1308.1136 [stat.ME]).

Regier, Jeffrey; Miller, Andrew C.; McAuliffe, Jon; Adams, Ryan P.; Hoffman, Matthew D.; Lang, Dustin; Schlegel, David; Prabhat,

Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015, (arXiv:1506.01351 [astro-ph.IM]).

Linderman, Scott W.; Adams, Ryan P.

2015, (arXiv:1507.03228 [stat.ML]).

Linderman, Scott W.; Johnson, Matthew J.; Adams, Ryan P.

Advances in Neural Information Processing Systems (NIPS) 28, 2015, (arXiv:1506.05843 [stat.ML]).

Nishihara, Robert; Murray, Iain; Adams, Ryan P.

Parallel MCMC with Generalized Elliptical Slice Sampling Journal Article

In: Journal of Machine Learning Research, vol. 15, no. 1, pp. 2087-2112, 2014.

Proceedings of the 31st International Conference on Machine Learning (ICML), 2014.

Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence (UAI), 2014, (arXiv:1403.5693 [stat.ML]).

Angelino, Elaine; Kohler, Eddie; Waterland, Amos; Seltzer, Margo; Adams, Ryan P.

Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence (UAI), 2014, (arXiv:1403.7265 [stat.ML]).

Lovell, Dan; Malmaud, Jonathan; Adams, Ryan P.; Mansinghka, Vikash K.

2013, (arXiv:1304.2302 [stat.ML]).

Murray, Iain; Adams, Ryan P.; MacKay, David J. C.

Elliptical Slice Sampling Conference

Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 2010, (arXiv:1001.0175 [stat.CO]).

Adams, Ryan P.; Wallach, Hanna M.; Ghahramani, Zoubin

Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 2010, (arXiv:1001.0160 [stat.ML]).

Advances in Neural Information Processing Systems (NIPS) 23, 2010, (arXiv:1006.0868 [stat.CO]).

Proceedings of the 26th International Conference on Machine Learning (ICML), 2009.

Adams, Ryan P.; Murray, Iain; MacKay, David J. C.

The Gaussian Process Density Sampler Conference

Advances in Neural Information Processing Systems 21 (NIPS), 2009.

Adams, Ryan P.; Murray, Iain; MacKay, David J. C.

Proceedings of the 26th International Conference on Machine Learning (ICML), Montréal, Canada, 2009.

Adams, Ryan P.; Murray, Iain; MacKay, David J. C.

2009, (arXiv:0912.4896 [stat.CO]).