## Article contents

# Optimal experimental design: Formulations and computations

Published online by Cambridge University Press:
**04 September 2024**

## Abstract

Questions of ‘how best to acquire data’ are essential to modelling and prediction in the natural and social sciences, engineering applications, and beyond. Optimal experimental design (OED) formalizes these questions and creates computational methods to answer them. This article presents a systematic survey of modern OED, from its foundations in classical design theory to current research involving OED for complex models. We begin by reviewing criteria used to formulate an OED problem and thus to encode the goal of performing an experiment. We emphasize the flexibility of the Bayesian and decision-theoretic approach, which encompasses information-based criteria that are well-suited to nonlinear and non-Gaussian statistical models. We then discuss methods for estimating or bounding the values of these design criteria; this endeavour can be quite challenging due to strong nonlinearities, high parameter dimension, large per-sample costs, or settings where the model is implicit. A complementary set of computational issues involves optimization methods used to find a design; we discuss such methods in the discrete (combinatorial) setting of observation selection and in settings where an exact design can be continuously parametrized. Finally we present emerging methods for sequential OED that build non-myopic design policies, rather than explicit designs; these methods naturally adapt to the outcomes of past experiments in proposing new experiments, while seeking coordination among all experiments to be performed. Throughout, we highlight important open questions and challenges.

- Type
- Research Article
- Information
- Creative Commons
- This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
- Copyright
- © The Author(s), 2024. Published by Cambridge University Press

## References

*Combin. Comput. Geom.*52, 1–30.Google Scholar

*Information Science for Materials Discovery and Design*(Lookman, T., Alexander, F. and Rajan, K., eds), Springer, pp. 13–44.CrossRefGoogle Scholar

*Inverse Problems*37, art. 043001.CrossRefGoogle Scholar

*SIAM J. Sci. Comput.*40, A2956–A2985.CrossRefGoogle Scholar

*Bayesian Anal.*11, 671–695.CrossRefGoogle Scholar

*SIAM J. Sci. Comput.*36, A2122–A2148.CrossRefGoogle Scholar

*SIAM J. Sci. Comput.*38, A243–A272.CrossRefGoogle Scholar

*SIAM/ASA J. Uncertain. Quantif.*9, 163–184.CrossRefGoogle Scholar

*Proceedings of the 34th International Conference on Machine Learning (ICML 2017)*, Vol. 70 of Proceedings of Machine Learning Research, PMLR, pp. 126–135.Google Scholar

*Proceedings of the 47th Annual ACM Symposium on Theory of Computing (STOC 2015)*, ACM, pp. 237–245.Google Scholar

*Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics*, Vol. 108 of Proceedings of Machine Learning Research, PMLR, pp. 3241–3251.Google Scholar

*Proceedings of the 38th AAAI Conference on Artificial Intelligence*(Wooldridge, M., Dy, J. and Natarajan, S., eds), AAAI Press, pp. 20311–20319.Google Scholar

*Math. Finance*9, 203–228.CrossRefGoogle Scholar

*Stochastic Simulation: Algorithms and Analysis*, Springer.CrossRefGoogle Scholar

*Optimum Experimental Designs, with SAS*, Oxford University Press.CrossRefGoogle Scholar

*Inverse Problems*34, art. 095009.CrossRefGoogle Scholar

*Ann. Math. Statist.*40, 1570–1602.CrossRefGoogle Scholar

*Optim. Engrg*5, 101–122.CrossRefGoogle Scholar

*SIAM J. Optim.*13, 889–903.CrossRefGoogle Scholar

*Found. Trends Mach. Learn.*6, 145–373.CrossRefGoogle Scholar

*J. Mach. Learn. Res.*3, 1–48.Google Scholar

*J. Comput. Phys.*503, art. 112844.CrossRefGoogle Scholar

*Found. Comput. Mat*

*h.*Available at doi:10.1007/s10208-023-09630-x.CrossRefGoogle Scholar

*Advances in Neural Information Processing Systems 16*, MIT Press, pp. 201–208.Google Scholar

*Proceedings of the 41st Annual ACM Symposium on Theory of Computing (STOC 2009)*, ACM, pp. 255–262.Google Scholar

*SIAM/ASA J. Uncertain. Quantif.*4, 739–766.CrossRefGoogle Scholar

*Int. J. Numer. Methods Engrg*121, 3482–3503.CrossRefGoogle Scholar

*Comput. Methods Appl. Mech. Engrg*334, 523–553.CrossRefGoogle Scholar

*Proceedings of the 35th International Conference on Machine Learning (ICML 2018)*, Vol. 80 of Proceedings of Machine Learning Research, PMLR, pp. 531–540.Google Scholar

*Lectures on Modern Convex Optimization*, SIAM.CrossRefGoogle Scholar

*SIAM Rev.*57, 483–531.CrossRefGoogle Scholar

*Statistical Decision Theory and Bayesian Analysis*, Springer Series in Statistics, Springer.CrossRefGoogle Scholar

*Test*3, 5–124.CrossRefGoogle Scholar

*An Introduction to Optimal Designs for Social and Biomedical Research*, Wiley.CrossRefGoogle Scholar

*Ann*.

*Statist.*7, 686–690.CrossRefGoogle Scholar

*Bayesian Adaptive Methods for Clinical Trials*, Chapman & Hall/CRC.CrossRefGoogle Scholar

*Dynamic Programming and Optimal Control*, Vol. 1, Athena Scientific.Google Scholar

*Stochastic Recursive Algorithms for Optimization*, Springer.CrossRefGoogle Scholar

*Proceedings of the 34th International Conference on Machine Learning (ICML 2017)*(Precup, D. and Teh, Y. W., eds), Vol. 70 of Proceedings of Machine Learning Research, PMLR, pp. 498–507.Google Scholar

*Proceedings of the 2nd Berkeley Symposium on Mathematical Statistics and Probability*, University of California Press, pp. 93–102.CrossRefGoogle Scholar

*Ann*.

*Math. Statist.*24, 265–272.CrossRefGoogle Scholar

*SIAM/ASA J. Uncertain. Quantif.*9, 564–592.CrossRefGoogle Scholar

*Proceedings of the 39th International Conference on Machine Learning (ICML 2022)*(Chaudhuri, K. et al., eds), Vol. 162 of Proceedings of Machine Learning Research, PMLR, pp. 2107–2128.Google Scholar

*Ann. Math. Statist.*25, 737–744.CrossRefGoogle Scholar

*Foundations of Modern Statistics*(Belomestny, D. et al., eds), Vol. 425 of Springer Proceedings in Mathematics & Statistics, Springer, pp. 355–380.CrossRefGoogle Scholar

*Sbornik Math.*196, art. 309.CrossRefGoogle Scholar

*Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics*(Storkey, A. and Perez-Cruz, F., eds), Vol. 84 of Proceedings of Machine Learning Research, PMLR, pp. 890–899.Google Scholar

*Inverse Problems*34, art. 115007.CrossRefGoogle Scholar

*Ann. Eugen.*9, 353–399.CrossRefGoogle Scholar

*Sankhyā*4, 337–372.Google Scholar

*Qual. Engrg*5, 321–330.CrossRefGoogle Scholar

*Convex Optimization*, Cambridge University Press.CrossRefGoogle Scholar

*J. Comput. Graph. Statist.*12, 566–584.CrossRefGoogle Scholar

*Proceedings of the Twenty-Fifth Annual ACM-SIAM Symposium on Discrete Algorithms*, SIAM, pp. 1433–1452.Google Scholar

*SIAM J. Sci. Comput.*35, A2494–A2523.CrossRefGoogle Scholar

*Acta Numer.*7, 1–49.CrossRefGoogle Scholar

*SIAM J. Comput.*40, 1740–1766.CrossRefGoogle Scholar

*Advances in Neural Information Processing Systems 32*(Wallach, H. et al., eds), Curran Associates, pp. 11461–11472.Google Scholar

*Proceedings of the 35th International Conference on Machine Learning (ICML 2018)*, Vol. 80 of Proceedings of Machine Learning Research, PMLR, pp. 698–706.Google Scholar

*J. Mach. Learn. Res.*20, 551–588.Google Scholar

*Ann. Statist.*44, 1165–1192.CrossRefGoogle Scholar

*Biometrics*54, 964–975.CrossRefGoogle ScholarPubMed

*Comput. Methods Appl. Mech. Engrg*363, art. 112909.CrossRefGoogle Scholar

*Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics*, Vol. 108 of Proceedings of Machine Learning Research, PMLR, pp. 4226–4235.Google Scholar

*Statist. Probab. Lett.*2, 223–227.CrossRefGoogle Scholar

*Neural Comput.*22, 887–905.CrossRefGoogle ScholarPubMed

*Ann. Statist.*12, 283–300.CrossRefGoogle Scholar

*J. Statist. Plann. Infer.*21, 191–208.CrossRefGoogle Scholar

*Statist. Sci.*10, 273–304.CrossRefGoogle Scholar

*European J. Oper. Res.*220, 684–694.CrossRefGoogle Scholar

*SIAM J. Comput.*43, 1831–1879.CrossRefGoogle Scholar

*9th International Conference on Learning Representations (ICLR 2021)*. Available at https://openreview.net/forum?id=AY8zfZm0tDd.Google Scholar

*Learning and Intelligent Optimization*, Vol. 7997 of Lecture Notes in Computer Science, Springer, pp. 59–69.CrossRefGoogle Scholar

*J. Agric. Biol. Environ. Statist.*8, 184–195.CrossRefGoogle Scholar

*International Encyclopedia of the Social & Behavioral Sciences*(Smelser, N. J. and Baltes, P. B., eds), Science Direct, pp. 5075–5081.CrossRefGoogle Scholar

*J. Artificial Intelligence Res.*4, 129–145.CrossRefGoogle Scholar

*Discrete Appl. Math.*7, 251–274.CrossRefGoogle Scholar

*Introduction to Derivative-Free Optimization*, SIAM.CrossRefGoogle Scholar

*Technometrics*22, 315–324.CrossRefGoogle Scholar

*Statist. Sci.*28, 424–446.CrossRefGoogle Scholar

*Elements of Information Theory*, second edition, Wiley.Google Scholar

*Amer. J. Phys.*14, 1–13.CrossRefGoogle Scholar

*Amer. Math. Monthly*43, 180.CrossRefGoogle Scholar

*Bernoulli*28, 2788–2815.CrossRefGoogle Scholar

*J. Comput. Phys.*485, art. 112103.CrossRefGoogle Scholar

*J. Comput. Phys.*304, 109–137.CrossRefGoogle Scholar

*Inverse Problems*30, art. 114015.CrossRefGoogle Scholar

*Advances in Neural Information Processing Systems 36*(Oh, A. et al., eds), Curran Associates, pp. 16957–16990.Google Scholar

*Proceedings of the 28th International Conference on Machine Learning (ICML 2011)*(Getoor, L. and Scheffer, T., eds), ACM, pp. 1057–1064.Google Scholar

*Theoret. Comput. Sci.*412, 1767–1781.CrossRefGoogle Scholar

*Handbook of Uncertainty Quantification*(Ghanem, R., Higdon, D. and Owhadi, H., eds), Springer, pp. 311–428.CrossRefGoogle Scholar

*Inverse Problems*29, art. 095017.CrossRefGoogle Scholar

*Acta Numer.*22, 133–288.CrossRefGoogle Scholar

*Commun. Pure Appl. Math.*36, 183–212.CrossRefGoogle Scholar

*J. Amer. Statist. Assoc.*103, 288–298.CrossRefGoogle Scholar

*Comput. Statist. Data Anal.*57, 320–335.CrossRefGoogle Scholar

*J. Comput. Graph. Statist.*23, 3–24.CrossRefGoogle Scholar

*SIAM J. Appl. Math.*19, 215–220.CrossRefGoogle Scholar

*Inverse Problems*39, art. 125008.CrossRefGoogle Scholar

*J. Comput. Phys.*231, 7815–7850.CrossRefGoogle Scholar

*Ann. Math. Statist.*23, 255–262.CrossRefGoogle Scholar

*Statist. Comput.*32, art. 82.CrossRefGoogle Scholar

*J. Funct. Anal.*286, art. 110338.CrossRefGoogle Scholar

*J. Math. Anal. Appl.*18, 262–268.CrossRefGoogle Scholar

*Linear Algebra Appl.*1, 33–38.CrossRefGoogle Scholar

*J. Combin. Inform. System Sci.*23, 237–250.Google Scholar

*Model-Oriented Design of Experiments*, Vol. 125 of Lecture Notes in Statistics, Springer.CrossRefGoogle Scholar

*mODa 8: Advances in Model-Oriented Design and Analysis*(López-Fidalgo, J., Rodríguez-Díaz, J. M. and Torsney, B., eds), Contributions to Statistics, Physica, Springer, pp. 57–66.CrossRefGoogle Scholar

*Proceedings of the 43rd Annual ACM Symposium on Theory of Computing (STOC 2011)*, ACM, pp. 569–578.Google Scholar

*Math. Program.*8, 73–87.CrossRefGoogle Scholar

*Technometrics*31, 49–60.CrossRefGoogle Scholar

*Proceedings of the 38th International Conference on Machine Learning (ICML 2021)*(Meila, M. and Zhang, T., eds), Vol. 139 of Proceedings of Machine Learning Research, PMLR, pp. 3384–3395.Google Scholar

*Advances in Neural Information Processing Systems 32*(Wallach, H. et al., eds), Curran Associates, pp. 14036–14047.Google Scholar

*Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics*, Vol. 108 of Proceedings of Machine Learning Research, PMLR, pp. 2959–2969.Google Scholar

*INFORMS TutORials in Operations Research*2018, 255–278.Google Scholar

*Mach. Learn.*28, 133–168.CrossRefGoogle Scholar

*Submodular Functions and Optimization*, Vol. 58 of Annals of Discrete Mathematics, second edition, Elsevier.Google Scholar

*Oszillationsmatrizen, Oszillationskerne und kleine Schwingungen mechanischer Systeme*, Vol. 5, Akademie.CrossRefGoogle Scholar

*k*-nearest neighbor information estimators,

*IEEE Trans. Inform. Theory*64, 5629–5661.CrossRefGoogle Scholar

*Acta Numer.*30, 445–554.CrossRefGoogle Scholar

*Mathematical Foundations of Infinite-Dimensional Statistical Models*, Cambridge University Press.Google Scholar

*Bayesian Anal.*2, 167–212.CrossRefGoogle Scholar

*Comput. Statist. Data Anal.*124, 252–276.CrossRefGoogle Scholar

*J. Amer. Statist. Assoc.*102, 359–378.CrossRefGoogle Scholar

*Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence*, Vol. 180 of Proceedings of Machine Learning Research, PMLR, pp. 718–727.Google Scholar

*SIAM J. Sci. Comput.*44, A286–A311.CrossRefGoogle Scholar

*Stoch. Anal. Appl.*38, 581–600.CrossRefGoogle Scholar

*SIAM/ASA J. Uncertain. Quantif.*4, 796–828.CrossRefGoogle Scholar

*Surrogates: Gaussian Process Modeling, Design, and Optimization for the Applied Sciences*, CRC Press.CrossRefGoogle Scholar

*J. Comput. Graph. Statist.*24, 561–578.CrossRefGoogle Scholar

*Bayesian Anal.*12, 1069–1103.CrossRefGoogle Scholar

*Proceedings of the 1994 Winter Simulation Conference (WSC ’94)*(Sadowski, D. A. et al., eds), ACM, pp. 247–254.Google Scholar

*Inverse Problems*24, art. 055012.CrossRefGoogle Scholar

*Inverse Problems*26, art. 025002.CrossRefGoogle Scholar

*mODa 11: Advances in Model-Oriented Design and Analysis*(Kunert, J., Müller, C. and Atkinson, A., eds), Contributions to Statistics, Springer, pp. 153–161.Google Scholar

*Statist. Comput.*32, art. 25.CrossRefGoogle ScholarPubMed

*The Oxford Handbook of Nonlinear Filtering*(Crisan, D. and Rozovskii, B., eds), Oxford University Press, pp. 833–873.Google Scholar

*J. Statist. Plann. Infer.*154, 87–101.CrossRefGoogle Scholar

*Annu. Rev. Statist. Appl.*9, 141–166.CrossRefGoogle Scholar

*Proceedings of the 1991 Winter Simulation Conference (WSC ’91)*(Nelson, B. L. et al., eds), IEEE Computer Society, pp. 901–906.CrossRefGoogle Scholar

*Statistics and Related Topics: International Symposium Proceedings*(Csorgo, M., ed.), Elsevier Science, pp. 39–56.Google Scholar

*Numer. Math.*150, 521–549.CrossRefGoogle Scholar

*SIAM J. Sci. Comput.*44, B506–B530.CrossRefGoogle Scholar

*Inverse Problems*36, art. 125011.CrossRefGoogle Scholar

*Proceedings of the 31st International Conference on Machine Learning (ICML 2014)*, Vol. 32 of Proceedings of Machine Learning Research, PMLR, pp. 739–747.Google Scholar

*ACM SIGACT News*28, 40–52.CrossRefGoogle Scholar

*J. Comput. Phys.*232, 288–317.CrossRefGoogle Scholar

*Int. J. Uncertain. Quantif.*4, 479–510.CrossRefGoogle Scholar

*Advances in Neural Information Processing Systems 29*(Lee, D. et al., eds), Curran Associates, pp. 4080–4088.Google Scholar

*Bayesian Anal.*18, 79–104.CrossRefGoogle ScholarPubMed

*Advances in Neural Information Processing Systems 34*(Ranzato, M. et al., eds), Curran Associates, pp. 25785–25798.Google Scholar

*J. Mach. Learn. Res.*22, 11397–11458.Google Scholar

*Probability Theory: The Logic of Science*, Cambridge University Press.CrossRefGoogle Scholar

*Linear Algebra Appl.*66, 177–193.CrossRefGoogle Scholar

*Technometrics*25, 271–277.Google Scholar

*J. Statist. Plann. Infer.*26, 131–148.CrossRefGoogle Scholar

*J. Global Optim.*13, 455–492.CrossRefGoogle Scholar

*Biometrika*102, 371–380.CrossRefGoogle Scholar

*J. Qual. Technol.*52, 343–354.CrossRefGoogle Scholar

*AStA Adv. Statist. Anal.*94, 341–351.CrossRefGoogle Scholar

*Artif. Intell.*101, 99–134.CrossRefGoogle Scholar

*J. Artif. Intell. Res.*4, 237–285.CrossRefGoogle Scholar

*Bayesian Theory and Applications*(Damien, P. et al., eds), Oxford University Press, pp. 644–672.CrossRefGoogle Scholar

*Statistical and Computational Inverse Problems*, Vol. 160 of Applied Mathematical Sciences, Springer.Google Scholar

*J. Comput. Appl. Math.*198, 493–504.CrossRefGoogle Scholar

*IEEE Conference on Decision and Control (CDC)*, IEEE, pp. 1118–1123.CrossRefGoogle Scholar

*Am. Acad. Sci. Fennicade, Ser. A, I*37, 3–79.Google Scholar

*Discrete Math.*44, 113–116.CrossRefGoogle Scholar

*Proceedings of the 37th AAAI Conference on Artificial Intelligence*(Williams, B., Chen, Y. and Neville, J., eds), AAAI Press, pp. 8220–8227.Google Scholar

*J. R. Statist. Soc. Ser. B. Statist. Methodol.*63, 425–464.CrossRefGoogle Scholar

*Ann. Math. Statist.*29, 675–699.CrossRefGoogle Scholar

*J. R. Statist. Soc. Ser. B. Statist. Methodol.*21, 272–304.CrossRefGoogle Scholar

*Ann. Math. Statist.*32, 298–325.CrossRefGoogle Scholar

*Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability*, Vol. 1, University of California Press, pp. 381–405.Google Scholar

*Ann. Statist.*2, 849–879.CrossRefGoogle Scholar

*Ann. Math. Statist.*23, 462–466.CrossRefGoogle Scholar

*Ann. Math. Statist.*30, 271–294.CrossRefGoogle Scholar

*Canad. J. Statist.*12, 363–366.Google Scholar

*Neural Comput.*26, 2565–2492.CrossRefGoogle ScholarPubMed

*Biometrics*56, 1263–1267.CrossRefGoogle ScholarPubMed

*Electron. J. Statist.*6, 354–381.CrossRefGoogle Scholar

*Proceedings of the 37th International Conference on Machine Learning (ICML 2020)*(Daumé, H. and Singh, A., eds), Vol. 119 of Proceedings of Machine Learning Research, PMLR, pp. 5316–5326.Google Scholar

*Bayesian Anal.*16, 773–802.CrossRefGoogle Scholar

*SIAM J. Optim.*12, 479–502.CrossRefGoogle Scholar

*Ann. Statist.*39, 2626–2657.CrossRefGoogle Scholar

*Michigan Math. J.*4, 39–52.CrossRefGoogle Scholar

*Oper. Res.*43, 684–691.CrossRefGoogle Scholar

*IEEE Trans. Pattern Anal. Mach. Intell.*43, 3964–3979.CrossRefGoogle Scholar

*Advances in Neural Information Processing Systems 12*(Solla, S. et al., eds), MIT Press, pp. 1008–1014.Google Scholar

*Scientific Computing in Chemical Engineering II*(Keil, F. et al., eds), Springer, pp. 338–345.Google Scholar

*Ukrains’kyi Matematychnyi Zhurnal*2, 94–101.Google Scholar

*SIAM/ASA J. Uncertain. Quantif.*10, 687–716.CrossRefGoogle Scholar

*Inverse Problems*36, art. 075007.CrossRefGoogle Scholar

*Probl. Inf. Transm.*23, 9–16.Google Scholar

*Phys. Rev. E*69, art. 066138.CrossRefGoogle ScholarPubMed

*Tractability, Practical Approaches to Hard Problems*(Bordeaux, L., Hamadi, Y. and Kohli, P., eds), Cambridge University Press, pp. 71–104.CrossRefGoogle Scholar

*J. Mach. Learn. Res.*9, 235–284.Google Scholar

*Operations Research & Management Science in the Age of Analytics*, INFORMS, pp. 130–166.CrossRefGoogle Scholar

*Stochastic Approximation and Recursive Algorithms and Applications*, second edition, Springer.Google Scholar

*Advances in Neural Information Processing Systems 30*(Guyon, I. et al., eds), Curran Associates, pp. 1890–1900.Google Scholar

*Acta Numer.*28, 287–404.CrossRefGoogle Scholar

*Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing (STOC 2020)*, ACM, pp. 826–839.CrossRefGoogle Scholar

*SIAM J. Comput.*51, 900–951.CrossRefGoogle Scholar

*Ann. Math. Statist.*35, 1419–1455.CrossRefGoogle Scholar

*Theory of Point Estimation*, Springer Texts in Statistics, Springer.Google Scholar

*Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, ACM, pp. 420–429.CrossRefGoogle Scholar

*f*-divergence and derangements for discriminative mutual information estimation. Available at arXiv:2305.20025.Google Scholar

*SIGIR Forum*29, 13–19.CrossRefGoogle Scholar

*ϕ*-Sobolev inequalities. To appear in

*Bernoulli*. Available at https://bernoullisociety.org/publications/ bernoulli-journal/bernoulli-journal-papers.CrossRefGoogle Scholar

*PLoS Comput. Biol.*9, e1002888.CrossRefGoogle ScholarPubMed

*Proceedings of the 4th International Conference on Learning Representations (ICLR 2016)*(Bengio, Y. and LeCun, Y., eds).Google Scholar

*Ann. Math. Statist.*27, 986–1005.CrossRefGoogle Scholar

*Processus Stochastique et Mouvement Brownien*(Lévy, P., ed.), Gauthier-Villars.Google Scholar

*Comput. Methods Appl. Mech. Engrg*259, 24–39.CrossRefGoogle Scholar

*Bayesian Statistics 9: Proceedings of the Ninth Valencia International Meeting*(Bernardo, J. M., Bayarri, M. J. and Berger, J. O., eds), Oxford University Press, pp. 361–392.CrossRefGoogle Scholar

*Mathematical Programming The State of the Art: Bonn 1982*(Bachem, A., Korte, B. and Grötschel, M., eds), Springer, pp. 235–257.CrossRefGoogle Scholar

*Combinatorial Problems and Exercises*, second edition, American Mathematical Society.Google Scholar

*Neural Comput.*4, 590–604.CrossRefGoogle Scholar

*IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS)*, IEEE, pp. 565–576.Google Scholar

*Proceedings of the 32nd Conference on Learning Theory*, Vol. 99 of Proceedings of Machine Learning Research, PMLR, pp. 2210–2258.Google Scholar

*Oper. Res. Lett.*24, 47–56.CrossRefGoogle Scholar

*Commun. Comput. Phys.*6, 826–847.CrossRefGoogle Scholar

*Handbook of Uncertainty Quantification*(Ghanem, R., Higdon, D. and Owhadi, H., eds), Springer, pp. 1–41.Google Scholar

*Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics*, Vol. 108 of Proceedings of Machine Learning Research, PMLR, pp. 875–884.Google Scholar

*Technometrics*21, 239–245.Google Scholar

*Advances in Neural Information Processing Systems 33*(Larochelle, H. et al., eds), Curran Associates, pp. 1117–1128.Google Scholar

*Technometrics*37, 60–69.CrossRefGoogle Scholar

*J. Amer. Statist. Assoc.*114, 1113–1125.CrossRefGoogle ScholarPubMed

*Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI 2015)*, pp. 1812–1818.Google Scholar

*Advances in Neural Information Processing Systems 26*(Burges, C. J. et al., eds), Curran Associates, pp. 2049–2057.Google Scholar

*Nature*518, 529–533.CrossRefGoogle ScholarPubMed

*Optimization Techniques IFIP Technical Conference*(Marchuk, G. I., ed.), Springer, pp. 400–404.CrossRefGoogle Scholar

*J. Mach. Learn. Res.*21, 5183–5244.Google Scholar

*SIAM/ASA J. Uncertain. Quantif.*6, 457–496.CrossRefGoogle Scholar

*J. Statist. Plann. Infer.*137, 3140–3150.CrossRefGoogle Scholar

*Pharma. Statist.*21, 729–739.CrossRefGoogle ScholarPubMed

*J. R. Statist. Soc. Ser. B. Statist. Methodol.*65, 331–366.CrossRefGoogle Scholar

*Psychol. Rev.*116, 499–518.CrossRefGoogle ScholarPubMed

*Comput. J.*7, 308–313.CrossRefGoogle Scholar

*Math. Oper. Res.*3, 177–188.CrossRefGoogle Scholar

*Math. Program.*14, 265–294.CrossRefGoogle Scholar

*Soviet Math. Doklady*27, 372–376.Google Scholar

*Proceedings of the 17th International Conference on Machine Learning (ICML 2000)*, Morgan Kaufmann, pp. 663–670.Google Scholar

*IEEE Trans. Inform. Theory*56, 5847–5861.CrossRefGoogle Scholar

*Random Number Generation and Quasi-Monte Carlo Methods*, SIAM.CrossRefGoogle Scholar

*Proceedings of the 48th Annual ACM Symposium on Theory of Computing (STOC 2016)*, ACM, pp. 192–201.Google Scholar

*Math. Oper. Res.*47, 847–877.CrossRefGoogle Scholar

*Math. Program.*83, 425–450.CrossRefGoogle Scholar

*Uncertain Judgements: Eliciting Experts’ Probabilities*, Wiley.CrossRefGoogle Scholar

*J. Statist. Plann. Infer.*218, 138–146.CrossRefGoogle Scholar

*Technometrics*59, 458–470.CrossRefGoogle Scholar

*Statist. Comput.*28, 343–358.CrossRefGoogle Scholar

*Statist. Sinica*2, 439–452.Google Scholar

*Combinatorial Optimization: Algorithms and Complexity*, Courier Corporation.Google Scholar

*J. Mach. Learn. Res.*22, 2617–2680.Google Scholar

*Math. Geol.*30, 95–108.CrossRefGoogle Scholar

*Neurocomput.*71, 1180–1190.CrossRefGoogle Scholar

*Bayesian Estimation and Experimental Design in Linear Regression Models*, Wiley.Google Scholar

*SIAM J. Control Optim.*30, 838–855.CrossRefGoogle Scholar

*Proceedings of the 36th International Conference on Machine Learning (ICML 2019)*, Vol. 97 of Proceedings of Machine Learning Research, PMLR, pp. 5171–5180.Google Scholar

*Approximate Dynamic Programming: Solving the Curses of Dimensionality*, second edition, Wiley.CrossRefGoogle Scholar

*Bayesian Anal.*18, 133–163.CrossRefGoogle Scholar

*Statist. Comput.*22, 681–701.CrossRefGoogle Scholar

*Statist. Methods Appl.*11, 277–292.CrossRefGoogle Scholar

*Math. Biosci.*75, 103–120.CrossRefGoogle Scholar

*Proceedings of the 35th International Conference on Machine Learning (ICML 2018)*, Vol. 80 of Proceedings of Machine Learning Research, PMLR, pp. 4267–4276.Google Scholar

*Statist. Sci.*39, 100–114.Google Scholar

*Gaussian Processes for Machine Learning*, MIT Press.Google Scholar

*Oper. Res.*63, 1026–1043.CrossRefGoogle Scholar

*Advances in Neural Information Processing Systems 35*(Koyejo, S. et al., eds), Curran Associates, pp. 12141–12153.Google Scholar

*J. Sound Vib.*441, 96–105.CrossRefGoogle Scholar

*Ann. Math. Statist.*22, 400–407.CrossRefGoogle Scholar

*SIAM J. Sci. Statist. Comput.*10, 341–358.CrossRefGoogle Scholar

*SIAM J. Optim.*25, 1179–1208.CrossRefGoogle Scholar

*J. Bank. Finance*26, 1443–1471.CrossRefGoogle Scholar

*Surv. Oper. Res. Manag. Sci.*18, 33–53.Google Scholar

*Ann. Math. Statist.*23, 470–472.CrossRefGoogle Scholar

*Found. Comput. Math.*18, 309–343.CrossRefGoogle Scholar

*SIAM J. Sci. Comput.*40, B1080–B1100.CrossRefGoogle Scholar

*Int. Statist. Rev.*84, 128–154.CrossRefGoogle Scholar

*J. Comput. Graph. Statist.*12, 585–603.CrossRefGoogle Scholar

*Statist. Sci.*4, 118–128.Google Scholar

*g*, Vol. 87 of Progress in Nonlinear Differential Equations and their Applications, Springer.CrossRefGoogle Scholar

*The Design and Analysis of Computer Experiments*, second edition, Springer.CrossRefGoogle Scholar

*Int. J. Uncertain. Quantif.*9, 365–394.CrossRefGoogle Scholar

*Int. J. Chem. Kinet.*47, 246–276.CrossRefGoogle Scholar

*Mach. Learn.*68, 235–265.CrossRefGoogle Scholar

*ESAIM Math. Model. Numer. Anal.*50, 1825–1856.CrossRefGoogle Scholar

*Numer. Math.*145, 915–971.CrossRefGoogle Scholar

*J. R. Statist. Soc. Ser. B. Statist. Methodol.*62, 145–157.CrossRefGoogle Scholar

*Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks (IJCNN 2000)*, Springer, pp. 27–34.Google Scholar

*Theory of Optimal Designs*, Vol. 54 of Lecture Notes in Statistics, Springer.CrossRefGoogle Scholar

*Proc. IEEE*104, 148–175.CrossRefGoogle Scholar

*Ann*.

*Oper. Res.*30, 169–186.CrossRefGoogle Scholar

*Lectures on Stochastic Programming: Modeling and Theory*, third edition, SIAM.CrossRefGoogle Scholar

*SIAM J. Optim.*28, 3145–3176.CrossRefGoogle Scholar

*Comput. Methods Appl. Mech. Engrg*416, art. 116304.CrossRefGoogle Scholar

*J. Appl. Statist.*14, 165–170.CrossRefGoogle Scholar

*Water Resour. Res.*53, 9860–9882.CrossRefGoogle Scholar

*Proceedings of the 31st International Conference on Machine Learning (ICML 2014)*, Vol. 32 of Proceedings of Machine Learning Research, PMLR, pp. 387–395.Google Scholar

*Math. Oper. Res.*45, 1512–1534.CrossRefGoogle Scholar

*J. Comput. Graph. Statist.*21, 234–252.CrossRefGoogle Scholar

*Proceedings of the 8th International Conference on Learning Representations (ICLR 2020)*. Available at https://openreview.net/forum?id=B1x62TNtDS.Google Scholar

*Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics*, Vol. 89 of Proceedings of Machine Learning Research, PMLR, pp. 3158–3167.Google Scholar

*IEEE Trans. Aerosp. Electron. Systems*34, 817–823.CrossRefGoogle Scholar

*Johns Hopkins APL Tech*.

*Digest*19, 482–492.Google Scholar

*J. Mach. Learn. Res.*19, 2639–2709.Google Scholar

*SIAM J. Sci. Comput.*39, S167–S196.CrossRefGoogle Scholar

*SIAM J. Sci. Comput.*37, A2451–A2487.CrossRefGoogle Scholar

*Proceedings of the 40th Annual ACM Symposium on Theory of Computing (STOC 2008)*, ACM, pp. 563–568.Google Scholar

*Environ. Model. Softw.*33, 48–60.CrossRefGoogle Scholar

*Stoch. Env. Res. Risk Assessment*24, 463–482.Google Scholar

*SIAM/ASA J. Uncertain. Quantif.*11, 1044–1068.CrossRefGoogle Scholar

*Inverse Problems*36, art. 055015.CrossRefGoogle Scholar

*European J. Oper. Res.*198, 878–890.CrossRefGoogle Scholar

*Technometrics*26, 71–97.CrossRefGoogle Scholar

*Ann. Math. Statist.*30, 55–70.CrossRefGoogle Scholar

*Geophys. J. Int.*236, 1309–1331.CrossRefGoogle Scholar

*Math. Comp.*87, 721–753.CrossRefGoogle Scholar

*Acta Numer.*19, 451–559.CrossRefGoogle Scholar

*Water Resour. Res.*43, 1–14.Google Scholar

*Reinforcement Leaning*, second edition, MIT Press.Google Scholar

*Advances in Neural Information Processing Systems 12*(Solla, S. et al., eds), MIT Press, pp. 1057–1063.Google Scholar

*Oper. Res. Lett.*32, 41–43.CrossRefGoogle Scholar

*Math. Oper. Res.*42, 1197–1218.CrossRefGoogle Scholar

*J. Amer. Statist. Assoc.*88, 1392–1397.CrossRefGoogle Scholar

*Amer. Statist.*77, 223–233.CrossRefGoogle Scholar

*Experimental Therm. Fluid Sci.*36, 178–193.CrossRefGoogle Scholar

*SIAM J. Optim.*7, 1–25.CrossRefGoogle Scholar

*IEEE Trans. Automat. Control*66, 1468–1483.CrossRefGoogle Scholar

*ACM Trans. Math. Software*47, 1–34.CrossRefGoogle Scholar

*Rev. Modern Phys.*83, 943–999.CrossRefGoogle Scholar

*Proceedings of the Fortieth Annual ACM Symposium on Theory of Computing (STOC 2008)*, ACM, pp. 67–74.Google Scholar

*RIMS Kokyuroku Bessatsu*B23, 253–266.Google Scholar

*Proceedings of the Forty-Third Annual ACM Symposium on Theory of Computing (STOC 2011)*, ACM, pp. 783–792.Google Scholar

*Ann. Math. Statist.*14, 134–140.CrossRefGoogle Scholar

*Statist. Probab. Lett.*112, 5–9.CrossRefGoogle Scholar

*Oper. Res.*68, 1850–1865.CrossRefGoogle Scholar

*ACM Comput. Surv.*55, 1–36.Google Scholar

*J. Comput. Graph. Statist.*15, 398–413.CrossRefGoogle Scholar

*Technometrics*63, 90–99.CrossRefGoogle Scholar

*Bayesian Anal.*11, 191–213.CrossRefGoogle Scholar

*J. R. Statist. Soc. Ser. B. Statist. Methodol.*35, 123–130.CrossRefGoogle Scholar

*Integer and Combinatorial Optimization*, Wiley.Google Scholar

*Advances in Neural Information Processing Systems 32*(Wallach, H. et al., eds), Curran Associates, pp. 9813–9823.Google Scholar

*SIAM J. Sci. Comput.*45, B57–B77.CrossRefGoogle Scholar

*J. Sci. Comput.*95, art. 30.CrossRefGoogle Scholar

*J. R. Statist. Soc. Ser. B. Statist. Methodol.*34, 133–147.CrossRefGoogle Scholar

*Ann. Statist.*12, 416–423.CrossRefGoogle Scholar

*Entropy*22, art. 258.CrossRefGoogle ScholarPubMed

*J. Agric. Sci.*23, 108–145.CrossRefGoogle Scholar

*Math. Comp.*91, 1789–1835.CrossRefGoogle Scholar

*J. Global Optim.*78, 507–544.CrossRefGoogle Scholar

*Proceedings of the 24th International Conference on Artificial Intelligence and Statistics*, Vol. 130 of Proceedings of Machine Learning Research, PMLR, pp. 3745–3753.Google Scholar

*Advances in Neural Information Processing Systems 33*(Larochelle, H. et al., eds), Curran Associates, pp. 4127–4137.Google Scholar

*J. Statist. Plann. Infer.*134, 583–603.CrossRefGoogle Scholar

*J. Agric. Biol. Environ. Statist.*11, 24–44.CrossRefGoogle Scholar