The twenty years since the publication of the first edition of this book have seen tremendous
progress in artificial intelligence, propelled in large part by advances in machine learning,
including advances in reinforcement learning. Although the impressive computational
power that became available is responsible for some of these advances, new developments
in theory and algorithms have been driving forces as well. In the face of this progress, a
second edition of our 1998 book was long overdue, and we finally began the project in
2012. Our goal for the second edition was the same as our goal for the first: to provide a
clear and simple account of the key ideas and algorithms of reinforcement learning that
is accessible to readers in all the related disciplines. The edition remains an introduction,
and we retain a focus on core, online learning algorithms. This edition includes some new
topics that rose to importance over the intervening years, and we expanded coverage of
topics that we now understand better. But we made no attempt to provide comprehensive
coverage of the field, which has exploded in many di↵erent directions. We apologize for
having to leave out all but a handful of these contributions.
1