Information-Directed Policy Search in Sparse-Reward Settings via the Occupancy Information Ratio

Wesley A. Suttle; Alec Koppel; Ji Liu

doi:10.1109/CISS56502.2023.10089655

Information-Directed Policy Search in Sparse-Reward Settings via the Occupancy Information Ratio

Wesley A. Suttle
, Alec Koppel
, Ji Liu

Electrical Engineering

Stony Brook University

U.S. Army Research Laboratory
J.P. Morgan Chase & Co.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

This paper examines a new measure of the exploration/exploitation trade-off in reinforcement learning (RL) called the occupancy information ratio (OIR). To this end, the paper derives the Information-Directed Actor-Critic (IDAC) algorithm for solving the OIR problem, provides an overview of the rich theory underlying IDAC and related OIR policy gradient methods, and experimentally investigates the advantages of such methods. The central contribution of this paper is to provide empirical evidence that, due to the form of the OIR objective, IDAC enjoys superior performance over vanilla RL methods in sparse-reward environments.

Original language	English
Title of host publication	2023 57th Annual Conference on Information Sciences and Systems, CISS 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665451819
DOIs	https://doi.org/10.1109/CISS56502.2023.10089655
State	Published - 2023
Event	57th Annual Conference on Information Sciences and Systems, CISS 2023 - Baltimore, United States Duration: Mar 22 2023 → Mar 24 2023

Publication series

Name	2023 57th Annual Conference on Information Sciences and Systems, CISS 2023

Conference

Conference	57th Annual Conference on Information Sciences and Systems, CISS 2023
Country/Territory	United States
City	Baltimore
Period	03/22/23 → 03/24/23

Keywords

exploration vs. exploitation
reinforcement learning
sparse rewards

Access to Document

10.1109/CISS56502.2023.10089655

Cite this

Suttle, W. A., Koppel, A., & Liu, J. (2023). Information-Directed Policy Search in Sparse-Reward Settings via the Occupancy Information Ratio. In 2023 57th Annual Conference on Information Sciences and Systems, CISS 2023 (2023 57th Annual Conference on Information Sciences and Systems, CISS 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CISS56502.2023.10089655

@inproceedings{9cc174cd77eb481fb52c00027909a5d2,

title = "Information-Directed Policy Search in Sparse-Reward Settings via the Occupancy Information Ratio",

abstract = "This paper examines a new measure of the exploration/exploitation trade-off in reinforcement learning (RL) called the occupancy information ratio (OIR). To this end, the paper derives the Information-Directed Actor-Critic (IDAC) algorithm for solving the OIR problem, provides an overview of the rich theory underlying IDAC and related OIR policy gradient methods, and experimentally investigates the advantages of such methods. The central contribution of this paper is to provide empirical evidence that, due to the form of the OIR objective, IDAC enjoys superior performance over vanilla RL methods in sparse-reward environments.",

keywords = "exploration vs. exploitation, reinforcement learning, sparse rewards",

author = "Suttle, \{Wesley A.\} and Alec Koppel and Ji Liu",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 57th Annual Conference on Information Sciences and Systems, CISS 2023 ; Conference date: 22-03-2023 Through 24-03-2023",

year = "2023",

doi = "10.1109/CISS56502.2023.10089655",

language = "English",

series = "2023 57th Annual Conference on Information Sciences and Systems, CISS 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2023 57th Annual Conference on Information Sciences and Systems, CISS 2023",

}

Suttle, WA, Koppel, A & Liu, J 2023, Information-Directed Policy Search in Sparse-Reward Settings via the Occupancy Information Ratio. in 2023 57th Annual Conference on Information Sciences and Systems, CISS 2023. 2023 57th Annual Conference on Information Sciences and Systems, CISS 2023, Institute of Electrical and Electronics Engineers Inc., 57th Annual Conference on Information Sciences and Systems, CISS 2023, Baltimore, United States, 03/22/23. https://doi.org/10.1109/CISS56502.2023.10089655

Information-Directed Policy Search in Sparse-Reward Settings via the Occupancy Information Ratio. / Suttle, Wesley A.; Koppel, Alec; Liu, Ji.
2023 57th Annual Conference on Information Sciences and Systems, CISS 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (2023 57th Annual Conference on Information Sciences and Systems, CISS 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Information-Directed Policy Search in Sparse-Reward Settings via the Occupancy Information Ratio

AU - Suttle, Wesley A.

AU - Koppel, Alec

AU - Liu, Ji

PY - 2023

Y1 - 2023

N2 - This paper examines a new measure of the exploration/exploitation trade-off in reinforcement learning (RL) called the occupancy information ratio (OIR). To this end, the paper derives the Information-Directed Actor-Critic (IDAC) algorithm for solving the OIR problem, provides an overview of the rich theory underlying IDAC and related OIR policy gradient methods, and experimentally investigates the advantages of such methods. The central contribution of this paper is to provide empirical evidence that, due to the form of the OIR objective, IDAC enjoys superior performance over vanilla RL methods in sparse-reward environments.

AB - This paper examines a new measure of the exploration/exploitation trade-off in reinforcement learning (RL) called the occupancy information ratio (OIR). To this end, the paper derives the Information-Directed Actor-Critic (IDAC) algorithm for solving the OIR problem, provides an overview of the rich theory underlying IDAC and related OIR policy gradient methods, and experimentally investigates the advantages of such methods. The central contribution of this paper is to provide empirical evidence that, due to the form of the OIR objective, IDAC enjoys superior performance over vanilla RL methods in sparse-reward environments.

KW - exploration vs. exploitation

KW - reinforcement learning

KW - sparse rewards

UR - https://www.scopus.com/pages/publications/85154040212

U2 - 10.1109/CISS56502.2023.10089655

DO - 10.1109/CISS56502.2023.10089655

M3 - Conference contribution

T3 - 2023 57th Annual Conference on Information Sciences and Systems, CISS 2023

BT - 2023 57th Annual Conference on Information Sciences and Systems, CISS 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 57th Annual Conference on Information Sciences and Systems, CISS 2023

Y2 - 22 March 2023 through 24 March 2023

ER -

Suttle WA, Koppel A, Liu J. Information-Directed Policy Search in Sparse-Reward Settings via the Occupancy Information Ratio. In 2023 57th Annual Conference on Information Sciences and Systems, CISS 2023. Institute of Electrical and Electronics Engineers Inc. 2023. (2023 57th Annual Conference on Information Sciences and Systems, CISS 2023). doi: 10.1109/CISS56502.2023.10089655

Information-Directed Policy Search in Sparse-Reward Settings via the Occupancy Information Ratio

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this