Optimal control of distributed Markov decision processes with network delays

Sachin Adlakha; Ritesh Madan; Sanjay Lall; Andrea Goldsmith

doi:10.1109/CDC.2007.4434792

Optimal control of distributed Markov decision processes with network delays

Sachin Adlakha
, Ritesh Madan
, Sanjay Lall
, Andrea Goldsmith

Stony Brook University

Stony Brook University

Stanford University
Qualcomm Incorporated

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Scopus citations

Abstract

We consider the problem of finding an optimal feedback controller for a network of interconnected subsystems, each of which is a Markov decision process. Each subsystem is coupled to its neighbors via communication links by which signals are delayed but are otherwise transmitted noise-free. One of the subsystems receives input from a controller, and the controller receives delayed statemeasurements from all of the subsystems. We show that an optimal controller requires only a finite amount of memory which does not grow with time, and obtain a bound on the amount of memory that a controller needs to have for each subsystem. This makes the computation of an optimal controller through dynamic programming tractable. We illustrate our result by a numerical example, and show that it generalizes previous results on Markov decision processes with delayed state measurements.

Original language	English
Title of host publication	Proceedings of the 46th IEEE Conference on Decision and Control 2007, CDC
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	3308-3314
Number of pages	7
ISBN (Print)	1424414989, 9781424414987
DOIs	https://doi.org/10.1109/CDC.2007.4434792
State	Published - 2007
Event	46th IEEE Conference on Decision and Control 2007, CDC - New Orleans, LA, United States Duration: Dec 12 2007 → Dec 14 2007

Publication series

Name	Proceedings of the IEEE Conference on Decision and Control

Conference

Conference	46th IEEE Conference on Decision and Control 2007, CDC
Country/Territory	United States
City	New Orleans, LA
Period	12/12/07 → 12/14/07

Access to Document

10.1109/CDC.2007.4434792

Cite this

Adlakha, S., Madan, R., Lall, S., & Goldsmith, A. (2007). Optimal control of distributed Markov decision processes with network delays. In Proceedings of the 46th IEEE Conference on Decision and Control 2007, CDC (pp. 3308-3314). Article 4434792 (Proceedings of the IEEE Conference on Decision and Control). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CDC.2007.4434792

@inproceedings{53c4f91fa4004f7b92f297dd6fa398a9,

title = "Optimal control of distributed Markov decision processes with network delays",

abstract = "We consider the problem of finding an optimal feedback controller for a network of interconnected subsystems, each of which is a Markov decision process. Each subsystem is coupled to its neighbors via communication links by which signals are delayed but are otherwise transmitted noise-free. One of the subsystems receives input from a controller, and the controller receives delayed statemeasurements from all of the subsystems. We show that an optimal controller requires only a finite amount of memory which does not grow with time, and obtain a bound on the amount of memory that a controller needs to have for each subsystem. This makes the computation of an optimal controller through dynamic programming tractable. We illustrate our result by a numerical example, and show that it generalizes previous results on Markov decision processes with delayed state measurements.",

author = "Sachin Adlakha and Ritesh Madan and Sanjay Lall and Andrea Goldsmith",

year = "2007",

doi = "10.1109/CDC.2007.4434792",

language = "English",

isbn = "1424414989",

series = "Proceedings of the IEEE Conference on Decision and Control",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "3308--3314",

booktitle = "Proceedings of the 46th IEEE Conference on Decision and Control 2007, CDC",

note = "46th IEEE Conference on Decision and Control 2007, CDC ; Conference date: 12-12-2007 Through 14-12-2007",

}

Adlakha, S, Madan, R, Lall, S & Goldsmith, A 2007, Optimal control of distributed Markov decision processes with network delays. in Proceedings of the 46th IEEE Conference on Decision and Control 2007, CDC., 4434792, Proceedings of the IEEE Conference on Decision and Control, Institute of Electrical and Electronics Engineers Inc., pp. 3308-3314, 46th IEEE Conference on Decision and Control 2007, CDC, New Orleans, LA, United States, 12/12/07. https://doi.org/10.1109/CDC.2007.4434792

Optimal control of distributed Markov decision processes with network delays. / Adlakha, Sachin; Madan, Ritesh; Lall, Sanjay et al.
Proceedings of the 46th IEEE Conference on Decision and Control 2007, CDC. Institute of Electrical and Electronics Engineers Inc., 2007. p. 3308-3314 4434792 (Proceedings of the IEEE Conference on Decision and Control).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Optimal control of distributed Markov decision processes with network delays

AU - Adlakha, Sachin

AU - Madan, Ritesh

AU - Lall, Sanjay

AU - Goldsmith, Andrea

PY - 2007

Y1 - 2007

N2 - We consider the problem of finding an optimal feedback controller for a network of interconnected subsystems, each of which is a Markov decision process. Each subsystem is coupled to its neighbors via communication links by which signals are delayed but are otherwise transmitted noise-free. One of the subsystems receives input from a controller, and the controller receives delayed statemeasurements from all of the subsystems. We show that an optimal controller requires only a finite amount of memory which does not grow with time, and obtain a bound on the amount of memory that a controller needs to have for each subsystem. This makes the computation of an optimal controller through dynamic programming tractable. We illustrate our result by a numerical example, and show that it generalizes previous results on Markov decision processes with delayed state measurements.

AB - We consider the problem of finding an optimal feedback controller for a network of interconnected subsystems, each of which is a Markov decision process. Each subsystem is coupled to its neighbors via communication links by which signals are delayed but are otherwise transmitted noise-free. One of the subsystems receives input from a controller, and the controller receives delayed statemeasurements from all of the subsystems. We show that an optimal controller requires only a finite amount of memory which does not grow with time, and obtain a bound on the amount of memory that a controller needs to have for each subsystem. This makes the computation of an optimal controller through dynamic programming tractable. We illustrate our result by a numerical example, and show that it generalizes previous results on Markov decision processes with delayed state measurements.

UR - https://www.scopus.com/pages/publications/62749192571

U2 - 10.1109/CDC.2007.4434792

DO - 10.1109/CDC.2007.4434792

M3 - Conference contribution

SN - 1424414989

SN - 9781424414987

T3 - Proceedings of the IEEE Conference on Decision and Control

SP - 3308

EP - 3314

BT - Proceedings of the 46th IEEE Conference on Decision and Control 2007, CDC

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 46th IEEE Conference on Decision and Control 2007, CDC

Y2 - 12 December 2007 through 14 December 2007

ER -

Adlakha S, Madan R, Lall S, Goldsmith A. Optimal control of distributed Markov decision processes with network delays. In Proceedings of the 46th IEEE Conference on Decision and Control 2007, CDC. Institute of Electrical and Electronics Engineers Inc. 2007. p. 3308-3314. 4434792. (Proceedings of the IEEE Conference on Decision and Control). doi: 10.1109/CDC.2007.4434792

Optimal control of distributed Markov decision processes with network delays

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this