train reinforcement learning agent in mdp environment -凯发k8网页登录

create mdp environment

create an mdp model with eight states and two actions ("up" and "down").

mdp = createmdp(8,["up";"down"]);

to model the transitions from the above graph, modify the state transition matrix and reward matrix of the mdp. by default, these matrices contain zeros. for more information on creating an mdp model and the properties of an mdp object, see createmdp.

specify the state transition and reward matrices for the mdp. for example, in the following commands:

mdp.t(1,2,1) = 1;
mdp.r(1,2,1) = 3;
mdp.t(1,3,2) = 1;
mdp.r(1,3,2) = 1;

similarly, specify the state transitions and rewards for the remaining rules in the graph.

% state 2 transition and reward
mdp.t(2,4,1) = 1;
mdp.r(2,4,1) = 2;
mdp.t(2,5,2) = 1;
mdp.r(2,5,2) = 1;
% state 3 transition and reward
mdp.t(3,5,1) = 1;
mdp.r(3,5,1) = 2;
mdp.t(3,6,2) = 1;
mdp.r(3,6,2) = 4;
% state 4 transition and reward
mdp.t(4,7,1) = 1;
mdp.r(4,7,1) = 3;
mdp.t(4,8,2) = 1;
mdp.r(4,8,2) = 2;
% state 5 transition and reward
mdp.t(5,7,1) = 1;
mdp.r(5,7,1) = 1;
mdp.t(5,8,2) = 1;
mdp.r(5,8,2) = 9;
% state 6 transition and reward
mdp.t(6,7,1) = 1;
mdp.r(6,7,1) = 5;
mdp.t(6,8,2) = 1;
mdp.r(6,8,2) = 1;
% state 7 transition and reward
mdp.t(7,7,1) = 1;
mdp.r(7,7,1) = 0;
mdp.t(7,7,2) = 1;
mdp.r(7,7,2) = 0;
% state 8 transition and reward
mdp.t(8,8,1) = 1;
mdp.r(8,8,1) = 0;
mdp.t(8,8,2) = 1;
mdp.r(8,8,2) = 0;

specify states "s7" and "s8" as terminal states of the mdp.

mdp.terminalstates = ["s7";"s8"];

create the reinforcement learning mdp environment for this process model.

env = rlmdpenv(mdp);

to specify that the initial state of the agent is always state 1, specify a reset function that returns the initial agent state. this function is called at the start of each training episode and simulation. create an anonymous function handle that sets the initial state to 1.

env.resetfcn = @() 1;

fix the random generator seed for reproducibility.

rng(0)

create q-learning agent

to create a q-learning agent, first create a q table using the observation and action specifications from the mdp environment. set the learning rate of the representation to 1.

obsinfo = getobservationinfo(env);
actinfo = getactioninfo(env);
qtable = rltable(obsinfo, actinfo);
qfunction = rlqvaluefunction(qtable, obsinfo, actinfo);
qoptions = rloptimizeroptions(learnrate=1);

next, create a q-learning agent using this table representation, configuring the epsilon-greedy exploration. for more information on creating q-learning agents, see and .

agentopts = rlqagentoptions;
agentopts.discountfactor = 1;
agentopts.epsilongreedyexploration.epsilon = 0.9;
agentopts.epsilongreedyexploration.epsilondecay = 0.01;
agentopts.criticoptimizeroptions = qoptions;
qagent = rlqagent(qfunction,agentopts); %#ok 

train q-learning agent

to train the agent, first specify the training options. for this example, use the following options:

for more information, see rltrainingoptions.

trainopts = rltrainingoptions;
trainopts.maxstepsperepisode = 50;
trainopts.maxepisodes = 500;
trainopts.stoptrainingcriteria = "averagereward";
trainopts.stoptrainingvalue = 13;
trainopts.scoreaveragingwindowlength = 30;

train the agent using the train function. this may take several minutes to complete. to save time while running this example, load a pretrained agent by setting dotraining to false. to train the agent yourself, set dotraining to true.

dotraining = false;
if dotraining
    % train the agent.
    trainingstats = train(qagent,env,trainopts); %#ok 
else
    % load pretrained agent for the example.
    load("genericmdpqagent.mat","qagent"); 
end

validate q-learning results

to validate the training results, simulate the agent in the training environment using the sim function. the agent successfully finds the optimal path which results in cumulative reward of 13.

data = sim(qagent,env);
cumulativereward = sum(data.reward)

cumulativereward = 13

since the discount factor is set to 1, the values in the q table of the trained agent match the undiscounted returns of the environment.

qtable = getlearnableparameters(getcritic(qagent));
qtable{1}

ans = 8x2 single matrix
   12.9912   11.6621
    8.2141    9.9950
   10.8645    4.0414
    4.8017   -1.6150
    5.1975    8.9975
    5.8058   -0.2353
         0         0
         0         0

truetablevalues = [13,12;5,10;11,9;3,2;1,9;5,1;0,0;0,0]

truetablevalues = 8×2
    13    12
     5    10
    11     9
     3     2
     1     9
     5     1
     0     0
     0     0