train dqn agent to balance cart-凯发k8网页登录

cart-pole matlab environment

the reinforcement learning environment for this example is a pole attached to an unactuated joint on a cart, which moves along a frictionless track. the training goal is to make the pole stand upright without falling over.

for this environment:

for more information on this model, see load predefined control system environments.

create environment interface

create a predefined environment interface for the system.

env = rlpredefinedenv("cartpole-discrete")

env = 
  cartpolediscreteaction with properties:
                  gravity: 9.8000
                 masscart: 1
                 masspole: 0.1000
                   length: 0.5000
                 maxforce: 10
                       ts: 0.0200
    thetathresholdradians: 0.2094
               xthreshold: 2.4000
      rewardfornotfalling: 1
        penaltyforfalling: -5
                    state: [4x1 double]

the interface has a discrete action space where the agent can apply one of two possible force values to the cart, –10 or 10 n.

get the observation and action specification information.

obsinfo = getobservationinfo(env)

obsinfo = 
  rlnumericspec with properties:
     lowerlimit: -inf
     upperlimit: inf
           name: "cartpole states"
    description: "x, dx, theta, dtheta"
      dimension: [4 1]
       datatype: "double"

actinfo = getactioninfo(env)

actinfo = 
  rlfinitesetspec with properties:
       elements: [-10 10]
           name: "cartpole action"
    description: [0x0 string]
      dimension: [1 1]
       datatype: "double"

fix the random generator seed for reproducibility.

rng(0)

create dqn agent

dqn agents can use vector q-value functions critics, which are generally more efficient than comparable single-output critics. a vector q-value function critic has observations as inputs and state-action values as outputs. each output element represents the expected cumulative long-term reward for taking the corresponding discrete action from the state indicated by the observation inputs. for more information on creating value-functions, see create policies and value functions.

to approximate the q-value function within the critic, use a neural network with one input channel (the 4-dimensional observed state vector) and one output channel with two elements (one for the 10 n action, another for the –10 n action). define the network as an array of layer objects, and get the dimension of the observation space and the number of possible actions from the environment specification objects.

net = [
    featureinputlayer(obsinfo.dimension(1))
    fullyconnectedlayer(20)
    relulayer
    fullyconnectedlayer(length(actinfo.elements))
    ];

convert to dlnetwork and display the number of weights.

net = dlnetwork(net);
summary(net)

   initialized: true
   number of learnables: 142
   inputs:
      1   'input'   4 features

view the network configuration.

plot(net)

create the critic approximator using net and the environment specifications. for more information, see .

critic = rlvectorqvaluefunction(net,obsinfo,actinfo);

check the critic with a random observation input.

getvalue(critic,{rand(obsinfo.dimension)})

ans = 2x1 single column vector
   -0.2257
    0.4299

create the dqn agent using critic. for more information, see .

agent = rldqnagent(critic);

check the agent with a random observation input.

getaction(agent,{rand(obsinfo.dimension)})

ans = 1x1 cell array
    {[10]}

specify the dqn agent options, including training options for the critic. alternatively, you can use and rloptimizeroptions objects.

agent.agentoptions.usedoubledqn = false;
agent.agentoptions.targetsmoothfactor = 1;
agent.agentoptions.targetupdatefrequency = 4;
agent.agentoptions.experiencebufferlength = 1e5;
agent.agentoptions.minibatchsize = 256;
agent.agentoptions.criticoptimizeroptions.learnrate = 1e-3;
agent.agentoptions.criticoptimizeroptions.gradientthreshold = 1;

train agent

to train the agent, first specify the training options. for this example, use the following options:

for more information, see rltrainingoptions.

trainopts = rltrainingoptions(...
    maxepisodes=1000, ...
    maxstepsperepisode=500, ...
    verbose=false, ...
    plots="training-progress",...
    stoptrainingcriteria="averagereward",...
    stoptrainingvalue=480); 

you can visualize the cart-pole system by using the plot function during training or simulation.

plot(env)

train the agent using the train function. training this agent is a computationally intensive process that takes several minutes to complete. to save time while running this example, load a pretrained agent by setting dotraining to false. to train the agent yourself, set dotraining to true.

dotraining = false;
if dotraining
    % train the agent.
    trainingstats = train(agent,env,trainopts);
else
    % load the pretrained agent for the example.
    load("matlabcartpoledqnmulti.mat","agent")
end

simulate dqn agent

to validate the performance of the trained agent, simulate it within the cart-pole environment. for more information on agent simulation, see rlsimulationoptions and sim. the agent can balance the cart-pole even when the simulation time increases to 500 steps.

simoptions = rlsimulationoptions(maxsteps=500);
experience = sim(env,agent,simoptions);

totalreward = sum(experience.reward)

totalreward = 500