train ac agent to balance cart-凯发k8网页登录

cart-pole matlab environment

the reinforcement learning environment for this example is a pole attached to an unactuated joint on a cart, which moves along a frictionless track. the training goal is to make the pendulum stand upright without falling over.

for this environment:

for more information on this model, see load predefined control system environments.

create environment interface

create a predefined environment interface for the pendulum.

env = rlpredefinedenv("cartpole-discrete")

env = 
  cartpolediscreteaction with properties:
                  gravity: 9.8000
                 masscart: 1
                 masspole: 0.1000
                   length: 0.5000
                 maxforce: 10
                       ts: 0.0200
    thetathresholdradians: 0.2094
               xthreshold: 2.4000
      rewardfornotfalling: 1
        penaltyforfalling: -5
                    state: [4x1 double]

env.penaltyforfalling = -10;

the interface has a discrete action space where the agent can apply one of two possible force values to the cart, –10 or 10 n.

obtain the observation and action information from the environment interface.

obsinfo = getobservationinfo(env);
actinfo = getactioninfo(env);

fix the random generator seed for reproducibility.

rng(0)

create ac agent

an ac agent approximates the discounted cumulative long-term reward using a value-function critic. a value-function critic must accept an observation as input and return a single scalar (the estimated discounted cumulative long-term reward) as output.

to approximate the value function within the critic, use a neural network. define the network as an array of layer objects, and get the dimension of the observation space and the number of possible actions from the environment specification objects. for more information on creating a deep neural network value function representation, see create policies and value functions.

criticnet = [
    featureinputlayer(obsinfo.dimension(1))
    fullyconnectedlayer(32)
    relulayer
    fullyconnectedlayer(1)];

convert to dlnetwork and display the number of weights.

criticnet = dlnetwork(criticnet);
summary(criticnet)

   initialized: true
   number of learnables: 193
   inputs:
      1   'input'   4 features

create the critic approximator object using criticnet, and the observation specification. for more information, see .

critic = rlvaluefunction(criticnet,obsinfo);

check the critic with a random observation input.

getvalue(critic,{rand(obsinfo.dimension)})

ans = single
    -0.3590

an ac agent decides which action to take using a stochastic policy, which for discrete action spaces is approximated by a discrete categorical actor. this actor must take the observation signal as input and return a probability for each action.

to approximate the policy function within the actor, use a deep neural network. define the network as an array of layer objects, and get the dimension of the observation space and the number of possible actions from the environment specification objects.

actornet = [
    featureinputlayer(obsinfo.dimension(1))
    fullyconnectedlayer(32)
    relulayer
    fullyconnectedlayer(numel(actinfo.elements))
    softmaxlayer];

convert to dlnetwork and display the number of weights.

actornet = dlnetwork(actornet);
summary(actornet)

   initialized: true
   number of learnables: 226
   inputs:
      1   'input'   4 features

create the actor approximator object using actornet and the observation and action specifications. for more information, see .

actor = rldiscretecategoricalactor(actornet,obsinfo,actinfo);

to return the probability distribution of the possible actions as a function of a random observation, and given the current network weights, use evaluate.

prb = evaluate(actor,{rand(obsinfo.dimension)})

prb = 1x1 cell array
    {2x1 single}

prb{1}

ans = 2x1 single column vector
    0.4414
    0.5586

create the agent using the actor and critic. for more information, see rlacagent.

agent = rlacagent(actor,critic);

check the agent with a random observation input.

getaction(agent,{rand(obsinfo.dimension)})

ans = 1x1 cell array
    {[-10]}

specify agent options, including training options for the actor and critic, using dot notation. alternatively, you can use rlacagentoptions and rloptimizeroptions objects before creating the agent.

agent.agentoptions.entropylossweight = 0.01;
agent.agentoptions.actoroptimizeroptions.learnrate = 1e-2;
agent.agentoptions.actoroptimizeroptions.gradientthreshold = 1;
agent.agentoptions.criticoptimizeroptions.learnrate = 1e-2;
agent.agentoptions.criticoptimizeroptions.gradientthreshold = 1;

train agent

to train the agent, first specify the training options. for this example, use the following options.

for more information, see rltrainingoptions.

trainopts = rltrainingoptions(...
    maxepisodes=1000,...
    maxstepsperepisode=500,...
    verbose=false,...
    plots="training-progress",...
    stoptrainingcriteria="averagereward",...
    stoptrainingvalue=480,...
    scoreaveragingwindowlength=10);

you can visualize the cart-pole system during training or simulation using the plot function.

plot(env)

train the agent using the train function. training this agent is a computationally intensive process that takes several minutes to complete. to save time while running this example, load a pretrained agent by setting dotraining to false. to train the agent yourself, set dotraining to true.

dotraining = false;
if dotraining    
    % train the agent.
    trainingstats = train(agent,env,trainopts);
else
    % load the pretrained agent for the example.
    load("matlabcartpoleac.mat","agent");
end

simulate ac agent

to validate the performance of the trained agent, simulate it within the cart-pole environment. for more information on agent simulation, see rlsimulationoptions and sim.

simoptions = rlsimulationoptions(maxsteps=500);
experience = sim(env,agent,simoptions);

totalreward = sum(experience.reward)

totalreward = 500