create and train custom pg agent -凯发k8网页登录

create environment

create the same training environment used in the train reinforcement learning policy using custom training loop example. the environment is a cart-pole balancing environment with a discrete action space. create the environment using the rlpredefinedenv function.

env = rlpredefinedenv("cartpole-discrete");

extract the observation and action specifications from the environment.

obsinfo = getobservationinfo(env);
actinfo = getactioninfo(env);

obtain the dimension of the observation space (numobs) and the number of possible actions (numact).

numobs = obsinfo.dimension(1);
numact = numel(actinfo.elements);

for more information on this environment, see load predefined control system environments.

define policy

the reinforcement learning policy in this example is a parametrized discrete-action stochastic policy, which is learned by a discrete categorical actor. this actor takes an observation as input and returns as output a random action sampled (among the finite number of possible actions) from a categorical probability distribution.

to model the parametrized policy within the actor, use a neural network with one input layer (which receives the content of the environment observation channel, as specified by obsinfo) and one output layer. the output layer must return a vector of probabilities for each possible action, as specified by actinfo.

define the network as an array of layer objects, using fullyconnectedlayer, , and layers. the softmaxlayer ensures that the policy outputs probability values in the range [0 1] and that all probabilities sum to 1.

actornetwork = [
    featureinputlayer(numobs)
    fullyconnectedlayer(24)
    relulayer
    fullyconnectedlayer(24)
    relulayer
    fullyconnectedlayer(2)
    softmaxlayer
    ];

convert to a dlnetwork object and summarize properties.

actornetwork = dlnetwork(actornetwork)

actornetwork = 
  dlnetwork with properties:
         layers: [7x1 nnet.cnn.layer.layer]
    connections: [6x2 table]
     learnables: [6x3 table]
          state: [0x3 table]
     inputnames: {'input'}
    outputnames: {'softmax'}
    initialized: 1
  view summary with summary.

summary(actornetwork)

   initialized: true
   number of learnables: 770
   inputs:
      1   'input'   4 features

create the actor using an object.

actor = rldiscretecategoricalactor(actornetwork,obsinfo,actinfo);

accelerate the gradient computation of the actor.

actor = accelerate(actor,true);

create the optimizer options and rloptimizeroptions function.

actoropts = rloptimizeroptions(learnrate=1e-3);

custom agent class

to define your custom agent, first create a class that is a subclass of the rl.agent.customagent class. the custom agent class for this example is defined in customreinforceagent.m.

the customreinforceagent class has the following class definition, which indicates the agent class name and the associated abstract agent.

classdef customreinforceagent < rl.agent.customagent

to define your agent you must specify the following:

agent properties

in the properties section of the class file, specify any parameters necessary for creating and training the agent.

the rl.agent.customagent class already includes properties for the agent sample time (sampletime) and the action and observation specifications (actioninfo and observationinfo, respectively).

the custom reinforce agent defines the following additional agent properties.

properties
    % actor
    actor
    actoroptimizer
    
    % agent options
    options
    
    % experience buffer
    observationbuffer
    actionbuffer
    rewardbuffer
end
properties (access = private)
    % training utilities
    counter
    numobservation
    numaction
end

constructor function

to create your custom agent, you must define a constructor function. the constructor function performs the following actions.

for example, the customreinforceagent constructor defines action and observation spaces based on the input actor.

function obj = customreinforceagent(actor,options)
    %customreinforceagent construct custom agent
    %   agent = customreinforceagent(actor,options) creates custom
    %   reinforce agent from rlstochasticactorrepresentation actor
    %   and structure options. options has fields:
    %       - discountfactor
    %       - maxstepsperepisode
    
    % (required) call the abstract class constructor.
    obj = obj@rl.agent.customagent();
    obj.observationinfo = actor.observationinfo;
    obj.actioninfo = actor.actioninfo;
    
    % (required for simulink environment) register sample time. 
    % for matlab environment, use -1.
    obj.sampletime = -1;
    
    % (optional) register actor and agent options.
    obj.actor = actor;
    obj.options = options;
    obj.actoroptimizer = rloptimizer(options.optimizeroptions);
    
    % (optional) cache the number of observations and actions.
    obj.numobservation = prod(obj.observationinfo.dimension);
    obj.numaction = prod(obj.actioninfo.dimension);
    
    % (optional) initialize buffer and counter.
    resetimpl(obj);
end

required functions

to create a custom reinforcement learning agent you must define the following implementation functions.

to call these functions in your own code, use the wrapper methods from the abstract base class. for example, to call getactionimpl, use getaction. the wrapper methods have the same input and output arguments as the implementation methods.

getactionimpl function

the getactionimpl function is used to evaluate the policy of your agent and select an action when simulating the agent using the sim function. this function must have the following signature, where obj is the agent object, observation is the current observation, and action is the selected action.

function action = getactionimpl(obj,observation)

for the custom reinforce agent, you select an action by calling the function for the actor. the object generates a discrete distribution from an observation and then samples an action from that distribution.

function action = getactionimpl(obj,observation)
    % compute an action using the policy given the current 
    % observation.
    
    action = getaction(obj.actor,observation);
end

getactionwithexplorationimpl function

the getactionwithexplorationimpl function selects an action using the exploration model of your agent when training the agent using the train function. using this function you can implement exploration techniques such as epsilon-greedy exploration or the addition of gaussian noise. this function must have the following signature, where obj is the agent object, observation is the current observation, and action is the selected action.

function action = getactionwithexplorationimpl(obj,observation)

for the custom reinforce agent, the getactionwithexplorationimpl function is the same as getactionimpl. by default, stochastic actors always explore, that is, they always select an action based on a probability distribution.

function action = getactionwithexplorationimpl(obj,observation)
    % compute an action using the exploration policy given the  
    % current observation.
    
    % reinforce: stochastic actors always explore by default
    % (sample from a probability distribution)
    action = getaction(obj.actor,observation);
end

learnimpl function

the learnimpl function defines how the agent learns from the current experience. this function implements the custom learning algorithm of your agent by updating the policy parameters and selecting an action with exploration for the next state. this function must have the following signature, where obj is the agent object, experience is the current agent experience, and action is the selected action.

function action = learnimpl(obj,experience)

the agent experience is the cell array experience = {observation,action,reward,nextstate,isdone}. here:

function action = learnimpl(obj,experience)
    % define how the agent learns from an experience, which is a
    % cell array with the following format.
    %   experience = ...
    %   {observation,action,reward,nextobservation,isdone}
    
    % reset buffer at the beginning of the episode.
    if obj.counter < 2
        resetbuffer(obj);
    end
    
    % extract data from experience.
    obs = experience{1};
    action = experience{2};
    reward = experience{3};
    nextobs = experience{4};
    isdone = experience{5};
    
    % save data to buffer.
    obj.observationbuffer(:,:,obj.counter) = obs{1};
    obj.actionbuffer(:,:,obj.counter) = action{1};
    obj.rewardbuffer(:,obj.counter) = reward;
    
    if ~isdone
        % choose an action for the next state.
        
        action = getactionwithexplorationimpl(obj, nextobs);
        obj.counter = obj.counter   1;
    else
        % learn from episodic data.
        
        % collect data from the buffer.
        batchsize = min(obj.counter,obj.options.maxstepsperepisode);
        observationbatch = obj.observationbuffer(:,:,1:batchsize);
        actionbatch = obj.actionbuffer(:,:,1:batchsize);
        rewardbatch = obj.rewardbuffer(:,1:batchsize);
        
        % compute the discounted future reward.
        discountedreturn = zeros(1,batchsize);
        for t = 1:batchsize
            g = 0;
            for k = t:batchsize
                g = g   ...
                obj.options.discountfactor ^ (k-t) * rewardbatch(k);
            end
            discountedreturn(t) = g;
        end
        
        % organize data to pass to the loss function.
        lossdata.batchsize = batchsize;
        lossdata.actinfo = obj.actioninfo;
        lossdata.actionbatch = actionbatch;
        lossdata.discountedreturn = discountedreturn;
        
        % compute the gradient of the loss with respect to the
        % actor parameters.
        actorgradient = gradient(obj.actor,@lossfunction,...
            {observationbatch},lossdata);
        
        % update the actor parameters using the computed gradients.
        [obj.actor,obj.actoroptimizer] = update( ...
            obj.actoroptimizer,obj.actor,actorgradient);
        
        % reset the counter.
        obj.counter = 1;
    end
end

for the custom reinforce agent, replicate steps 2 through 7 of the custom training loop in train reinforcement learning policy using custom training loop. you omit steps 1, 8, and 9 since you will use the built-in train function to train your agent.

the actor computes the gradient of the loss function with respect to the parameters. the loss function in the reinforce algorithm the product between the discounted reward and the logarithm of the probability distribution of the action (coming from the policy evaluation for a given observation), summed across all time steps.

this computation is implemented in the local function lossfunction in customreinforceagent.m. the function first input parameter must be a cell array like the one returned from the evaluation of a function approximator object. for more information, see the description of outdata in evaluate. the second, optional, input argument contains additional data that might be needed by the loss calculation function. for more information, see .

function loss = lossfunction(actprobcell,lossdata)
    
    actprob = actprobcell{1};
    % create the action indication matrix.
    batchsize = lossdata.batchsize;
    z = repmat(lossdata.actinfo.elements',1,batchsize);
    actionindicationmatrix = lossdata.actionbatch(:,:) == z;
    
    % resize the discounted return to the size of actprob.
    g = actionindicationmatrix .* lossdata.discountedreturn;
    g = reshape(g,size(actprob));
    
    % round any action probability values less than eps to eps.
    actprob(actprob < eps) = eps;
    
    % compute the loss.
    loss = -sum(g .* log(actprob),'all');
    
end

optional functions

optionally, you can define how your agent is reset at the start of training by specifying a resetimpl function with the following function signature, where obj is the agent object.

function resetimpl(obj)

using this function, you can set the agent into a know or random condition before training.

function resetimpl(obj)
    % (optional) define how the agent is reset before training/
    
    resetbuffer(obj);
    obj.counter = 1;
end

also, you can define any other helper functions in your custom agent class as required. for example, the custom reinforce agent defines a resetbuffer function for reinitializing the experience buffer at the beginning of each training episode.

function resetbuffer(obj)
    % reinitialize observation buffer.
    obj.observationbuffer = zeros( ...
        obj.numobservation, ...
        1, ...
        obj.options.maxstepsperepisode);
    % reinitialize action buffer.
    obj.actionbuffer = zeros(obj.numaction, ...
        1, ...
        obj.options.maxstepsperepisode);
    % reinitialize reward buffer.
    obj.rewardbuffer = zeros(1,obj.options.maxstepsperepisode);
    
end

create custom agent

once you have defined your custom agent class, create an instance of it in the matlab workspace. to create the custom reinforce agent, first specify the agent options.

options.maxstepsperepisode = 250;
options.discountfactor = 0.995;
options.optimizeroptions = actoropts;

then, using the options and the previously defined actor, call the constructor function of the custom agent.

agent = customreinforceagent(actor,options);

train custom agent

configure the training to use the following options.

for more information, see rltrainingoptions.

numepisodes = 5000;
avewindowsize = 100;
trainingterminationvalue = 220;
trainopts = rltrainingoptions(...
    maxepisodes=numepisodes,...
    maxstepsperepisode=options.maxstepsperepisode,...
    scoreaveragingwindowlength=avewindowsize,...
    stoptrainingvalue=trainingterminationvalue);

train the agent using the train function. training this agent is a computationally intensive process that takes several minutes to complete. to save time while running this example, load a pretrained agent by setting dotraining to false. to train the agent yourself, set dotraining to true.

dotraining = false;
if dotraining
    % train the agent.
    trainstats = train(agent,env,trainopts);
else
    % load pretrained agent for the example.
    load("customreinforce.mat","agent");
end

simulate custom agent

enable the environment visualization, which is updated each time the environment step function is called.

plot(env)

to validate the performance of the trained agent, simulate it within the cart-pole environment. for more information on agent simulation, see rlsimulationoptions and sim.

simopts = rlsimulationoptions(maxsteps=options.maxstepsperepisode);
experience = sim(env,agent,simopts);