twin-凯发k8网页登录

create an environment with a continuous action space, and obtain its observation and action specifications. for this example, load the environment used in the example compare ddpg agent to lqr controller. the observation from the environment is a vector containing the position and velocity of a mass. the action is a scalar representing a force, applied to the mass, ranging continuously from -2 to 2 newton.

env = rlpredefinedenv("doubleintegrator-continuous");
obsinfo = getobservationinfo(env);
actinfo = getactioninfo(env);

the agent creation function initializes the actor and critic networks randomly. ensure reproducibility by fixing the seed of the random generator.

rng(0)

create a td3 agent from the environment observation and action specifications.

agent = rltd3agent(obsinfo,actinfo);

to check your agent, use getaction to return the action from a random observation.

getaction(agent,{rand(obsinfo.dimension)})

ans = 1x1 cell array
    {[0.0087]}

you can now test and train the agent within the environment. you can also use and to extract the actor and critic, respectively, and to extract the approximator model (by default a deep neural network) from the actor or critic.

create an environment with a continuous action space and obtain its observation and action specifications. for this example, load the environment used in the example train ddpg agent to swing up and balance pendulum with image observation. this environment has two observations: a 50-by-50 grayscale image and a scalar (the angular velocity of the pendulum). the action is a scalar representing a torque ranging continuously from -2 to 2 nm.

% load predefined environment
env = rlpredefinedenv("simplependulumwithimage-continuous");
% obtain observation and action specifications
obsinfo = getobservationinfo(env);
actinfo = getactioninfo(env);

create an agent initialization option object, specifying that each hidden fully connected layer in the network must have 128 neurons (instead of the default number, 256).

initopts = rlagentinitializationoptions(numhiddenunit=128);

the agent creation function initializes the actor and critic networks randomly. ensure reproducibility by fixing the seed of the random generator.

rng(0)

create a ddpg agent from the environment observation and action specifications.

agent = rltd3agent(obsinfo,actinfo,initopts);

extract the deep neural networks from the actor.

actornet = getmodel(getactor(agent));

extract the deep neural networks from the two critics. note that getmodel(critics) only returns the first critic network.

critics = getcritic(agent);
criticnet1 = getmodel(critics(1));
criticnet2 = getmodel(critics(2));

to verify that each hidden fully connected layer has 128 neurons, you can display the layers on the matlab® command window,

criticnet1.layers

or visualize the structure interactively using analyzenetwork.

analyzenetwork(criticnet1)

plot the networks of the actor and of the second critic, and display the number of weights.

plot(layergraph(actornet))

summary(actornet)

   initialized: true
   number of learnables: 18.9m
   inputs:
      1   'input_1'   50x50x1 images
      2   'input_2'   1 features

plot(layergraph(criticnet2))

summary(criticnet2)

   initialized: true
   number of learnables: 18.9m
   inputs:
      1   'input_1'   50x50x1 images
      2   'input_2'   1 features
      3   'input_3'   1 features

to check your agent, use getaction to return the action from a random observation.

getaction(agent,{rand(obsinfo(1).dimension),rand(obsinfo(2).dimension)})

ans = 1x1 cell array
    {[0.0675]}

you can now test and train the agent within the environment.

create an environment with a continuous action space and obtain its observation and action specifications. for this example, load the environment used in the example compare ddpg agent to lqr controller. the observation from the environment is a vector containing the position and velocity of a mass. the action is a scalar representing a force ranging continuously from -2 to 2 newton.

env = rlpredefinedenv("doubleintegrator-continuous");
obsinfo = getobservationinfo(env);
actinfo = getactioninfo(env);

td3 agents use two q-value function critics. a q-value function critic takes the current observation and an action as inputs and returns a single scalar as output (the estimated discounted cumulative long-term reward for taking the action from the state corresponding to the current observation, and following the policy thereafter).

to model the parametrized q-value function within the critics, use a neural network with two input layers (one for the observation channel, as specified by obsinfo, and the other for the action channel, as specified by actinfo) and one output layer (which returns the scalar value). note that prod(obsinfo.dimension) and prod(actinfo.dimension) return the number of dimensions of the observation and action spaces, respectively, regardless of whether they are arranged as row vectors, column vectors, or matrices.

define each network path as an array of layer objects. assign names to the input and output layers of each path, so you can connect them.

% observation path
obspath = [
    featureinputlayer(prod(obsinfo.dimension),name="obspathin")
    fullyconnectedlayer(32)
    relulayer
    fullyconnectedlayer(16,name="obspathout")
    ];
% action path
actpath = [
    featureinputlayer(prod(actinfo.dimension),name="actpathin")
    fullyconnectedlayer(32)
    relulayer
    fullyconnectedlayer(16,name="actpathout")
    ];
% common path
commonpath = [
    concatenationlayer(1,2,name="concat")
    relulayer
    fullyconnectedlayer(1)
    ];
% add layers to layergraph object
criticnet = layergraph;
criticnet = addlayers(criticnet,obspath);
criticnet = addlayers(criticnet,actpath);
criticnet = addlayers(criticnet,commonpath);
% connect layers
criticnet = connectlayers(criticnet,"obspathout","concat/in1");
criticnet = connectlayers(criticnet,"actpathout","concat/in2");

to initialize the network weights differently for the two critics, create two different dlnetwork objects. you must do this because if the agent constructor function does not accept two identical critics.

criticnet1 = dlnetwork(criticnet);
criticnet2 = dlnetwork(criticnet);

display the number of weights.

summary(criticnet1)

   initialized: true
   number of learnables: 1.2k
   inputs:
      1   'obspathin'   2 features
      2   'actpathin'   1 features

create the two critics using the two networks with different weights. alternatively, if you use exactly the same network with the same weights, you must explicitly initialize the network each time (to make sure weights are initialized differently) before passing it to rlqvaluefunction. to do so, use .

critic1 = rlqvaluefunction(criticnet1,obsinfo,actinfo);
critic2 = rlqvaluefunction(criticnet2,obsinfo,actinfo);

for more information about q-value function approximators, using rlqvaluefunction.

check the critics with a random observation and action input.

getvalue(critic1,{rand(obsinfo.dimension)},{rand(actinfo.dimension)})

ans = single
    -0.1330

getvalue(critic2,{rand(obsinfo.dimension)},{rand(actinfo.dimension)})

ans = single
    -0.1526

td3 agents use a parametrized deterministic policy over continuous action spaces, which is learned by a continuous deterministic actor. this actor takes the current observation as input and returns as output an action that is a deterministic function of the observation.

to model the parametrized policy within the actor, use a neural network with one input layer (which receives the content of the environment observation channel, as specified by obsinfo) and one output layer (which returns the action to the environment action channel, as specified by actinfo).

define the network as an array of layer objects.

actornet = [
    featureinputlayer(prod(obsinfo.dimension))
    fullyconnectedlayer(400)
    relulayer
    fullyconnectedlayer(300)
    relulayer
    fullyconnectedlayer(prod(actinfo.dimension))                       
    tanhlayer
];

convert to dlnetwork and display the number of weights.

actornet = dlnetwork(actornet);
summary(actornet)

   initialized: true
   number of learnables: 121.8k
   inputs:
      1   'input'   2 features

create the actor using actornet, and the environment specification objects.

actor  = rlcontinuousdeterministicactor(actornet,obsinfo,actinfo);

for more information about continuous deterministic actor approximators, see rlcontinuousdeterministicactor.

check the actor with a random observation input.

getaction(actor,{rand(obsinfo.dimension)})

ans = 1x1 cell array
    {[0.0221]}

specify training options for the critics.

criticoptions = rloptimizeroptions( ...
    optimizer="adam", ...
    learnrate=1e-3,...
    gradientthreshold=1, ...
    l2regularizationfactor=2e-4);

specify training options for the actor.

actoroptions = rloptimizeroptions( ...
    optimizer="adam", ...
    learnrate=1e-3,...
    gradientthreshold=1, ...
    l2regularizationfactor=1e-5);

specify agent options, including training options for actor and critics.

agentoptions = rltd3agentoptions;
agentoptions.discountfactor = 0.99;
agentoptions.targetsmoothfactor = 5e-3;
agentoptions.targetpolicysmoothmodel.variance = 0.2;
agentoptions.targetpolicysmoothmodel.lowerlimit = -0.5;
agentoptions.targetpolicysmoothmodel.upperlimit = 0.5;
agentoptions.criticoptimizeroptions = criticoptions;
agentoptions.actoroptimizeroptions = actoroptions;

create td3 agent using actor, critics, and options.

agent = rltd3agent(actor,[critic1 critic2],agentoptions);

you can also create an rltd3agent object with a single critic. in this case, the object represents a ddpg agent with target policy smoothing and delayed policy and target updates.

delayedddpgagent = rltd3agent(actor,critic1,agentoptions);

to check your agents, use getaction to return the action from a random observation.

getaction(agent,{rand(obsinfo.dimension)})

ans = 1x1 cell array
    {[0.0351]}

getaction(delayedddpgagent,{rand(obsinfo.dimension)})

ans = 1x1 cell array
    {[0.0296]}

you can now test and train either agent within the environment.

for this example, load the environment used in the example compare ddpg agent to lqr controller. the observation from the environment is a vector containing the position and velocity of a mass. the action is a scalar representing a force ranging continuously from -2 to 2 newton.

env = rlpredefinedenv("doubleintegrator-continuous");

obtain observation and action specifications.

obsinfo = getobservationinfo(env);
actinfo = getactioninfo(env);

td3 agents use two q-value function critics. to model the parametrized q-value function within the critics, use a recurrent neural network with two input layers (one for the observation channel, as specified by obsinfo, and the other for the action channel, as specified by actinfo) and one output layer (which returns the scalar value).

define each network path as an array of layer objects. to create a recurrent network, use a sequenceinputlayer as the input layer and include at least one lstmlayer.

% define observation path
obspath = [
    sequenceinputlayer(prod(obsinfo.dimension),name="obsin")
    fullyconnectedlayer(400)
    relulayer
    fullyconnectedlayer(300,name="obsout")
    ];
% define action path
actpath = [
    sequenceinputlayer(prod(actinfo.dimension),name="actin")
    fullyconnectedlayer(300,name="actout")
    ];
% define common path
commonpath = [
    concatenationlayer(1,2,name="cat")
    relulayer
    lstmlayer(16);
    fullyconnectedlayer(1)
    ];
% add layers to layergraph object
criticnet = layergraph;
criticnet = addlayers(criticnet,obspath);
criticnet = addlayers(criticnet,actpath);
criticnet = addlayers(criticnet,commonpath);
% connect paths
criticnet = connectlayers(criticnet,"obsout","cat/in1");
criticnet = connectlayers(criticnet,"actout","cat/in2");

to initialize the network weights differently for the two critics, create two different dlnetwork objects. you must do this because the agent constructor function does not accept two identical critics.

criticnet1 = dlnetwork(criticnet);
criticnet2 = dlnetwork(criticnet);

display the number of weights.

summary(criticnet1)

   initialized: true
   number of learnables: 161.6k
   inputs:
      1   'obsin'   sequence input with 2 dimensions
      2   'actin'   sequence input with 1 dimensions

create the critic using the same network structure for both critics. the td3 agent initializes the two networks using different default parameters.

critic1 = rlqvaluefunction(criticnet1,obsinfo,actinfo);
critic2 = rlqvaluefunction(criticnet2,obsinfo,actinfo);

check the critics with a random observation and action input.

getvalue(critic1,{rand(obsinfo.dimension)},{rand(actinfo.dimension)})

ans = single
    -0.0060

getvalue(critic2,{rand(obsinfo.dimension)},{rand(actinfo.dimension)})

ans = single
    0.0481

since the critics has a recurrent network, the (continuous deterministic) actor must have a recurrent network as approximation model too. the network that models the parametrized policy, must take the observation and return the action.

define the network as an array of layer objects.

actornet = [
    sequenceinputlayer(prod(obsinfo.dimension))
    fullyconnectedlayer(400)
    lstmlayer(8)
    relulayer
    fullyconnectedlayer(300,"name","actorfc2")
    relulayer
    fullyconnectedlayer(prod(actinfo.dimension))                       
    tanhlayer
    ];

convert to dlnetwork and display the number of weights.

actornet = dlnetwork(actornet);
summary(actornet)

   initialized: true
   number of learnables: 17.2k
   inputs:
      1   'sequenceinput'   sequence input with 2 dimensions

create the actor using actornet, and the environment specification objects.

actor  = rlcontinuousdeterministicactor(actornet,obsinfo,actinfo);

check the actor with a random observation input.

getaction(actor,{rand(obsinfo.dimension)})

ans = 1x1 cell array
    {[0.0014]}

specify training options for the critics.

criticoptions = rloptimizeroptions( ...
    optimizer="adam", ...
    learnrate=1e-3,...
    gradientthreshold=1, ...
    l2regularizationfactor=2e-4);

specify training options for the actor.

actoroptions = rloptimizeroptions( ...
    optimizer="adam", ...
    learnrate=1e-3,...
    gradientthreshold=1, ...
    l2regularizationfactor=1e-5);

specify agent options, including training options for actor and critics. to use a td3 agent with recurrent neural networks, you must specify a sequencelength greater than 1.

agentoptions = rltd3agentoptions;
agentoptions.discountfactor = 0.99;
agentoptions.sequencelength = 32;
agentoptions.targetsmoothfactor = 5e-3;
agentoptions.targetpolicysmoothmodel.variance = 0.2;
agentoptions.targetpolicysmoothmodel.lowerlimit = -0.5;
agentoptions.targetpolicysmoothmodel.upperlimit = 0.5;
agentoptions.criticoptimizeroptions = criticoptions;
agentoptions.actoroptimizeroptions = actoroptions;

create td3 agent using actor, critics, and options.

agent = rltd3agent(actor,[critic1 critic2],agentoptions);

you can also create an rltd3agent object with a single critic. in this case, the object represents a ddpg agent with target policy smoothing and delayed policy and target updates.

delayedddpgagent = rltd3agent(actor,critic1,agentoptions);

to check your agents, use getaction to return the action from a random observation.

getaction(agent,{rand(obsinfo.dimension)})

ans = 1x1 cell array
    {[0.0018]}

getaction(delayedddpgagent,{rand(obsinfo.dimension)})

ans = 1x1 cell array
    {[0.0015]}

to evaluate the agent using sequential observations, use the sequence length (time) dimension. for example, obtain actions for a sequence of 9 observations.

[action,state] = getaction(agent, ...
    {rand([obsinfo.dimension 1 9])});

display the action corresponding to the seventh element of the observation.

action = action{1};
action(1,1,1,7)

ans = -0.0034

you can now test and train the agent within the environment.