(not recommended) options set for reinforcement learning agent representations (critics and actors) -凯发k8网页登录
(not recommended) options set for reinforcement learning agent representations (critics and actors)
since r2019a
rlrepresentationoptions
is not recommended. use an rloptimizeroptions
object within an agent options object instead. for more information, see rlrepresentationoptions is not recommended.
description
use an rlrepresentationoptions
object to specify an options set for critics (,
)
and actors (, ).
creation
description
creates a
default option set to use as a last argument when creating a reinforcement learning actor
or critic. you can modify the object properties using dot notation.repopts
= rlrepresentationoptions
creates an options set with the specified properties using one or more
name-value pair arguments.repopts
= rlrepresentationoptions(name,value
)
properties
learnrate
— learning rate for the representation
0.01
(default) | positive scalar
learning rate for the representation, specified as a positive scalar. if the learning rate is too low, then training takes a long time. if the learning rate is too high, then training might reach a suboptimal result or diverge.
example: 'learnrate',0.025
optimizer
— optimizer for representation
"adam"
(default) | "sgdm"
| "rmsprop"
optimizer for training the network of the representation, specified as one of the following values.
"adam"
— use the adam optimizer. you can specify the decay rates of the gradient and squared gradient moving averages using thegradientdecayfactor
andsquaredgradientdecayfactor
fields of theoptimizerparameters
option."sgdm"
— use the stochastic gradient descent with momentum (sgdm) optimizer. you can specify the momentum value using themomentum
field of theoptimizerparameters
option."rmsprop"
— use the rmsprop optimizer. you can specify the decay rate of the squared gradient moving average using thesquaredgradientdecayfactor
fields of theoptimizerparameters
option.
for more information about these optimizers, see in the algorithms section of in deep learning toolbox™.
example: 'optimizer',"sgdm"
optimizerparameters
— applicable parameters for optimizer
optimizerparameters
object
applicable parameters for the optimizer, specified as an
optimizerparameters
object with the following parameters.
parameter | description |
---|---|
momentum | contribution of previous step, specified as a scalar from 0 to 1. a value of 0 means no contribution from the previous step. a value of 1 means maximal contribution. this parameter applies only when
|
epsilon | denominator offset, specified as a positive scalar. the optimizer adds this offset to the denominator in the network parameter updates to avoid division by zero. this parameter applies only when
|
gradientdecayfactor | decay rate of gradient moving average, specified as a positive scalar from 0 to 1. this parameter applies only when
|
squaredgradientdecayfactor | decay rate of squared gradient moving average, specified as a positive scalar from 0 to 1. this parameter applies only when
|
when a particular property of optimizerparameters
is not
applicable to the optimizer type specified in the optimizer
option,
that property is set to "not applicable"
.
to change the default values, create an rlrepresentationoptions
set
and use dot notation to access and change the properties of
optimizerparameters
.
repopts = rlrepresentationoptions; repopts.optimizerparameters.gradientdecayfactor = 0.95;
gradientthreshold
— threshold value for gradient
inf
(default) | positive scalar
threshold value for the representation gradient, specified as inf
or a positive scalar. if the gradient exceeds this value, the gradient is clipped as
specified by the gradientthresholdmethod
option. clipping the
gradient limits how much the network parameters change in a training iteration.
example: 'gradientthreshold',1
gradientthresholdmethod
— gradient threshold method
"l2norm"
(default) | "global-l2norm"
| "absolute-value"
gradient threshold method used to clip gradient values that exceed the gradient threshold, specified as one of the following values.
"l2norm"
— if the l2 norm of the gradient of a learnable parameter is larger thangradientthreshold
, then scale the gradient so that the l2 norm equalsgradientthreshold
."global-l2norm"
— if the global l2 norm, l, is larger thangradientthreshold
, then scale all gradients by a factor ofgradientthreshold/
l. the global l2 norm considers all learnable parameters."absolute-value"
— if the absolute value of an individual partial derivative in the gradient of a learnable parameter is larger thangradientthreshold
, then scale the partial derivative to have magnitude equal togradientthreshold
and retain the sign of the partial derivative.
for more information, see in the algorithms section of in deep learning toolbox.
example: 'gradientthresholdmethod',"absolute-value"
l2regularizationfactor
— factor for l2 regularization
0.0001 (default) | nonnegative scalar
factor for l2 regularization (weight decay), specified as a nonnegative scalar. for more information, see in the algorithms section of in deep learning toolbox.
to avoid overfitting when using a representation with many parameters, consider
increasing the l2regularizationfactor
option.
example: 'l2regularizationfactor',0.0005
usedevice
— computation device for training
"cpu"
(default) | "gpu"
computation device used to perform deep neural network operations such as gradient
computation, parameter update and prediction during training. it is specified as either
"cpu"
or "gpu"
.
the "gpu"
option requires both parallel computing toolbox™ software and a cuda® enabled nvidia® gpu. for more information on supported gpus see gpu computing requirements (parallel computing toolbox).
you can use gpudevice
(parallel computing toolbox) to query or select a local gpu device to be used
with matlab®.
note
training or simulating an agent on a gpu involves device-specific numerical round off errors. these errors can produce different results compared to performing the same operations a cpu.
note that if you want to use parallel processing to speed up training, you do not
need to set usedevice
. instead, when training your agent, use an
rltrainingoptions
object in which the useparallel
option is set to
true
. for more information about training using multicore
processors and gpus for training, see train agents using parallel computing and gpus.
example: 'usedevice',"gpu"
object functions
(not recommended) value function critic representation for reinforcement learning agents | |
(not recommended) q-value function critic representation for reinforcement learning agents | |
(not recommended) deterministic actor representation for reinforcement learning agents | |
(not recommended) stochastic actor representation for reinforcement learning agents |
examples
configure options for creating representation
create an options set for creating a critic or actor representation for a reinforcement learning agent. set the learning rate for the representation to 0.05, and set the gradient threshold to 1. you can set the options using name,value pairs when you create the options set. any options that you do not explicitly set have their default values.
repopts = rlrepresentationoptions(learnrate=5e-2,...
gradientthreshold=1)
repopts = rlrepresentationoptions with properties: learnrate: 0.0500 gradientthreshold: 1 gradientthresholdmethod: "l2norm" l2regularizationfactor: 1.0000e-04 usedevice: "cpu" optimizer: "adam" optimizerparameters: [1x1 rl.option.optimizerparameters]
alternatively, create a default options set and use dot notation to change some of the values.
repopts = rlrepresentationoptions; repopts.learnrate = 5e-2; repopts.gradientthreshold = 1
repopts = rlrepresentationoptions with properties: learnrate: 0.0500 gradientthreshold: 1 gradientthresholdmethod: "l2norm" l2regularizationfactor: 1.0000e-04 usedevice: "cpu" optimizer: "adam" optimizerparameters: [1x1 rl.option.optimizerparameters]
if you want to change the properties of the optimizerparameters
option, use dot notation to access them.
repopts.optimizerparameters.epsilon = 1e-7; repopts.optimizerparameters
ans = optimizerparameters with properties: momentum: "not applicable" epsilon: 1.0000e-07 gradientdecayfactor: 0.9000 squaredgradientdecayfactor: 0.9990
version history
introduced in r2019ar2022a: rlrepresentationoptions
is not recommended
rlrepresentationoptions
objects are no longer recommended. to specify
optimization options for actors and critics, use rloptimizeroptions
objects instead.
specifically, you can create an agent options object and set its
criticoptimizeroptions
and actoroptimizeroptions
properties to suitable rloptimizeroptions
objects. then you pass the agent
options object to the function that creates the agent. this workflow is shown in the
following table.
rlrepresentationoptions : not recommended | rloptimizeroptions : recommended |
---|---|
crtopts = rlrepresentationoptions(... 'gradientthreshold',1); critic = rlvaluerepresentation(... net,obsinfo,'observation',{'obs'},ctropts) |
criticopts = rloptimizeroptions(... 'gradientthreshold',1); agentopts = rlacagentoptions(... 'criticoptimizeroptions',crtopts); agent = rlacagent(actor,critic,agentopts) |
alternatively, you can create the agent and then use dot notation to access the
optimization options for the agent actor and critic, for example:
agent.agentoptions.actoroptimizeroptions.gradientthreshold = 1;
.
see also
打开示例
您曾对此示例进行过修改。是否要打开带有您的编辑的示例?
matlab 命令
您点击的链接对应于以下 matlab 命令:
请在 matlab 命令行窗口中直接输入以执行命令。web 浏览器不支持 matlab 命令。
select a web site
choose a web site to get translated content where available and see local events and offers. based on your location, we recommend that you select: .
you can also select a web site from the following list:
how to get best site performance
select the china site (in chinese or english) for best site performance. other mathworks country sites are not optimized for visits from your location.
americas
- (español)
- (english)
- (english)
europe
- (english)
- (english)
- (deutsch)
- (español)
- (english)
- (français)
- (english)
- (italiano)
- (english)
- (english)
- (english)
- (deutsch)
- (english)
- (english)
- switzerland
- (english)
asia pacific
- (english)
- (english)
- (english)
- 中国
- (日本語)
- (한국어)