model-凯发k8网页登录
this example shows how to define a custom training loop for a model-based reinforcement learning (mbrl) algorithm. you can use this workflow to train an mbrl policy with your custom training algorithm using policy and value function representations from reinforcement learning toolbox™ software.
for an example on how to use the built in model-based policy optimization (mbpo) agent, see train mbpo agent to balance cart-pole system. for an overview of built-in mbpo agents, see .
in this example, you use transition models to generate more experiences while training a custom dqn [2] agent in a cart-pole environment. the algorithm used in this example is based on an mbpo algorithm [1]. the original mbpo algorithm trains an ensemble of stochastic models and a soft actor-critic (sac) agent in tasks with continuous actions. in contrast, this example trains three deterministic models and a dqn agent in a task with discrete actions. the following figure summarizes the algorithm used in this example.
the agent generates real experiences by interacting with the environment. these experiences are used to train a set of transition models, which are used to generate additional experiences. the training algorithm then uses both the real and generated experiences to update the agent policy.
create environment
for this example, a reinforcement learning policy is trained in a discrete cart-pole environment. the objective in this environment is to balance the pole by applying forces (actions) on the cart. create the environment using the rlpredefinedenv function. fix the random generator seed for reproducibility. for more information on this environment, see load predefined control system environments.
clear
clc
rngseed = 1;
rng(rngseed);
env = rlpredefinedenv("cartpole-discrete");extract the observation and action specifications from the environment.
obsinfo = getobservationinfo(env); actinfo = getactioninfo(env);
obtain the number of observations (numobservations) and actions (numactions).
numobservations = obsinfo.dimension(1); % number of discrete actions, -10 or 10 numactions = numel(actinfo.elements); numcontinuousactions = 1; % force
critic construction
dqn agents use a parametrized q-value function approximator to estimate the value of the policy. since dqn agents have a discrete action space, you have the option to create a vector (that is multi-output) q-value function critic, which is generally more efficient than a comparable single-output critic.
a vector q-value function takes only the observation as input and returns as output a single vector with as many elements as the number of possible actions. the value of each output element represents the expected discounted cumulative long-term reward when an agent starts from the state corresponding to the given observation and executes the action corresponding to the element number (and follows the given policy afterwards).
to model the parametrized q-value function within the critic, use a neural network. for more information about the layers, see fullyconnectedlayer, and .
qnetwork = [
featureinputlayer(obsinfo.dimension(1))
fullyconnectedlayer(24)
relulayer
fullyconnectedlayer(24)
relulayer
fullyconnectedlayer(length(actinfo.elements))
];convert to dlnetwork object.
qnetwork = dlnetwork(qnetwork);
display the number of learnable parameters.
summary(qnetwork)
initialized: true
number of learnables: 770
inputs:
1 'input' 4 features
create the critic representation using the specified neural network and options. for more information, .
critic = rlvectorqvaluefunction(qnetwork,obsinfo,actinfo);
create optimizer objects for updating the critic. for more information, see rloptimizeroptions.
optimizeropt = rloptimizeroptions(... learnrate=1e-3, ... gradientthreshold=1); criticoptimizer = rloptimizer(optimizeropt);
create a max q value policy
policy = rlmaxqpolicy(critic);
create transition models
model-based reinforcement learning uses transition models of the environment. the model usually consists of a transition function, a reward function, and a terminal state function.
the transition function predicts the next observation given the current observation and the action.
the reward function predicts the reward given the current observation, the action, and the next observation.
the terminal state function predicts the terminal state given the observation.
as shown in the following figure, this example uses three transition functions as an ensemble of transition models to generate samples without interacting with the environment. the true reward function and the true terminal state function are given in this example.
define three neural networks for transition models. the neural network predicts the difference between the next observation and the current observation
nummodels = 3; transitionnetwork1 = ... createtransitionnetwork(numobservations, numcontinuousactions); transitionnetwork2 = ... createtransitionnetwork(numobservations, numcontinuousactions); transitionnetwork3 = ... createtransitionnetwork(numobservations, numcontinuousactions); transitionnetworkvector = [transitionnetwork1, ... transitionnetwork2, ... transitionnetwork3];
create experience buffers
create an experience buffer for storing agent experiences (observation, action, next observation, reward, and isdone).
mybuffer.buffersize = 1e5; mybuffer.bufferindex = 0; mybuffer.currentbufferlength = 0; mybuffer.observation = zeros(numobservations,mybuffer.buffersize); mybuffer.nextobservation = ... zeros(numobservations,mybuffer.buffersize); mybuffer.action = ... zeros(numcontinuousactions,1,mybuffer.buffersize); mybuffer.reward = zeros(1,mybuffer.buffersize); mybuffer.isdone = zeros(1,mybuffer.buffersize);
create a model experience buffer for storing the experiences generated by the models.
mymodelbuffer.buffersize = 1e5; mymodelbuffer.bufferindex = 0; mymodelbuffer.currentbufferlength = 0; mymodelbuffer.observation =... zeros(numobservations,mymodelbuffer.buffersize); mymodelbuffer.nextobservation =... zeros(numobservations,mymodelbuffer.buffersize); mymodelbuffer.action = ... zeros(numcontinuousactions,mymodelbuffer.buffersize); mymodelbuffer.reward = zeros(1,mymodelbuffer.buffersize); mymodelbuffer.isdone = zeros(1,mymodelbuffer.buffersize);
configure training
configure the training to use the following options.
maximum number of training episodes — 250
maximum steps per training episode — 500
discount factor — 0.99
training termination condition — average reward across 10 episodes reaches the value of 480
numepisodes = 250; maxstepsperepisode = 500; discountfactor = 0.99; avewindowsize = 10; trainingterminationvalue = 480;
configure the model options.
train transition models only after 2000 samples are collected.
train the models using all experiences in the real experience buffer in each episode. use a mini-batch size of 256.
the models generate trajectories with a length of 2 at the beginning of each episode.
the number of generated trajectories is
numgeneratesampleiterationxnummodelsxminibatchsize= 20 x 3 x 256 = 15360.use the same epsilon-greedy parameters as the dqn agent, except for the minimum epsilon value.
use a minimum epsilon value of 0.1, which is higher than the value used for interacting with the environment. doing so allows the model to generate more diverse data.
warmstartsamples = 2000; numepochs = 1; minibatchsize = 256; horizonlength = 2; epsilonminmodel = 0.1; numgeneratesampleiteration = 20; samplegenerationoptions.horizonlength = horizonlength; samplegenerationoptions.numgeneratesampleiteration = ... numgeneratesampleiteration; samplegenerationoptions.minibatchsize = minibatchsize; samplegenerationoptions.numobservations = numobservations; samplegenerationoptions.epsilonminmodel = epsilonminmodel; % optimizer options velocity1 = []; velocity2 = []; velocity3 = []; decay = 0.01; momentum = 0.9; learnrate = 0.0005;
configure the dqn training options.
use the epsilon greedy algorithm with an initial epsilon value is 1, a minimum value of 0.01, and a decay rate of 0.005.
update the target network every 4 steps.
set the ratio of the real experiences to generated experiences to 0.2:0.8 by setting
realratioto 0.2. settingrealratioto 1.0 is the same as the model-free dqn.take 5 gradient steps at each environment step.
epsilon = 1;
epsilonmin = 0.01;
epsilondecay = 0.005;
targetupdatefrequency = 4;
realratio = 0.2; % set to 1 to run a standard dqn
numgradientsteps = 5;create a vector for storing the cumulative reward for each training episode.
episodecumulativerewardvector = [];
create a figure for model training visualization using the hbuildfiguremodel helper function.
[trainingplotmodel, ... linelosstrain1, ... linelosstrain2, ... linelosstrain3, ... axmodel] = hbuildfiguremodel();
create a figure for model validation visualization using the hbuildfiguremodeltest helper function.
[testplotmodel, linelosstest1, axmodeltest] ...
= hbuildfiguremodeltest();create a figure for dqn agent training visualization using the hbuildfigure helper function.
[trainingplot,linereward,lineavereward, ax] = hbuildfigure;
train agent
train the agent using a custom training loop. the training loop uses the following algorithm. for each episode:
train the transition models.
generate experiences using the transition models and store the samples in the model experience buffer.
generate a real experience. to do so, generate an action using the policy, apply the action to the environment, and obtain the resulting observation, reward, and is-done values.
create a mini-batch by sampling experiences from both the experience buffer and the model experience buffer.
compute the target q value.
compute the gradient of the loss function with respect to the critic representation parameters.
update the critic representation using the computed gradients.
update the training visualization.
terminate training if the critic is sufficiently trained.
training the policy is a computationally intensive process. to save time while running this example, load a pretrained agent by setting dotraining to false. to train the policy yourself, set dotraining to true.
dotrianing = false; if dotrianing targetcritic = critic; modeltrainedatleastonce = false; totalstepct = 0; start = tic; set(trainingplotmodel,visible = "on"); set(testplotmodel,visible = "on"); set(trainingplot,visible = "on"); for episodect = 1:numepisodes if mybuffer.currentbufferlength > minibatchsize && ... totalstepct > warmstartsamples if realratio < 1.0 %---------------------------------------------- % 1. train transition models. %---------------------------------------------- % training three transition models [transitionnetworkvector(1),loss1,velocity1] = ... traintransitionmodel(... transitionnetworkvector(1),... mybuffer,velocity1,minibatchsize,... numepochs,momentum,learnrate); [transitionnetworkvector(2),loss2,velocity2] = ... traintransitionmodel(... transitionnetworkvector(2),... mybuffer,velocity2,minibatchsize,... numepochs,momentum,learnrate); [transitionnetworkvector(3),loss3,velocity3] = ... traintransitionmodel(... transitionnetworkvector(3),... mybuffer,velocity3,minibatchsize,... numepochs,momentum,learnrate); modeltrainedatleastonce = true; % display the training progress d = duration(0,0,toc(start),"format",'hh:mm:ss'); addpoints(linelosstrain1,episodect,loss1) addpoints(linelosstrain2,episodect,loss2) addpoints(linelosstrain3,episodect,loss3) legend(axmodel,"model1","model2","model3"); title(axmodel, ... "model training progress - episode: "... episodect ", elapsed: " string(d)) drawnow %---------------------------------------------- % 2. generate experience using models. %---------------------------------------------- % create numgeneratesampleiteration x % horizonlength xnummodels x minibatchsize % ex) 20 x 2 x 3 x 256 = 30720 samples mymodelbuffer = generatesamples(mybuffer,... mymodelbuffer,... transitionnetworkvector,policy,actinfo,... epsilon,samplegenerationoptions); end end %---------------------------------------------- % interact with environment and train agent. %---------------------------------------------- % reset the environment at the start of the episode observation = reset(env); episodereward = zeros(maxstepsperepisode,1); errorpreddiction = zeros(maxstepsperepisode,1); for stepct = 1:maxstepsperepisode %---------------------------------------------- % 3. generate an experience. %---------------------------------------------- totalstepct = totalstepct 1; % compute an action using the policy based on % the current observation. if rand() < epsilon action = actinfo.usample; else action = getaction(policy,{observation}); end action = action{1}; % udpate epsilon if totalstepct > warmstartsamples epsilon = max(epsilon*(1-epsilondecay),... epsilonmin); end % apply the action to the environment and obtain the % resulting observation and reward. [nextobservation,reward,isdone] = step(env,action); % check prediction dx = predict(transitionnetworkvector(1),... dlarray(observation,"cb"),dlarray(action,"cb")); predictednextobservation = observation dx; errorpreddiction(stepct) = ... sqrt(sum((nextobservation - ... predictednextobservation).^2)); % store the action, observation, reward and is-done % experience mybuffer = storeexperience(mybuffer,... observation,... action,... nextobservation,reward,isdone); episodereward(stepct) = reward; observation = nextobservation; % train dqn agent for gradientct = 1:numgradientsteps if mybuffer.currentbufferlength >= minibatchsize ... && totalstepct>warmstartsamples %---------------------------------------------- % 4. sample minibatch from experience buffers. %---------------------------------------------- [sampledobservation,... sampledaction,... samplednextobservation,... sampledreward,... sampledisdone] ... = sampleminibatch(... modeltrainedatleastonce,... realratio,... minibatchsize,... mybuffer,mymodelbuffer); %---------------------------------------------- % 5. compute target q value. %---------------------------------------------- % compute target q value [targetqvalues, maxactionindices] = ... getmaxqvalue(targetcritic, ... {reshape(samplednextobservation,... [numobservations,1,minibatchsize])}); % compute target for nonterminal states targetqvalues(~logical(sampledisdone)) = ... sampledreward(~logical(sampledisdone)) ... discountfactor.*... targetqvalues(~logical(sampledisdone)); % compute target for terminal states targetqvalues(logical(sampledisdone)) = ... sampledreward(logical(sampledisdone)); lossdata.batchsize = minibatchsize; lossdata.actinfo = actinfo; lossdata.actionbatch = sampledaction; lossdata.targetqvalues = targetqvalues; %---------------------------------------------- % 6. compute gradients. %---------------------------------------------- criticgradient = ... gradient(critic,... @criticlossfunction, ... {reshape(sampledobservation,... [numobservations,1,minibatchsize])},... lossdata); %---------------------------------------------- % 7. update the critic network using gradients. %---------------------------------------------- [critic, criticoptimizer] = update(... criticoptimizer, critic,... criticgradient); % update the policy parameters using the critic % parameters. policy = setlearnableparameters(... policy,... getlearnableparameters(critic)); end end % update target critic periodically if mod(totalstepct, targetupdatefrequency)==0 targetcritic = critic; end % stop if a terminal condition is reached. if isdone break; end end % end of episode %--------------------------------------------------------- % 8. update the training visualization. %--------------------------------------------------------- episodecumulativereward = sum(episodereward); episodecumulativerewardvector = cat(2,... episodecumulativerewardvector,episodecumulativereward); movingavereward = movmean(episodecumulativerewardvector,... avewindowsize,2); addpoints(linereward,episodect,episodecumulativereward); addpoints(lineavereward,episodect,movingavereward(end)); title(ax, "training progress - episode: " episodect ... ", total step: " string(totalstepct) ... ", epsilon:" string(epsilon)) drawnow; errorpreddiction = errorpreddiction(1:stepct); % display one step prediction error. addpoints(linelosstest1,episodect,mean(errorpreddiction)) legend(axmodeltest,"model1"); title(axmodeltest, ... "model one-step prediction error - episode: " ... episodect ", error: " ... string(mean(errorpreddiction))) drawnow % display training progress every 10th episode if (mod(episodect,10) == 0) fprintf("ep:%d, reward:%.4f, avereward:%.4f, " ... "steps:%d, totalsteps:%d, epsilon:%f," ... "error model:%f\n",... episodect, ... episodecumulativereward,... movingavereward(end),... stepct,totalstepct,... epsilon,... mean(errorpreddiction)) end %--------------------------------------------------------- % 9. terminate training % if the network is sufficiently trained. %--------------------------------------------------------- if max(movingavereward) > trainingterminationvalue break end end else load("cartpolemodelbasedcustomlooppolicy.mat"); end
simulate agent
to simulate the trained agent, first reset the environment.
obs0 = reset(env); obs = obs0;
enable the environment visualization, which is updated each time the environment step function is called.
plot(env)
for each simulation step, perform the following actions.
get the action by sampling from the policy using the
getactionfunction.step the environment using the obtained action value.
terminate if a terminal condition is reached.
actionvector = zeros(1,maxstepsperepisode); obsvector = zeros(numobservations,maxstepsperepisode 1); obsvector(:,1) = obs0; for stepct = 1:maxstepsperepisode % select action according to trained policy. action = getaction(policy,{obs}); action= action{1}; % step the environment. [nextobs,reward,isdone] = step(env,action); obsvector(:,stepct 1) = nextobs; actionvector(1,stepct) = action; % check for terminal condition. if isdone break end obs = nextobs; end
laststepct = stepct;
test model
test one of the models by predicting a next observation given a current observation and an action.
modelid = 3; predictedobsvector = zeros(numobservations,laststepct); obs = dlarray(obsvector(:,1),"cb"); predictedobsvector(:,1) = obs; for stepct = 1:laststepct obs = dlarray(obsvector(:,stepct),"cb"); action = dlarray(actionvector(1,stepct),"cb"); dx = predict(transitionnetworkvector(modelid),obs, action); predictedobs = obs dx; predictedobsvector(:,stepct 1) = predictedobs; end predictedobsvector = predictedobsvector(:, 1:laststepct); figure(5) layout = tiledlayout(4,1, "tilespacing", "compact"); for i = 1:4 nexttile; errorprediction = abs(predictedobsvector(i,1:laststepct) - ... obsvector(i,1:laststepct)); line1 = plot(errorprediction,"displayname", "absolute error"); title("observation " num2str(i)); end title(layout,"prediction absolute error")
the small absolute prediction error shows that the model is successfully trained to predict the next observation.
references
[1] minh, volodymyr, koray kavukcuoglu, david silver, alex graves, ioannis antonoglou, daan wierstra, and martin riedmiller. “playing atari with deep reinforcement learning.” preprint, submitted december 19, 2013. .
[2] janner, michael, justin fu, marvin zhang, and sergey levine. "when to trust your model: model-based policy optimization." preprint, submitted november 5, 2019. .
see also
functions
objects
related examples
- train reinforcement learning agents
- create and train custom pg agent
- train mbpo agent to balance cart-pole system
more about
select a web site
choose a web site to get translated content where available and see local events and offers. based on your location, we recommend that you select: .
you can also select a web site from the following list:
americas
- (español)
- (english)
- (english)
europe
- (english)
- (english)
- (deutsch)
- (español)
- (english)
- (français)
- (english)
- (italiano)
- (english)
- (english)
- (english)
- (deutsch)
- (english)
- (english)
- switzerland
- (english)
asia pacific
- (english)
- (english)
- (english)
- 中国
- (日本語)
- (한국어)