train off-凯发k8网页登录

to collect training data, first, create an environment.

env = rlpredefinedenv("cartpole-discrete");

create a built-in ppo agent with default networks.

agent1 = rlppoagent( ...
    getobservationinfo(env), ...
    getactioninfo(env));

create a filelogger object.

flgr = rldatalogger;

to log the experiences on disk, assign an appropriate logger function to the logger object. this function is automatically called by the training loop at the end of each episode, and is defined at the end of the example.

flgr.episodefinishedfcn = @myepisodefinishedfcn;

define a training option object to train agent1 for no more than 100 epochs, without visualizing any training progress.

topts = rltrainingoptions(maxepisodes=100,plots="none");

train agent1, logging the experience data.

train(agent1,env,topts,logger=flgr);

at the end of this training, files containing experience data for each episode are saved in the logs folder.

note that the only purpose of training agent1 is to collect experience data from the environment. collecting experiences by simulating the environment in closed loop with a controller (using a for loop), or indeed collecting a series of observations caused by random actions, would also accomplish the same result.

to allow the trainfromdata function to read the experience data stored in the logs folder, create a read function that, given a file name, returns the respective experience structure. for this example, the myreadfcn function is defined at the end of the example.

check that the function can successfully retrieve data from an episode.

cd logs
exp = myreadfcn("loggeddata002")

exp=11×1 struct array with fields:
    nextobservation
    observation
    action
    reward
    isdone

size(cell2mat([exp.action]))

ans = 1×2
     1    11

cd ..

create a filedatastore object using . pass as arguments the name of the folder where files are stored and the read function. the read function is called automatically when the datastore is accessed for reading and is defined at the end of the example.

fds = filedatastore("./logs", "readfcn", @myreadfcn);

create a built-in dqn agent with default networks to be trained from the collected dataset.

agent2 = rldqnagent( ...
    getobservationinfo(env), ...
    getactioninfo(env));

define an options object to train agent2 from data for 50 epochs.

tfdopts = rltrainingfromdataoptions("maxepochs",50);

to train agent2 from data, use trainfromdata. pass the filedatastore object fds as second input argument.

trainfromdata(agent2,fds,tfdopts);

here, the estimated q-value seems to grow indefinitely over time. this often happens during offline training because the agent updates its estimated q-value based on the current estimated q-value, without using any environment feedback. to prevent the q-value from becoming increasingly large (and inaccurate) over time, stop the training earlier or use data regularizer options such as (for dqn or sac agents) or (for ddpg, td3 or sac agents).

in general, the q-value calculated as above for an agent trained offline is not necessarily indicative of the performance of the agent within an environment. therefore, best practice is to validate the agent within an environment after offline training.

function datatolog = myepisodefinishedfcn(data)
    datatolog.experience = data.experience;
end

function experiences = myreadfcn(filename)
if contains(filename,"loggeddata")
    data = load(filename);
    experiences = data.episodedata.experience{1};
else
    experiences = [];
end
end