main content

log reinforcement learning training data to monitor window -凯发k8网页登录

log reinforcement learning training data to monitor window

since r2022b

description

use a monitorlogger object to log data to a monitor window, within the train function or inside a custom training loop. to log data when using the train function, specify appropriate callback functions in monitorlogger, as shown in the examples. these callbacks are executed at different stages of training, for example, episodefinishedfcn is executed after the completion of an episode. the output of a callback function is a structure containing the data to log at that stage of training.

note

using a monitorlogger object to log data when using the train function does not affect (and is not affected by) any option to save agents during training specified within an rltrainingoptions object.

note

monitorlogger is an handle object. if you assign an existing monitorlogger object to a new monitorlogger object, both the new object and the original one refer to the same underlying object in memory. to preserve the original object parameters for later use, save the object to a mat-file. for more information about handle objects, see .

creation

create a monitorlogger object using rldatalogger specifying a trainingprogressmonitor object as input argument.

properties

object containing a set of logging options, returned as a monitorloggingoptions object. a monitorloggingoptions object has the following properties, which you can access using dot notation after creating the monitorlogger object.

monitor data write period, specified as a positive integer. it is the number of episodes after which data is transmitted to the trainingprogressmonitor object. for example, if datawritefrequency is 5 then data from episodes 1 to 5 will be cached in memory and be transmitted to the monitor object at the end of the 5-th episode. this improves performance in some cases. the default is 1.

example: datawritefrequency=10

maximum number of episodes, specified as a positive integer. when using train, the value is automatically initialized. set this value when using the logger object in a custom training loop. the default is 500.

example: maxepisodes=1000

callback to log data after episode completion, specified as a function handle object. the specified function is automatically called by the training loop at the end of each episode, and must return a structure containing the data to log, such as experiences, simulation information, or initial observations.

your function must have the following signature.

function datatolog = myepisodefinishedfcn(data)

here, data is a structure that contains the following fields:

  • episodecount — current episode number

  • environment — environment object

  • agent — agent object

  • experience — structure array containing the experiences. each element of this array corresponds to a step and is a structure containing the fields nextobservation, observation, action, reward and isdone.

  • episodeinfo — structure containing the fields cumulativereward, stepstaken and initialobservation.

  • simulationinfo — contains simulation information from the episode. for matlab environments this is a structure with the field simulationerror, and for simulink® environments it is a simulink.simulationoutput object.

the function output datatolog is the structure containing the data to be logged to disk.

example: episodefinishedfcn=@myeploggingfcn

callback to log data after training step completion within an episode, specified as a function handle object. the specified function is automatically called by the training loop at the end of each training step, and must return a structure containing the data to log, such as for example the state of the agent's exploration policy.

your function must have the following signature.

function datatolog = myagentstepfinishedfcn(data)

here, data is a structure that contains the following fields:

  • episodecount — current episode number

  • agentstepcount — cumulative number of steps taken by the agent

  • simulationtime — current simulation time in the environment

  • agent — agent object

the function output datatolog is the structure containing the data to be logged to disk.

for multi agent training, agentstepfinishedfcn can be a cell array of function handles with as many elements as number of agent groups.

note

logging data using the agentstepfinishedfcn callback is not supported when training agents in parallel with the train function.

example: agentstepfinishedfcn=@myagtsteploggingfcn

callback to log data after completion of the learn subroutine, specified as a function handle object. the specified function is automatically called by the training loop at the end of each learning subroutine, and must return a structure containing the data to log, such as the training losses of the actor and critic networks, or, for a model-based agent, the environment model training losses.

your function must have the following signature.

function datatolog = myagentlearnfinishedfcn(data)

here, data is a structure that contains the following fields:

  • episodecount — current episode number

  • agentstepcount — cumulative number of steps taken by the agent

  • agentlearncount — cumulative number of learning steps taken by the agent

  • envmodeltraininginfo — structure containing model-based agents related fields transitionfcnloss, rewardfcnloss and isdonefcnloss.

  • agent — agent object

  • actorloss — loss of the actor

  • criticloss — loss of the critic

the function output datatolog is the structure containing the data to be logged to disk.

for multi agent training, agentlearnfinishedfcn can be a cell array of function handles with as many elements as number of agent groups.

example: agentlearnfinishedfcn=@myagtlearnloggingfcn

object functions

setupset up reinforcement learning environment or initialize data logger object
cleanupclean up reinforcement learning environment or data logger object

examples

this example shows how to log and visualize data to the window of a trainingprogressmonitor object when using train.

create a trainingprogressmonitor object. creating the object also opens a window associated with the object.

monitor = trainingprogressmonitor();

create a monitorlogger object using rldatalogger.

logger = rldatalogger(monitor);

create callback functions to log the data (for this example, see the helper function section), and specify the appropriate callback functions in the logger object. for the specific function signatures and more information on the function input structure, see the corresponding properties of monitorlogger.

logger.agentlearnfinishedfcn = @myagentlearnfinishedfcn;

to train the agent, you can now call train, passing logger as an argument such as in the following command.

trainresult = train(agent, env, trainopts, logger=logger);

while the training progresses, data will be logged to the training monitor object, and visualized in the associated window.

note that only scalar data can be logged with a monitor logger object.

example logging functions

define a logging function that logs data periodically at the completion of the learning subroutine. this function is automatically called by the training loop at the end of each learning subroutine, and must return a structure containing the learning-related data to log, such as the training losses of the actor and critic networks, or, for a model-based agent, the environment model training losses.

function datatolog = myagentlearnfinishedfcn(data)
    if mod(data.agentlearncount, 2) == 0
        datatolog.actorloss  = data.actorloss;
        datatolog.criticloss = data.criticloss;
    else
        datatolog = [];
    end
    
end

limitations

  • only scalar data is supported when logging data with a monitorlogger object. the structure returned by the callback functions must contain fields with scalar data.

  • resuming of training from a previous training result is not supported when logging data with a monitorlogger object.

version history

introduced in r2022b

see also

functions

objects

topics

网站地图