Skip to main content

During training

During training, the model outputs different metrics and plots, as well as loss calculated on the test set. These can be used to evaluate the model.

Which metrics and plots are output is decided by the evaluators used. There are evaluators implemented and selected by default for most decoders, but these defaults may be overridden. For details please refer to the evaluators page.

Whether training or test data is used to calculate the metrics and create the plots can be set using the evaluation_set_for_metrics and evaluation_set_for_plots parameters (options: "train", "test", "both").

danger

If you want to use optuna hyperparameter search or early stopping, metrics need to be evaluated on the test set! Also make sure to set test_timeseries_indices!

Which timeseries are used for the metrics and plots are set using the evaluation_test_timeseries_indices and evaluation_train_timeseries_indices parameters. The expected type is the same as for test_timeseries_indices, so please refer to that part of the documentation for details. If not set for the test parameter, all test_timeseries_indices are used. If not set for the train parameter, all timeseries not used for testing are used.

warning

If you have many timeseries and don't set evaluation_train_timeseries_indices, creating plots and metrics can take a long time, as the model will try to create plots for all timeseries. It is advised to set the evaluation_train_timeseries_indices parameter to a small, but still representative subset of the timeseries.

Cheap and expensive saving intervals​

The config defines two saving intervals cheap_saving_intervals, expensive_saving_intervals. These are intervals of epochs at which cheaply and expensively obtainable data about the model is saved.

Cheap saving​

At the cheap saving intervals the following data is saved:

  • Learning rates: The learning rates of all parts of the model.
  • Value scheduler: The values of all scheduled scalars.
  • Loss: The training loss at the current epoch.
  • Gradient norms: The current gradient norms during model training.
  • Median maximum eigenvalue of AA: For each AA matrix of the hierarchised model the eigenvalues are calculated, then the median of the maximal eigenvalues are taken. Can be used to evaluate the effectiveness of the MAR.

Expensive saving​

At the expensive saving intervals the following data is saved:

  • Metrics calculated by the evaluators
  • Plots made by the evaluators
  • Test loss
  • The model parameters at the given epoch. This is saved to the model directory as model_<epoch_number>.pt.

Integration with TensorBoard​

The aforementioned results and visualization outputs of the model during training can be accessed by running tensorboard in the respective logdir.

tensorboard --logdir <log_directory>

Where log_directory is results_dir/experiment_name.

TensorBoard is a tool for visualizing plots and metrics during the traning progress.