Early stopping

Early stopping allows the training to terminate automatically when a monitored metric (or loss) stops improving. This prevents overfitting and saves computational resources.

The configuration consists of two parts: a common configuration (shared with Optuna pruning/reporting) and a specific configuration for the stopping criteria.

Configuration

Common Config (early_stopping_optuna_common_config)

Defines what to monitor. This is used by both the Early Stopping mechanism and the Optuna reporter (to report intermediate values).

• metric_name_pattern: (Optional) A string or regex to match a specific metric name (e.g., "/R_2/") evaluated on the test set. To use regex, start and end the string with forward slashes ("/"). If null (default), the Test Loss is used.
• direction: "min" (lower is better, e.g., loss) or "max" (higher is better, e.g., R² score). Default: "min".
• loss_term_scales: (Used only if monitoring Loss, i.e., metric_name_pattern is null) A dictionary defining which test loss terms to include and their weights.
  • Default: {"dsr": 1.0} (Monitors only the DSR component of the test loss).
  • Example: {"dsr": 1.0, "reconstruction": 1.0} sums both terms.
• metric_reduction_method: (Used only if monitoring Metrics) How to aggregate the metrics if multiple match the given pattern. Options: "mean", "sum", "max", "min", "median". Default: "mean".

"early_stopping_optuna_common_config": {
    "metric_name_pattern": null,    # Monitor Test Loss
    "direction": "min",             # Minimize Loss
    "loss_term_scales": {
        "dsr": 1.0                  # Only consider DSR loss term (Default)
    }
}

note

The default metric used by early stopping and optuna is the DSR test loss. This is chosen over the total loss as it captures the performance of the DSR model best. This is in contrast with regularization loss terms and reconstruction loss terms which measure different capabilities of the model. Visit the related documentation for further details on the loss terms.

warning

When setting the metric_name_pattern make sure to not accidentally use metrics calculated on the train set. The evaluation set used (train or test) is marked in the metric's name after the evaluator name, e.g. GaussianEvaluator_test_D_stsp. For further details please refer to the evaluators page.

Specific Config (early_stopping_config)

Defines when to stop. Note that if this section is present, early_stopping_optuna_common_config must also be provided.

• patience: Number of epochs with no improvement after which training stops.
  • Constraint: Must be greater than or equal to expensive_saving_interval, as the test loss is typically calculated at this interval.
• min_delta: The minimum change to qualify as an improvement.
  • value: The numeric threshold.
  • relative: If true, value is treated as a percentage/fraction of the best observed value.

"early_stopping_config": {
    "patience": 500,
    "min_delta": {
        "value": 0.01,   # 1% improvement required
        "relative": true
    }
}

Example Configuration

Here is a complete example setup in your config dictionary:

{
    "expensive_saving_interval": 100,
    
    # Define what to watch (e.g., Total Loss)
    "early_stopping_optuna_common_config": {
        "metric_name_pattern": null, 
        "direction": "min",
        "loss_term_scales": {
             "dsr": 1.0,
             "reconstruction": 1.0
        }
    },
    
    # Define stopping rules
    "early_stopping_config": {
        "patience": 200,
        "min_delta": { "value": 1e-4, "relative": false }
    }
}