Plot results

The command atr plot-results saves multiple plots to analyze training convergence and evaluation results.

Parameters

This command can either plot training curves or evaluation results, depending of its input.

Parameter Description Type Default

--metrics

Path to the metrics CSV file created by PyLaia.

Path

--json-report

Path to the JSON report created by the atr pylaia-analyze command.

Path

--output

Path to the output directory where plots will be saved.

Path

The user must provide either:

  • the metrics file. In this case, the script will plot training and validation curves.

  • the json_report file. In this case, the script will plot CER/WER distribution, relationship between error rates and confidences, and the automation plot.

  • if both files are provided, the script will plot everything.

Format

The --json-report argument must follow this specific format:

{
    "<id>": {
        "score": 0.0, // float between 0 and 1
        "cer": {
            "error_rate": 0.0 // float between 0 and 1
        },
        "wer": {
            "error_rate": 0.0 // float between 0 and 1
        }
    },
    ...
}

Examples

Plot metrics

To plot training and validation curves depending on the epochs, run the following command:

atr plot-results --metrics tests/examples/metrics.csv --output plots/

Expected output:

INFO:utils.py:Plotted losses in plots/plot_training_losses.png.
INFO:utils.py:Plotted error rates in plots/plot_training_cer_wer.png.

These plots can be used to visualise and analyse the model’s convergence. plot training losses plot training cer wer

Plot evaluation results

To plot evaluation results, run the following command:

atr plot-results --json-report tests/examples/report.json --output plots/

Expected output:

INFO:utils.py:Plotted automation rate in plots/plot_automation.png.
INFO:utils.py:Plotted confidence and error rates in plots/plot_confidence.png.
INFO:utils.py:Plotted CER/WER distribution in plots/plot_error_rates.png.

Note that the JSON report can be generated using this command (see the full documentation):

atr pylaia-analyze --format json --labels tests/examples/truth_test.txt --predictions tests/examples/pred_test_conf.txt --images path/to/images --confidence-scores > tests/examples/report.json

These plots can be used to visualize:

  • the error rate distribution plot error rates

  • the correlation between the Character Error Rates and confidence scores plot confidence

  • the automation capabilities of the model plot automation