Dataset analysis

The teklia-qwen dataset analyze command requires the following arguments:

the path to the folder containing the dataset files,
the name of the base model that will be used for training, provided through the --base-model argument.

This command describes the content of the dataset and can help define some inference parameters like the maximum model token context (--max-model-len). The higher this value is (or when it’s unset), the higher the memory usage will be at inference time. Setting this up accurately reduces RAM usage and helps with inference speed and avoiding hallucinations. This command suggests a value, in the logs, based on the dataset. It looks for the maximum context size in the dataset and rounds the value to add a margin.

Analysis

The following steps are conducted during dataset analysis:

per set image dimensions statistics
per set token length statistics for
- system prompt
- query prompt
- provided images
- expected response
- total count (used to compute the recommended maximum context size)
total number of images that have an empty expected response (if there are any)
prompt statistics (how many there are and their content) for
- system prompt
- query prompt

An example of the generated statistics file is available at tests/fixtures/dataset/expected_stats.md.

Examples

Below is a command to analyze a dataset that will be used to fine-tune Qwen/Qwen3.5-9B:

teklia-qwen dataset analyze tests/fixtures/dataset \
    --base-model Qwen/Qwen3.5-9B

2026-09-25 11:29:58,841 INFO/qwen.dataset.analyze: Loading tokenizer from Qwen/Qwen3.5-9B
2026-09-25 11:30:03,760 INFO/qwen.dataset.analyze: Tokenizer was loaded
2026-09-25 11:30:03,761 INFO/qwen.dataset.analyze: Train
2026-09-25 11:30:03,774 INFO/qwen.dataset.analyze: Train images statistics

| Metric | Width | Height |
|:-------|:-----:|:------:|
| Min    |   -1  |   -1   |
| Max    |   -1  |   -1   |
| Mean   |   -1  |   -1   |
| Median |  -1.0 |  -1.0  |
| Count  |   2   |   2    |

2026-09-25 11:30:03,774 INFO/qwen.dataset.analyze: Train tokens statistics
2026-09-25 11:30:03,775 INFO/qwen.dataset.analyze: Train system tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   11  |
| Max    |   11  |
| Mean   |   11  |
| Median |  11.0 |

2026-09-25 11:30:03,775 INFO/qwen.dataset.analyze: Train query tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   15  |
| Max    |   15  |
| Mean   |   15  |
| Median |  15.0 |

2026-09-25 11:30:03,775 INFO/qwen.dataset.analyze: Train response tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   0   |
| Max    |   29  |
| Mean   |  14.5 |
| Median |  14.5 |

2026-09-25 11:30:03,776 INFO/qwen.dataset.analyze: Train images tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   0   |
| Max    |   0   |
| Mean   |   0   |
| Median |  0.0  |

2026-09-25 11:30:03,776 INFO/qwen.dataset.analyze: Train total tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   26  |
| Max    |   55  |
| Mean   |  40.5 |
| Median |  40.5 |

2026-09-25 11:30:03,776 INFO/qwen.dataset.analyze: Val
2026-09-25 11:30:03,777 INFO/qwen.dataset.analyze: Val images statistics

| Metric | Width | Height |
|:-------|:-----:|:------:|
| Min    |   -1  |   -1   |
| Max    |   -1  |   -1   |
| Mean   |   -1  |   -1   |
| Median |   -1  |   -1   |
| Count  |   1   |   1    |

2026-09-25 11:30:03,777 INFO/qwen.dataset.analyze: Val tokens statistics
2026-09-25 11:30:03,777 INFO/qwen.dataset.analyze: Val system tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   11  |
| Max    |   11  |
| Mean   |   11  |
| Median |   11  |

2026-09-25 11:30:03,778 INFO/qwen.dataset.analyze: Val query tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   23  |
| Max    |   23  |
| Mean   |   23  |
| Median |   23  |

2026-09-25 11:30:03,778 INFO/qwen.dataset.analyze: Val response tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   0   |
| Max    |   0   |
| Mean   |   0   |
| Median |   0   |

2026-09-25 11:30:03,778 INFO/qwen.dataset.analyze: Val images tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   0   |
| Max    |   0   |
| Mean   |   0   |
| Median |   0   |

2026-09-25 11:30:03,778 INFO/qwen.dataset.analyze: Val total tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   34  |
| Max    |   34  |
| Mean   |   34  |
| Median |   34  |

2026-09-25 11:30:03,778 INFO/qwen.dataset.analyze: Test
2026-09-25 11:30:03,779 INFO/qwen.dataset.analyze: Test images statistics

| Metric | Width | Height |
|:-------|:-----:|:------:|
| Min    |   -1  |   -1   |
| Max    |   -1  |   -1   |
| Mean   |   -1  |   -1   |
| Median |  -1.0 |  -1.0  |
| Count  |   2   |   2    |

2026-09-25 11:30:03,780 INFO/qwen.dataset.analyze: Test tokens statistics
2026-09-25 11:30:03,780 INFO/qwen.dataset.analyze: Test system tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   11  |
| Max    |   11  |
| Mean   |   11  |
| Median |  11.0 |

2026-09-25 11:30:03,780 INFO/qwen.dataset.analyze: Test query tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   15  |
| Max    |   15  |
| Mean   |   15  |
| Median |  15.0 |

2026-09-25 11:30:03,780 INFO/qwen.dataset.analyze: Test response tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   0   |
| Max    |   0   |
| Mean   |   0   |
| Median |  0.0  |

2026-09-25 11:30:03,781 INFO/qwen.dataset.analyze: Test images tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   0   |
| Max    |   0   |
| Mean   |   0   |
| Median |  0.0  |

2026-09-25 11:30:03,781 INFO/qwen.dataset.analyze: Test total tokens

| Metric | Count |
|:-------|:-----:|
| Min    |   26  |
| Max    |   26  |
| Mean   |   26  |
| Median |  26.0 |

2026-09-25 11:30:03,781 INFO/qwen.dataset.analyze: Images with no response

| Metric | Images |
|:-------|:------:|
| Count  |   3    |

2026-09-25 11:30:03,781 INFO/qwen.dataset.analyze: Prompts statistics
2026-09-25 11:30:03,781 INFO/qwen.dataset.analyze: System

| Metric | Prompts |
|:-------|:-------:|
| Count  |    1    |

2026-09-25 11:30:03,781 INFO/qwen.dataset.analyze: Query

| Metric | Prompts |
|:-------|:-------:|
| Count  |    2    |

2026-09-18 10:15:46,681 INFO/qwen.dataset.analyze: Statistics of the dataset saved at tests/fixtures/dataset/stats.md
2026-09-18 10:15:46,681 INFO/qwen.dataset.analyze: Based on the context in the dataset, a max_model_len set to 1000 is recommended to limit model output.