FINLAM | Newspaper

Command Line

The newspaper package exposes a single command-line interface to run the full processing pipeline.

Export

The newspaper export command applies a multi-step processing chain on a IIIF manifest pointing to a press document and outputs a JSON representation of said newspaper enhanced with segmentation, reading order, OCR and grouped articles.

The steps are as follows:

parse the provided IIIF manifest and download the images it refers to in a temporary directory
segment the pages that were downloaded in Step 1 using the provided YOLO model
transcribe the boxes segmented in Step 2 using the provided PeroOCR model
predict the reading order of the boxes segmented in Step 2 using the provided LayoutReader model
group ordered boxes from Step 4 in newspaper articles
export predicted data for the whole manifest in a JSON file

The structure of the generated JSON export directly matches the serialization of a Manifest model (including its nested Page, Article, and YOLOBox instances).

It has the following parameters:

Parameter Description Type Default

Parameter	Description	Type	Default
`manifest_url`	URL to the IIIF manifest to process with the full pipeline.	`str`
`--image-height`	Restrict the height of images during their download.	`int`	`2500`
`--segmentation-model`	Path to the detection model to use for page segmentation.	`pathlib.Path`
`--segmentation-batch-size`	Batch size to use for page segmentation.	`int`	`4`
`--ocr-model`	Path to the PeroOCR model to transcribe the segmented boxes.	`pathlib.Path`
`--ocr-overlap-threshold`	If a line detected by PeroOCR overlaps a segmented YOLO box more than that threshold, it will be assigned to said box to forge its transcription. The value should be between 0 and 1.	`float`	`0.65`
`--reading-order-model`	Path to the LayoutReader model to predict the reading order of segmented boxes.	`pathlib.Path`

manifest_url

URL to the IIIF manifest to process with the full pipeline.

str

--image-height

Restrict the height of images during their download.

int

2500

--segmentation-model

Path to the detection model to use for page segmentation.

pathlib.Path

--segmentation-batch-size

Batch size to use for page segmentation.

int

4

--ocr-model

Path to the PeroOCR model to transcribe the segmented boxes.

pathlib.Path

--ocr-overlap-threshold

If a line detected by PeroOCR overlaps a segmented YOLO box more than that threshold, it will be assigned to said box to forge its transcription. The value should be between 0 and 1.

float

0.65

--reading-order-model

Path to the LayoutReader model to predict the reading order of segmented boxes.

pathlib.Path

Example

The following command was used on this manifest:

newspaper export https://gallica.bnf.fr/iiif/ark:/12148/bpt6k41008602/manifest.json \
    --segmentation-model models/segmentation.pt \
    --ocr-model models/ocr \
    --reading-order-model models/reading_order

Show logs

2026-09-03 13:47:00,122 INFO/datasets: PyTorch version 2.13.0 available.
2026-09-03 13:47:00,123 INFO/datasets: Polars version 1.43.2 available.
2026-09-03 13:47:05,548 INFO/root: numba available, importing jit
2026-09-03 13:47:05,826 INFO/newspaper.export: 📍 STEP 1/5: Downloading the manifest and its associated images 📍
2026-09-03 13:47:05,826 INFO/newspaper.download: Base folder to process the manifest https://gallica.bnf.fr/iiif/ark:/12148/bpt6k41008602/manifest.json is available @ /tmp/4d5ab18d8f53e386153b06295ab45a0d
2026-09-03 13:47:05,826 INFO/newspaper.download: Downloading the manifest in a JSON file...
2026-09-03 13:47:06,187 INFO/newspaper.download: Manifest was successfully retrieved and parsed.
2026-09-03 13:47:06,187 INFO/newspaper.download: Images will be downloaded and saved @ /tmp/4d5ab18d8f53e386153b06295ab45a0d/images
Downloading images...: 100%|█████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 1478.34it/s]
2026-09-03 13:47:06,192 INFO/newspaper.export: 📍 STEP 2/5: Segmenting pages from the manifest 📍
2026-09-03 13:47:06,192 INFO/newspaper.segment: Loading the segmentation model @ models/segmentation.pt...
2026-09-03 13:47:06,296 INFO/newspaper.segment: Segmentation model was successfully loaded.
Segmenting pages...:   0%|                                                                     |
0: 1024x1024 1 HEADER-TITLE, 11 HEADER-TEXTs, 3 SECTION-TITLEs, 16 ARTICLE-TITLEs, 4 ARTICLE-SUBTITLEs, 4 ARTICLE-ILLUSTRATIONs, 113 ARTICLE-TEXTs, 2 ARTICLE-INSIDEHEADINGs, 3 ARTICLE-AUTHORs, 1739.4ms
1: 1024x1024 3 SECTION-TITLEs, 1 SECTION-ILLUSTRATEDTEXT, 16 ARTICLE-TITLEs, 1 ARTICLE-SUBTITLE, 4 ARTICLE-ILLUSTRATIONs, 152 ARTICLE-TEXTs, 1 ARTICLE-INSIDEHEADING, 2 ARTICLE-AUTHORs, 3 ARTICLE-ILLUSTRATEDTEXTs, 1739.4ms
2: 1024x1024 1 SECTION-TITLE, 24 ARTICLE-TITLEs, 4 ARTICLE-SUBTITLEs, 124 ARTICLE-TEXTs, 10 ARTICLE-INSIDEHEADINGs, 1739.4ms
3: 1024x1024 2 SECTION-TITLEs, 13 ARTICLE-TITLEs, 6 ARTICLE-SUBTITLEs, 115 ARTICLE-TEXTs, 10 ARTICLE-INSIDEHEADINGs, 1 ARTICLE-AUTHOR, 1739.4ms
4: 1024x1024 2 SECTION-TITLEs, 1 SECTION-SUBTITLE, 6 ARTICLE-TITLEs, 1 ARTICLE-SUBTITLE, 49 ARTICLE-TEXTs, 1 ARTICLE-AUTHOR, 1 ARTICLE-TABLE, 1739.4ms
5: 1024x1024 (no detections), 1739.4ms
Segmenting pages...:  17%|██████████▏                                                  | 1/6 [00:11<00:56, 11.20s/it]Speed: 9.3ms preprocess, 1739.4ms inference, 1.3ms postprocess per image at shape (4, 3, 1024, 1024)
Segmenting pages...: 100%|█████████████████████████████████████████████████████████████| 6/6 [00:11<00:00,  1.87s/it]
2026-09-03 13:47:17,529 INFO/newspaper.export: 📍 STEP 3/5: Transcribing the segmented boxes 📍
2026-09-03 13:47:17,529 INFO/newspaper.transcribe: Loading the OCR model @ models/ocr...
2026-09-03 13:47:17,529 INFO/newspaper.transcribe: Loading CPU configuration for PeroOCR
LayoutEngine params are line_end_weight:1.0 vertical_line_connection_range:3 smooth_line_predictions:False line_detection_threshold:0.2 adaptive_downsample:True
2026-09-03 13:47:18,133 INFO/newspaper.transcribe: OCR model was successfully loaded.
Transcribing boxes...:   0%|                                                                   | 0/6 [00:00<?, ?it/s]NET INPUT 196608 Mpx.
NET INPUT 720896 Mpx.
GET MAPS TIME: 1.8798284530639648
MAP RES: (971, 673, 5)
656/877 lines detected by PeroOCR were assigned to a segmented YOLO box.
Box 53 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 58 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 85 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 107 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 109 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 111 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 120 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 129 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 141 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 142 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 144 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 146 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 149 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 150 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 152 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 153 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 155 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 156 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 157 does not have any assigned line from PeroOCR, its transcription will be empty.
Transcribing boxes...:  17%|█████████▊                                                 | 1/6 [01:12<06:02, 72.40s/it]NET INPUT 720896 Mpx.
NET INPUT 4587520 Mpx.
GET MAPS TIME: 8.535595893859863
MAP RES: (2500, 1765, 5)
842/1211 lines detected by PeroOCR were assigned to a segmented YOLO box.
Box 102 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 108 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 113 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 117 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 124 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 139 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 144 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 148 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 154 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 155 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 158 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 161 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 162 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 163 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 164 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 166 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 168 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 169 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 171 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 172 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 173 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 174 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 176 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 177 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 178 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 179 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 180 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 181 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 182 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 183 does not have any assigned line from PeroOCR, its transcription will be empty.
Transcribing boxes...:  33%|███████████████████▋                                       | 2/6 [03:00<06:12, 93.11s/it]NET INPUT 4587520 Mpx.
GET MAPS TIME: 6.995337247848511
MAP RES: (2500, 1732, 5)
834/1031 lines detected by PeroOCR were assigned to a segmented YOLO box.
Box 120 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 134 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 156 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 160 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 161 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 162 does not have any assigned line from PeroOCR, its transcription will be empty.
Transcribing boxes...:  50%|█████████████████████████████▌                             | 3/6 [04:41<04:50, 96.84s/it]NET INPUT 4587520 Mpx.
GET MAPS TIME: 8.203315258026123
MAP RES: (2500, 1758, 5)
538/902 lines detected by PeroOCR were assigned to a segmented YOLO box.
Box 57 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 58 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 72 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 83 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 99 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 101 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 114 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 122 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 127 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 128 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 131 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 135 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 138 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 139 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 141 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 142 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 144 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 145 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 146 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 147 does not have any assigned line from PeroOCR, its transcription will be empty.
Transcribing boxes...:  67%|███████████████████████████████████████▎                   | 4/6 [06:14<03:10, 95.42s/it]NET INPUT 4587520 Mpx.
GET MAPS TIME: 7.507846832275391
MAP RES: (2500, 1742, 5)
388/1022 lines detected by PeroOCR were assigned to a segmented YOLO box.
Box 30 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 38 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 47 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 57 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 59 does not have any assigned line from PeroOCR, its transcription will be empty.
Box 60 does not have any assigned line from PeroOCR, its transcription will be empty.
Transcribing boxes...:  83%|█████████████████████████████████████████████████▏         | 5/6 [07:46<01:34, 94.27s/it]NET INPUT 4587520 Mpx.
GET MAPS TIME: 8.053626537322998
MAP RES: (2500, 1758, 5)
0/535 lines detected by PeroOCR were assigned to a segmented YOLO box.
Transcribing boxes...: 100%|███████████████████████████████████████████████████████████| 6/6 [08:32<00:00, 85.48s/it]
2026-09-03 13:55:51,127 INFO/newspaper.export: 📍 STEP 4/5: Predicting the reading order of the segmented boxes 📍
2026-09-03 13:55:51,127 INFO/newspaper.predict_order: Loading the reading order model @ models/reading_order...
2026-09-03 13:55:51,648 INFO/newspaper.predict_order: Reading order model was successfully loaded.
Predicting boxes reading order...: 100%|███████████████████████████████████████████████| 6/6 [00:03<00:00,  1.58it/s]
2026-09-03 13:55:55,535 INFO/newspaper.export: 📍 STEP 5/5: Grouping ordered boxes in newspaper articles 📍
2026-09-03 13:55:55,536 WARNING/newspaper.group: Found no article for the box 151 from page 1
2026-09-03 13:55:55,537 INFO/newspaper.export: 📍 STEP 6/6: Exporting predicted data as JSON 📍
2026-09-03 13:55:55,575 INFO/newspaper.export: The JSON export of the whole manifest is available @ /tmp/4d5ab18d8f53e386153b06295ab45a0d/export.json

The models were downloaded from Arkindex:

The JSON export for the whole manifest is available here.