ahaspel
Joined 6 months ago
Karma
221
Submissions
31
Member since
Apr 21
Joined 6 months ago
I didn’t do OCR myself, except for the topic index and to fill in a few gaps. I started from existing Wikisource text and then built a pipeline around that: cleaning (headers, hyphenation, etc.), detecting article boundaries, reconstructing sections, and linking things back to the original page images. Most of the effort went into rendering the complex layouts, and handling the cross-linking, not the initial ingestion.
Glad to go into more detail if you’re interested, but that’s the gist of it.
People asking for dataset access has definitely been one of the themes of this thread. I’m taking that seriously. If I do expose it, I’d want to do it in a form that preserves the structure and doesn't just dump plain text.
Thanks for the suggestion.