It turns out this problem even shows up for relatively small datasets, things that would easily fit into memory on any consumer-level laptop from a couple years ago. I did some contract work for the NYPD a few years ago and the primary data sources I had to deal with was a block of free text describing a 911 call. All they really wanted was some frequency counts on specific terms of interest, sounds easy enough.
Oh, but their data warehouse thing that nobody knew what it was called, inserted a random space every so often and removed random spaces every so often making the text completely unreliable. Compiled with all the normal misspellings, fat fingered acronyms, slang and other nonstandard ways of writing things it turned out to be a huge job.
"Bicycle stolen at 0930 from residence on the 300 block of E 10th St, East Village"
would get turned into "Bicycle stol en a t093 0 from r es iden ceon the 300 block of E 10 thst, Ea st Vil lage."
Oh, but their data warehouse thing that nobody knew what it was called, inserted a random space every so often and removed random spaces every so often making the text completely unreliable. Compiled with all the normal misspellings, fat fingered acronyms, slang and other nonstandard ways of writing things it turned out to be a huge job.
"Bicycle stolen at 0930 from residence on the 300 block of E 10th St, East Village"
would get turned into "Bicycle stol en a t093 0 from r es iden ceon the 300 block of E 10 thst, Ea st Vil lage."
Good luck with that.