> Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions.
Absolutely, there's a lack of consistency even within the same jurisdiction and document type. It only gets worse once you want to add multiple jurisdictions and different types of documents. My best strategy so far has been to use recursive chunking where you begin chunking at the largest section of newlines. Ideally though you want some form of semantic chunking where you already know what parts of the document represent Parts, Divisions, Schedules, Sections, Sub-sections, etc...
> I've reach out via your LinkedIn profile - would be great if there was an opportunity to collaborate.
Absolutely, there's a lack of consistency even within the same jurisdiction and document type. It only gets worse once you want to add multiple jurisdictions and different types of documents. My best strategy so far has been to use recursive chunking where you begin chunking at the largest section of newlines. Ideally though you want some form of semantic chunking where you already know what parts of the document represent Parts, Divisions, Schedules, Sections, Sub-sections, etc...
> I've reach out via your LinkedIn profile - would be great if there was an opportunity to collaborate.
Great! Always happy to connect.