phildini
Joined 14 years ago
Karma
171
Submissions
53
Member since
Apr 4
Recent submissions
Comment · 10 months ago
Fixed Austin TX!
Comment · 10 months ago
Hello! The crawlers are not currently public, but I'm happy to take volunteers behind the curtain.
I also fixed Cook County
Comment · 10 months ago
Hi there! I've built the beginnings of this platform at https://civic.band and https://civic.observer.
We track City Councils, Boards of Supervisors, really any municipality we can get our hands on. I'm very open to how to make this better!
Comment · 10 months ago
I don't work with them right now but maybe this will help? https://oakland.ca.civic.band/-/search?q=flock
Comment · 10 months ago
Hi! I've written crawlers for about a dozen municipal hosting platforms, and you can learn the bare-bones of it from our "How" page: https://civic.band/how.html
I also gave a talk on this concept that walks through the whole process: https://www.youtube.com/watch?v=MtWzNnZvQ6w
The short answer is: there's no common API for any of these sites, and even the ones that do have an API are sometimes misconfigured. It's why I wrote all the scrapers by hand.
Comment · 10 months ago
thanks! the next major hurdle is school boards; gotta get EBoard and BoardDocs to make that work
Comment · 10 months ago
Where would you want more detail? I've been working on adding Canada to CivicBand: https://civic.band/sites/sites?_sort_desc=pages&state__in=BC...
I also track Puerto Rico, but only at the Senate level: https://senado.pr.civic.band/
Comment · 10 months ago
Granicus is six providers in a trench coat it turns out. IQM2, NovusAgenda, Legistar, Granicus, PriveGov, and CivicClerk are all Granicus projects that share absolutely 0 apis that I've found, and a city having one of these operational is no guarantee they have any of the others.
Legistar and CivicClerk have actual APIs, which is nice, although it's extremely easy for the City Clerk's staff to trip and make the Legistar API unusable.
My experiments with using LLMs to write crawlers for these has been extremely mixed; it's good at getting first page of data and less good at following weird pagination trails or follow-on requests.
All of this led me to build CivicBand (which tracks all the municipalities I can get my hands on) and CivicObserver (which is generalized full-text search alerting for municipalities via email, mastodon, bluesky, and slack webhook)
Comment · 10 months ago
I _think_ (but am not actually certain) we're monitoring more municipal agencies at CivicBand, but I know some of the folks at MuckRock and the work they're doing is absolutely critical.
Comment · 10 months ago
The question would ultimately get settled in court, I think, but a DA who was feeling cop-aligned and vicious could try to ding you for interfering with police operations by _not_ allowing your plate to get scanned.
Comment · 10 months ago
Search context is legitimately hard, especially since this is unstructured text data that (ime building CivicBand) needs to be OCR'd not parsed for best results.
You might be terrified the number of municipalities that are still posting PDFs of scans of printouts of their minutes, which were originally a word document, and round and round we go.
Part of why I haven't guaranteed results building CivicObserver is because of how hard search context is. Maybe making this an MCP helps, but I'm not actually sure it does.
Comment · 10 months ago
Thanks for asking!
- The framework for crawling is open-source. https://github.com/civicband
- There is absolutely not a standardized API for nearly any of this. I build generalized crawlers when I can, and then build custom crawlers when I need.
- Can you let me know which city? The crawlers run for every municipality at least once every day, so that's probably a bug
Comment · 10 months ago
Fixed the 404s on civic.band, thanks
Comment · 10 months ago
This is super important work, and is kind of why I built https://civic.band and https://civic.observer, which are generalized tools for monitoring civic govts. (You can search for anything, not just ALPR)