rupestrecampos

Joined 2 years ago

Karma

Submissions

Member since

Sep 26

Recent submissions

  Location: Brazil
  Remote: Yes
  Willing to relocate: Yes
  Technologies: Python, Postgres, open source Geospatial (Postgis, maplibre, GDAL 'n friends), API, JS, HTML and more.
  Résumé/CV: https://car.rupestre-campos.org/contact
  Email: rupestre.campos [at] gmail.com

Comment · 1 year ago

Thanks for your point of view, I appreciate very much. About data freshness, we would like to serve on our system the same information that you can find on the source website, and we have a few tickets pointing the difference to solve, so maybe the continuous strategy each hour may minimize it as today we do it once a day.

Strategies to download data constantly changing via API

2 rupestrecampos 1 year ago 2 comments

Comment · 1 year ago

Brazil´s Farms, Rural Environmental Register data map.

+300Gb data updated monthly, for $7 besides what was already at home (internet, hardware).

ETL + Database + API in a raspberry pi 4. Frontend at Stremlit.io

https://car-viewer.streamlit.app

Comment · 1 year ago

Thanks for pointing it out, will try to use it when needed. I could put it on internet archive problem is it is updated in a daily basis so uploading to there we could more likely create a temporal analysis rather than use it for real. Other problem that some datasets are kind of big, between 1Gb and 300 Gb

Comment · 1 year ago

I understand but the problem is that we need bots or someone will have to do it manually, and believe me we deal with a lot of layers, daily.

There are other methods to limit bad behavior like having usage limits and issuing a 429 too many requests in case of not respecting it.

For other countries to not access data, they could issue a api key or create a geofence to block external IP´s.

There are developer friendly ways to do it all, but it seems they are moving to a pattern to have massive amount of real people downloading the data instead of analyzing it, what a waste!

Why login to get public data?

2 rupestrecampos 1 year ago 4 comments

Comment · 2 years ago

https://adguard-dns.io/en/public-dns.html go to Our server addresses > Plain DNS > default servers

Comment · 2 years ago

After piHole and AdGuard app in the router now my wide used solution that works it to override dns settings to adguard, works in maaany routers i have tested.

AdGuard DNS 94.140.14.14 94.140.15.15

Comment · 2 years ago

The overengineered I wrote this morning to get to know at lunch it would not be part of the project deliverables.

Comment · 2 years ago

After +5 years working for companies building their own geospatial platform, I felt like I needed a portfolio for showcasing my skills to get a new job, without spending more than a few bucks.

So I created https://car-viewer.streamlit.app/ and a few others that you may find in contacts link at top.

The idea was to show and keep updated a massive (+250Gb) dataset on map on my own end to end using open source and proprietary tools within free tier limits.

After releasing it I got interview invitations that led me to a new job.