19-84
Joined 10 months ago
Karma
118
Submissions
13
Member since
Jan 7
"Who controls the past controls the future: who controls the present controls the past."
View on Hacker NewsRecent submissions
Comment · 9 months ago
there is nearly 10TB of youtube metadata available on archive.org https://archive.org/details/youtube-metadata
Comment · 9 months ago
redd-archiver uses postgres full text search. for static search you could use lunr.js
Comment · 9 months ago
the torrent has data for the top 40,000 subs on reddit. thanks to watchful1 splitting the data by subreddit, you can download only the subreddit you want from the torrent
Comment · 9 months ago
the data from 2025-12 has been released already, it is usually released every month, it just needs to be split and reprocessed for 2025 by watchful1. i will probably eventually add support for importing data from the monthly arctic shift dumps so that archives can be updated monthly.
https://github.com/ArthurHeitmann/arctic_shift/releases
Arctic Shift https://academictorrents.com/browse.php?search=RaiderBDev
Watchful1 https://academictorrents.com/browse.php?search=Watchful1
Comment · 9 months ago
I included a metadata dump of every subreddit found in the torrent. it includes a status field which will show of a subreddit is private along with a much more details
data catalog readme: https://github.com/19-84/redd-archiver/blob/main/tools/READM...
reddit data: https://github.com/19-84/redd-archiver/blob/main/tools/subre...
Comment · 9 months ago
ive created tooling for an instance registry and team based leaderboard. the API has function to support this as well, so that we can collectively host archives in a decentralized and distributed manner
registry readme: https://github.com/19-84/redd-archiver/blob/main/docs/REGIST...
register instances: https://github.com/19-84/redd-archiver/blob/main/.github/ISS...
Comment · 9 months ago
I've updated the workflow and added a placeholder page that will serve before archives are created. thanks again! https://github.com/19-84/redd-archiver/commit/0dfd505ca81cb2...
Comment · 9 months ago
thank you for your comment, some example dot files were not copied in my original repo, they have now been added.
https://github.com/19-84/redd-archiver/commit/0bb103952195ae...
the docs have been updated with mkdir steps
https://github.com/19-84/redd-archiver/commit/c3754ea3a0238f...
Comment · 9 months ago
thank you for your comment, I will support any platform that has complete dataset available. I will take submissions for any complete datasets through github issues. https://github.com/19-84/redd-archiver/blob/main/.github/ISS...
Comment · 9 months ago
the API and MCP server is very powerful ;)
Comment · 9 months ago
I have also published sub statistics and profiling for each platform. these can be used to help identify which subs to prioritize for archiving.
reddit: https://github.com/19-84/redd-archiver/blob/main/tools/subre...
voat: https://github.com/19-84/redd-archiver/blob/main/tools/subve...
ruqqus: https://github.com/19-84/redd-archiver/blob/main/tools/guild...
Comment · 9 months ago
the API should allow for a lot of different integrations