pQd
Joined 9 years ago
Karma
39
Submissions
18
Member since
Oct 23
Recent submissions
Comment · 1 year ago
Comment · 1 year ago
manticore, earlier sphinx search, has been rock solid for us for the past 16 years. now serving searches across nearly 300M short documents. we're using it in the old mode - where full index is re-created every 24h.
it's great to see that the project is alive and adding embeddings-related functions needed for semantic search.
Comment · 1 year ago
HE is using plain stateless IPv6 in IPv4 tunnel - it's neither TCP nor UDP, it's not NAT'able.
it's relatively simple for them to implement [ the stateless part ] but due to that puts some requirements on the party establishing the tunnel.
Comment · 1 year ago
aspect worth noting: up to my knowledge HE's tunnel will work only if you're assigned public IPv4 by your ISP. if you're behind a carrier grade NAT - too bad, you'll need to use another solution to get IPv6 to your home.
Comment · 2 years ago
environment: KVM VMs running on physical hardware managed by us.
we have a belt & suspenders approach:
* backups of selected files / database dumps [ via dedicated tools like mysqldump or pg_dumpall ] from within VMs
* backups of whole VMs
backups of whole VMs are done by creating snapshot files via virsh snapshot-create-as, rsync followed by virsh blockcommit. this provides crash-consistant images of the whole virtual disks. we zero-fill virtual disks before each backup.
all types of backups later go to borg backup[1] [ kopia[2] would be fine as well ] deduplicated, compressed repository.
this approach is wasteful in terms of bandwidth, backup size but gives us peace of mind - even if we miss to take file-level backup of some folder - we'll have it in the VM-level backups.
[1]: https://www.borgbackup.org/
[2]: https://kopia.io/
Comment · 2 years ago
and elinks as well. it had better handling of page layout.
Comment · 2 years ago
Allegro has amazing metadata allowing you to precisely filter out the results. Search experience on amazon is an utter abomination compared to Allegro.
Comment · 2 years ago
We're using BTRFS to host PostgreSQL and MySQL replication slaves. We're snapshoting drives holding data for both every 15 minutes, 1h, 8h and 12h and keep few snapshots for each frequency.
Those replicas are not used for any workload, besides nightly consistency checks for MySQLs via pt-table-checksum to ensure we don't have data drift.
Snapshots are crash consistent. Once in a while they give us ability to very quickly inspect how data looked like few minutes or hours ago. This can be life-saver in case of fat-fingering a production data and saved us from lenghty grepping of backups when we needed to recover few records from a specific table.
Yes, I know soft deletes, audit logs - all of those could help and we do have them, but sometimes that's not enough or not feasible.
Due to it's COW nature BTRFS is far from perfect for data that changes all the time [ databases busy with writes, images of VMs with plenty of disk write activity ]. There's plenty of write amplification, but that can be solved with NVMe drives thrown on the problem.
Comment · 3 years ago
Also - thanks to that FIDO2 does not seem to be usable with Microsoft's MS365 services [ Teams, Outlook, Excel etc ] on Android or iOS. there's no way to provide pin for the security key, regardless if it's plugged in via the USB port or used via NFC.
https://learn.microsoft.com/en-us/azure/active-directory/aut...
bummer
Comment · 3 years ago
coincidentally - we've been using first sphinx and then manticore for over 15 years as well. in our case it's fed each night with XML generated by Java code from data stored in MySQL databases. we index over 294M pseudo documents.
it's been rock solid for all those years.
Comment · 3 years ago
borg is great. we've been using it for the past 3 years to archive hundreds of file-level backups of servers, database dumps and VM images. average size of each borg repo is few GB but there are few outliers up to few hundreds of GB. most of backups are done daily, with 7-24 past days preserved in borg archive. borg repos are verified, copied to external disks, verified again and rotated offline each week.
borg replaced https://rdiff-backup.net/ for us and gave: * nice speedup of backups/backup tests, * decent saving in the disk space thanks to compression and deduplication, * decreased backup replication time [ borg repo tends to have much less, larger files compared to rdiff which has in its repo at least as many files as your source data; rsync likes it ].
to finish backups in a reasonable time we had to parallelize backup gathering [ each server / vm goes to separate borg repo; this limits a failure domain in case of corrupted repo, but denies us benefit of deduplication on larger scale - across servers ] and borg archiving. without that - we would be a limited by a single cpu core performance [ borg is not multithreaded yet ].
it's worth testing the backups - we're doing it each day by using borg's repo self test and by extracting few key files and checking their checksums and content... just in case.
echoing other comments - https://kopia.io/ looks interesting but we have not tried it yet.
Comment · 4 years ago
if you're running your test over the internet [ fluctuating latency, some packet losses ] - try enabling BBR [1] tcp congestion control algorithm on the sender side to utilize the available bandwidth more efficiently.
[1] https://en.wikipedia.org/wiki/TCP_congestion_control#TCP_BBR
Comment · 4 years ago
I use it at work. Sphinx search has been powering our text search since the inception in 2008 and was scaling greatly. In 2021 we've moved from to Manticore without any hiccups.
It's now handling index over 280M+ documents and has been drama-free - just works.
Comment · 6 years ago
i've been using BTRFS since 2014 to store backups. there is a noticeable performance penalty when rsync'ing hundreds of thousands of files to a spinning-rust disk connected to USB-SATA dock when BTRFS is used instead of EXT4. i'm accepting it in exchange for ability to run scheduled scrub of the data to detect potential bitrot.
since 2017 i'm also using BTRFS to host mysql replication slaves. every 15 min, 1h, 12h crash-consistent snapshots of the running database files are taken and kept for couple of days. there's consensus that - due to its COW nature - BTRFS is not well suited for hosting vms, databases or any other type of files that change frequently. performance is significantly worse compared to EXT4 - this can lead to slave lag. but slave-lag can be mitigated by: using NVMe drives and relaxing durability of MySQL innodb engine. i've used those snapshots few times each year - it worked fine so far. snapshots should never be the main backup strategy, independently of them there's a full database backup done daily from masters using mysqldump. snapshots are useful whenever you need to very quickly access state of the production data from few minutes or hours ago - for instance after fat fingering some live data.
during those years i've seen kernel crashes most likely due to BTRFS but i did not lose data as long as the underlying drives were healthy.
Comment · 7 years ago
https://manticoresearch.com/ is the lively, open source, fork of Sphinxsearch. that's where some of the earlier developers from the project moved to. it's used as a text-search backend on craigslist.
as far as i understand apparent death of sphinx and demand for continued development/support from big users of it led to creation of manticore.