fendale
Joined 19 years ago
Joined 19 years ago
What do you mean by namespaces here? Are they created by having different partitions or LVM volumes? As you mentioned consumer grade SSDs only have a single namespace, I am guessing this is something that needs some config when mounting the drive?
Sqlite can go pretty far if you have fast SSD disk. The biggest problem is that your app is constrained to a single host. For many apps, with backups and a failover plan, that may be OK. For other its a non starter.
I’m sure many good engineers have tried management and found it to be not for them. Have an open discussion with your manager and see where it goes, as if you don’t you will likely end up applying for an engineering role elsewhere and leave anyway.
This is the wider issue with small files. On HDFS each file uses up some namenode memory, but if there are jobs that need to touch 100k+ files (which I have seen plenty of), that puts a real strain on the Namenode too.
I have no experience with S3 to know how it would behave in terms of metadata queries for lots of small objects.
The meta data is not the only problem with small files. Massive parallel jobs that need to read tiny files will always be slower than if the files were larger. The overhead of getting the metadata for the file, setting up a connection to do the read is quite large to read only a few 100kb or a few MB.
The other issue with the HDFS namenode, is that it has a single read/write lock to protect all the in memory data. Breaking that lock into a more fine grained set of locks would be a big win, but quite tricky at this stage.
The price difference was all tax - and lots of it!
I am a total novice, and have fairly little invested, but those are the sort of numbers that pension projects are based on.
If you can keep you personal burn rate low, and over time invest a significant amount of money at 5% return for 10 years, the interest really starts to snowball.