goldfishcaura
Joined 12 years ago
Karma
23
Submissions
54
Member since
May 6
https://segah.me
View on Hacker NewsRecent submissions
Comment · 9 years ago
It amazes me how quickly our industry has forgotten the need for DBAs. With these MongoDBs, MPP cloud dbs and Hadoops, everyone seems to have assumed that engineers can now do all db work. This is reflected in the titles too: Data "Engineer".
But from my perspective, this is delusional. There is a lot that goes into DBA's experience that is not solved by the performance improvements in databases over the past decade. But there are more choices. 20-30 years ago, you would have been forced to write code on Oracle and you would have asked for help before deciding how to structure the data. Today, with more choices, you just read some online opinions, and jump on it without any internal resource to guide you.
Not saying the world of Oracle was great, but the young on this thread (me included) would benefit from respecting the experience of the old.
Comment · 9 years ago
Who would have though given that at some point this "Don't use MongoDB" seemed to have taken over HN:
https://news.ycombinator.com/item?id=3202081
Comment · 9 years ago
Filter by Product: Data Studio, and the result says it all:
https://www.google.com/analytics/success-stories/#?modal_act...
"Sorry, we couldn't find any success stories to match those filters."
Comment · 9 years ago
A BigQuery partner here and a long-time user of Redshift.
(https://cloud.google.com/partners/directory/#?q=caura
https://discourse.looker.com/t/bigquery-migrating-from-legac...
and you can Google my creds for Redshift with "Troubleshooting Redshift performance")
So having benchmarks tests is great as a general guideline for what works under different architectures/schema designs. Unfortunately, benchmarking is highly subjective to the initial choices. I am a big fan of BigQuery (enough to go through Google's vetting process), but there are plenty of performance issues that I've run into it that would have been easily resolved with Redshift. Here are some concrete examples:
1) Running a Query across several very small tables. It turns out that occasionally querying small tables causes heavy network traffic within Google's distributed system. The solution on Redshift would be to adjust distribution. On Google, however, you don't have any control over this. You just have to hope that Google's algorithms pick up the issue based on usage (they don't).
2) Joining large tables. Avoid joining large tables in BigQuery. In Redshift the join would have been done by making sure that the sortkey is set on the column that is used for a join on the (typically) right table. Then having a common distkey between the two tables (this way the relevant data on both tables lives on the same node. BigQuery just throws resources at the problem. Well, it turns out that throwing resources at the problem is super slow (think 5-15 Redshift seconds vs. 200 BQ seconds).
Re: Snowflake. Can't speak to it as I haven't had personal experience. I have worked with Data people who had opinions on both favorable and negative sides of the spectrum. This just suggests to me that just like Redshift and BigQuery, Snowflake is not a universal solution. You really need to understand: 1) what your goals are for the usage among varying consumers 2) what skill set do the various users of the database have