ozataman
Joined 17 years ago
Joined 17 years ago
For example, you may need to provision a 20-node cluster only because you need the 10+ terabytes in storage across several datasets you need to keep "hot" for sporadic use throughout the day/week, but don't nearly need all that computational capacity around the clock. Unlike BigQuery, Redshift doesn't separate storage from querying. Redshift also doesn't offer a practically acceptable way to scale up/down; resizes at that scale take up to a day, deleting/restoring datasets would cause lots of administrative overhead and even capacity tuning between multiple users is a frequent concern.
Making matters worse, it is common for a small number of tables to be the large "source of truth" tables that you need to keep around to re-populate various intermediate tables even if they themselves don't get queries that often. In Redshift, you will provision a large cluster just to be able to keep them around even though 99% of your queries will hit one of the smaller tables.
That said, I haven't tried the relatively new "query data on S3" Redshift functionality. It doesn't seem quite the equivalent of what BigQuery does, but may perhaps alleviate this issue.
Sidenote: I have been a huge Redshift fan pretty much since its release under AWS. I do however think that it is starting to lose its edge and show its age among the recent advances in the space; I have been increasingly impressed with the ease of use (including intra team and even inter-team collaboration) in the BigQuery camp.
There is something potentially harmful, or perhaps that needs addressing, about end-user tools growing in expressive power. A good friend who does statistical genetics work once told me "but I don't want every user running their own regressions and drawing nonsensical conclusions from badly prepared data!"
Don't forget that, from the manufacturer's perspective, Amazon is not only the logistical facilitator, but also a sales channel where huge numbers of potential customers are introduced to companies they wouldn't naturally be exposed to.
They play thousands and thousands of hours and, yes, also spend quite a bit of time reading, thinking, analyzing (both their own and others') games and learning from mentors/teachers. However, practice is king and all the reading/analysis would be worthless in its absence. They would have no anchors to grab onto in your brain - no way to really become operational.
A chess "player" that mostly reads, studies and analyzes with a little bit of practice sprinkled in between would indeed be hilariously weak.
Soostone is a technology provider in the enterprise analytics space with a focus on performance improvement for e-commerce clients. We are the developers of a highly scalable platform that revolves around data, modeling, machine learning, real-time decision-making and sustained optimization. We routinely build features or capabilities that make use of in-process and multi-node concurrency, parallelism, stream processing, domain-specific languages (DSLs/EDSLs), Functional Reactive Programming (FRP) on GHCJS, MapReduce on Hadoop (via Haskell) and many other methodologies common in the functional programming world.
We also care a great deal about contributing back to the community by open sourcing useful components wherever possible. Our team members help organize a number of community events in eastern United States including the monthly NY Haskell Users Group, Hac NYC and the Compose conference.
If interested, please submit applications to [email protected]
Website: http://www.soostone.com
It is common to hear in the Haskell community remarks like "Haskell is the best imperative language I've used."
Cassandra et. al. are completely different, in that you don't use them because they are more fun to use. You use them despite their awkward, low-level interfaces because you're going to dump billions of data cells into your database from day one with no end in sight and want all the easy scaling/availability features provided.