fat0wl

This is true but keep in mind that you have to set up a pipeline for collecting the data into HDFS (Storm? batch loading?) & you have to pay for the machines.

So while your analysis is valid, there are more "costs" at play like developer time, cluster maintenance, hardware. I like to play with Spark's ML libraries but am wary about designing projects specifically around them because of this overhead, especially when trying to distribute some API/tech that you'd like others to use.

Not trying to be a downer, I actually wish the choice to go distributed was more of a no-brainer, hah. Would love for some APIs to emerge that could be used locally/distributed transparently without actually having to run a dummy cluster & data migration to run locally.

yeah i do Scala now but same deal. Where I think the OP comment really goes wrong is when he says something like "you're gonna need all that stuff and you're gonna end up re-implementing it yourself". I really don't find that to be the case... when you're developing with libs you are cognizant of your app endpoints and just add whats needed. It's better to consciously add a lib or turn a feature on than to just accept a big ball of yarn that some cowboy framework devs shat out. In the Rails app at my job it takes a ton of debugging just to figure out how many layers of decorators are firing off around every model.

I might even go so far as to say that a lot of where JVM has strength is that contexts are pretty measurable... Spring containers, Akka systems. You instantiate them and hold them basically in the palm of your hand, you understand your scopes. Any technology that allows application scope to slip out of your fingers and intermingle with everything else under the sun is something I'd be wary of.

There is a huge legal difference between getting caught doing drugs vs facilitating the trafficking of mass quantities of controlled substances. I don't think casual marijuana smokers should get all up in arms like "hey man they are just going after everybody now!", do they offer the same support to heroin cartels?

Even if you're against the war on drugs I don't think you should really take it as a personal slight when someone operating on this scale gets arrested. Unless you really 100% believe that distribution of heavy narcotics doesn't damage society.

I'd like to at least offer some support to this -- I'm not sure I agree with all of your conclusions but since switching to Scrum mentality I definitely notice that the goals of our team have become much less ambitious... Rather than insulating for development cycles that may not yield much for a few of months but in the end equals proper enterprise architecture, the business/devs have found it much easier to focus on small iterative tasks that are much more superficial in the grand scheme of things.

I think engineering processes benefit when they are insulated from direct scrutiny of timelines. Resorting to tackling smaller, neater issues in Agile seems to be a sign that there is a lack of technical leadership/vision. This is not necessarily the fault of Agile (I don't mind Agile, I think projects can succeed with it if approached properly), I simply agree that increasing process rigor seems to be a symptom of an engineering/mgmt team that is floundering.

In short, if your team's problem is that things are getting "messy" and "hectic" because of too much crappy copy/paste code, I have seen how Agile can just deepen that wound while masking it in a veil of "tangible progress" seen in burndown charts. Then, again, if you don't have the right people you don't have the right people. What else can you say?

my company is "totally agile" (they wish) and it gives the management layer something to do cuz they are analyzing JIRA all day coming up with horse-shit charts, so of course they keep pushing all of it like its solving high-level organizational issues in ways we will experience tangibly sometime by 2018...

if you got rid of all that process infrastructure and just had a few programmers discussing proper design patterns now & then our company would be lightyears ahead of where it is now. i think everything becomes cargo cult when you refocus technical people onto processes that discourage technicality to make management feel fuzzy about their stats. people aren't more productive now, they are just gaming the agile system

process planning has its place i just feel like its becoming the focal point of discussion rather than a means to it. bureaucracies will continue to grow, what can you do other than look for a job with a more technical group that is focused on code?

i'm a former academic, i was astronomically more productive in that setting and there were no real process restrictions imposed at all. people need to be cultured into code & self-education, not process. optimize process once there is a good technical core in place, not in lieu of one

(i've also noticed that fake agile just promotes a lack of requirements docs from business... in that case i'd be better off doing damn waterfall with a half-decent reqs doc anyway...)

Do the repo owners get a cut? :)

I tried it on Clojure then imagined how guilty I'd feel buying it knowing Rich Hickey wouldn't get any money but the printers would get lots, hah.

heh thanks for the words of encouragement. i'm going to try now, yes. but basically.... to be fair i'm getting paid a pretty good wage so i really just need some experience to avoid looking like a dabbler. I'm looking at senior level positions now & they seem to want to see that you're really committed to a certain type of code.

I went from research Matlab / dabbling in C# at a dayjob, to RoR, to javascript, to now pretty hardcore backend Java (EE, Spring, high concurrency for last 2 yrs), trying to move into Clojure a bit more. Most well-funded companies don't seem to like polyglots from what I can tell... But yeah I will be looking for the ones that do, plus I am studying C++ to make a bit of a more lateral move into audio digital signal processing / data-mining / machine learning.

I don't know... companies tell me I'm too expensive for the level of experience I have, yet all the high-paid seniors I work with are awful so I don't want to lower my salary expectations unless I can find a company thats actually worth sacrificing for.

yep there are also a lot of situations in enterprise where you don't exactly know how your code will be used in near future (in <5 years it could grow to exponentially more customers using it compared to soft releases to select test customers).

Because of that I'd say.... it's not like I see a lot of people get raked over the coals for it, but there is definitely some organizational disappointment when a silly scaling bug crops up in production[1]. I think as a mature dev it is proper to internalize as many performance-related techniques as possible so that you can write more performant code with each subsequent project. Sometimes it doesn't even affect the amount of initial coding hours, it is just one of the freebie gains that comes with experience.

[1] Many more such bugs would probably occur if the company didn't occasionally get around to load-testing, so that should be part of CI or dev cycle if possible.

Honestly, I'd rather they make it a paid service than a dystopian clusterfuck of advertising espionage.

I'm more likely to quit the wonnnnderful "free" service you describe and defend because it "costs millions of dollars a month to run". Those millions of dollars are just R&D on how to get advertisers more intimate data. If they didn't pay for all that gross research, the infrastructure wouldn't cost nearly as much.

This cost-center point is key. I work for one of the big international electronics corps as an enterprise web dev, but they have no interest in the quality of our work. They don't understand it, they have no belief in codebase maintainability (everything is new development or bugfix, basically as soon as an app is written it becomes frozen legacy code that will sit for 5+ years until a rewrite is finally budgeted).

A lot of the superiors they hire really affect the architectural policies of the company. For example, we can use Spring & jQuery but little else. Now they are trying to write apps with heavier front-end & getting into trouble cuz its a mess of jQuery callbacks with no real js framework. Why? The main architect has never heard of React, Angular, backbone, etc. (not to mention the fact that I'm one of like 2 devs who can actually write js, and unfortunately the other does not possess the equally rare skill: "uses proper design patterns"). You'd think I was joking until you met a bunch of corporate consultant programmers.

Basically, these companies try to boil web development into an old-school "IT" process but the level of intellect is astonishingly low. I've been lookin for a new gig at a tech company for quite a while but it was tough to make a change cuz this is my first gig out of grad school, had to prove myself a bit & build resume.

The only relief is that the corp "cost-center" IT kindof job I have is such a mess that its a good place to study & relax, pace of work is veeeeery slow.

o man i would love to try "brothers" :) does the offer still stand? i am one of those "controller" pc gamers...

(i am "fat0wl" on steam too!)

hah i am jealous of you both... i don't get the kickbacks, i just get one more team of consultants that i have to work with on every project who relish the tiny bit of authority they have now been granted (Message Broker team thinks they're hot shit cuz they are a bottleneck on every project & we can't revise our service interfaces without their permission).

Funny thing is, all the consultants who specialize in these products are awful pseudo-programmers & I end up bypassing their techs (WMB, BPM, etc.) with pure Java solutions whenever possible since the overhead of speaking to them creates like a 10x productivity loss. If the nosy busybodies (awful below average pseudo-programmers within our own group) didn't tattle to mgmt to help them reign us back in, the whole company would be none-the-wiser & we'd get work done at breakneck speed... ALL MOST OF US NEED IS A GODDAMN SPRING OR EE CONTAINER + THE JVM, hah.