zissou

Glad you're enjoying Open Torque Viewer.

It's my creation. :)

What are you talking about? Did you even look at their section on job search? It is most definitely a macro application.

Disregarding the fact that job search is a subtopic of unemployment, one of the main concepts in macro, you'll notice their model parameters are human capital, investment, and wage. The only quasi micro flavor of that model is search effort as these variables will almost always be in equilibrium based on some kind of an optimal stopping rule. Whether that rule has to do with finding a new job or discovering information about, say, the prices in the market dictates whether it would be a micro or macro use case. Again, if you actually looked at the model, their optimal stopping rule is when the person finds another job. And once again, the idea of jobs are the main ingredient in the concept of unemployment.... a macro topic. Not to mention, whether or not I'm searching for a job dictates whether I'm factored into the unemployment rate or not.

So..... yeah, job search is definitely not micro.

I thinks it's great that 2 well known economists with household names in the academy have taken to marketing programming to economists. I just wish they would have called this venture something other than Quantitative Economics when they launched it last year. It's not that it's the wrong name, it is just misleading.

A better name would have been Quantitative Macroeconomics, because that is clearly the target market. Yes, I know there are some generic topics in there like Linear Algebra that both micro and macro economists could use, but it's not like there is a lack of numpy and scipy examples on the Internet.

You bring up a good point in that the use cases for each language varies. R is a good example because one of the main reasons it is chosen is due to the fact that it already has the largest set of drop in econometrics operations for tons of extremely specific situations. In any other language, you'd often have to code a bunch of complex linear algebra by hand, which would totally negate any marginal benefits due to computation time.

In terms of econometrics, Python is going to surpass closed source software packages like SPSS, SAS, eViews, etc soon simply because the rate at which econometric procedures are being implemented in Python is growing steadily (e.g. via a statsmodels/pandas/numpy/scipy based stack). I don't think Python will ever pass R in this respect though as R has a large share of the statisticians helping add the implementations relative to Python.

I used to be gung-ho about convincing economists to use Python (started econpy.org in the 1st year of my PhD in economics, stopped updating it after 2 years). But now I just don't care. The vast majority of economists are horrible programmers, however most know how to script in at least 1 language, and a small number of them are actually really good at scripting. An extremely small number of economists know they're way around at least 1 entire general purpose programming language.

It seems that every time this gets brought up online a sea of infamous Fortran-programming-economists always tell you how fast Fortran is compared to everything else, signing it with "Fortran or GTFO". I've met hundreds/thousands of economists and econ grad students at schools ranked from 1 to 200, and I don't recall a single one that actually used Fortran (I'm sure there is a non-zero quantity -- I probably just haven't cared to talk with them b/c they are most likely macro theorists).

The fact of the matter is that in economics graduate school, your professors could care less what language you use. It isn't an algorithm competition, it's a story telling competition. But the type of story telling economists do is no less noble than writing elegant algos. Writing a story based on economic mechanisms and behaviors and supporting it with quality data, sound statistics, and logical/exact theories is no simple task.

A case of self-selection, of course.

I have been researching the used car market for the last year in an academic lab. I would love to share stories about data collection and management if Carlypso folks are interested. Not looking to be a consultant or being consulted or any kind of a "gig". As I'm sure they know, used car data is a crazy world of imperfect data points, especially as the car becomes older (>6 years old I'd say data quality begins to break down fast). What proportions of your data are scraped, purchased, or maybe even obtained from a free API like Edmunds.com? I doubt you're using Edmunds as I didn't catch any affiliation info on the site.

I laughed at the "oh yeah that makes sense" story of the big truck being priced higher in TX. My goto story is always the price of convertibles in the winter in MN vs. TX, or the price of an AWD sedan in the winter in MN vs. TX.

Speaking of AWD sedans, I know it's a rare car so this is nitpicky but your 2008 Saab 9-3 data classifies a Turbo X as an Aero. Also the Turbo X trim isn't listed for the wagon. At any rate, your estimate for the Sedan Turbo X is about 20% too low. Granted they only made 600 for the US market, I was still bummed that a car which is so dear to me had this issue. Haha, I'm not trying to be that guy whatsoever! In fact I was happy when I saw Saab was even listed as a Make as many don't even acknowledge them anymore! :(

Anyways let me know if you guys are up for a friendly talk.

Thanks for the info and I do look forward to the new API!

However my question still remains with regards to historic posts/comments. The historic aspect is really the import element here. Generally speaking, building an ngram viewer requires a collection of texts over time, with each text having some kind of metadata that is categorical, boolean, datetime, or numeric. Categorical data can always be made of numeric data by creating bins -- i.e. posts by people with karma or a ranking of 1-50, 51-150, 151-300, etc at the time the comment/post was created. Datetimes can also be made into useful categorical variables for an ngram viewer such as day of the week (to spot weekly seasonality trends) or day of the year (annual seasonality trends).

If I was allowed, I would be willing to write a scraper/crawler to discover as many historic threads (since: threads -> comments) as possible using HNSearch, but this could take a long time depending on rate limits and/or be subject to unknown biases within my discovery method. I'm sure you can understand why a "top-down" approach like a database dump would make for a much higher quality corpus than attempting the "bottom-up" approach of a crawler. I have no idea if a "database dump of everything" is even feasible as I don't know anything about the HN's backend infrastructure. However, if it is feasible, then I'm certain that I can work with whatever would be available. Adding structure to unstructured data is my bread and butter.

I really think this would be a very cool tool that a lot of people would enjoy, so I'm willing to do what is needed on my end to help make it work. After all, I'd be on the clock while working on this rather than just a hobby project, so the incentives are definitely aligned on my end.

If you want to discuss anything in private, I can be reached at the following reversed address: moc{dot}liamg{at}yalkcin{dot}wehttam

Economist and long time web scraper here.

In your original business model you wanted to understand the price of everything. In what ways did the problem of a lack of information on the demand side come up? That is, it is easy to scrape the price in many markets (supply side), but what kinds of conversations came up within your team about the lack of information on how many units were actually sold at a posted price?

By the way, glad to see you guys were able to make a business out of crawling. I've landed a handful of freelance gigs since leaving grad school based on scraping data for clients, but never tried to expand it to anything beyond consulting projects.

Minor in economics. Skip on taking extra macro classes and instead take as many econometrics and micro that you can. Economics (or another quantitative social science) is often the missing ingredient for a solid data science background. Sure physicists have good quantitative skills, but you also need to understand what the important/relevant [business] questions are to ask.