clarecorthell
Joined 15 years ago
Karma
428
Submissions
16
Member since
Mar 11
@clarecorthell
Data Scientist, Product Manager
datasciencemasters.org
View on Hacker NewsRecent submissions
Comment · 5 years ago
No bothersome problem is too small! If you'd like to share, we would be eager to look into this. Reach us through the app or at osm-questions [a] lyft.com.
Comment · 5 years ago
Lyft study shows crucial OpenStreetMap road attributes are fresh and high quality in 30 North American cities, as compared to groundtruth. Blog post and paper detail the process, methodology, and results.
Comment · 12 years ago
Of people who use these tools, only a very small percentage of them also build said tools. Which makes sense. In industry, companies want to build meaningful products and data sets that provide value which can be exchanged for money. That means delivering some majority of the value in the shortest amount of time possible. For that reason, engineering teams have little room for R&D, including the kind you're hinting at. Scikit, for example, has largely academic contributors (http://scikit-learn.org/stable/about.html). Not surprising. Talents of pedantry, proof-writing, and pushing the boundaries of theory probably lies in academia as opposed to industry. Industry rewards shipping code (useful oversimplification) and academia rewards novel theory (yet another oversimplification).
Solving problems requires understanding what solutions exist (and whether they can be used, must be built upon, could be used in ensemble, etc). Choosing among those solutions requires understanding (to some varying degree) why and how the solution solves the problem. Choosing the correct out-of-the-box solution is not trivial.
One very real danger here is unwittingly lying with statistics. Which is arguably worse than wittingly doing so.
Comment · 12 years ago
I'm curious about the audience here -
What career goal is driving your interest in learning Data Science?
Comment · 12 years ago
Deterrent roadblocks are a huge problem. In fact, it's the primary reason education can't happen solely through a book or video lecture series. Students need mentorship to gain motivation and remove roadblocks. Teachers, TAs, and cohorts fill this need in the university setting. It makes sense that when you learn something new, you face more unknown unknowns than known unknowns. It's nearly impossible to ask questions about the former because you don't know how to begin formulating a question, while the latter is googleable and likely solvable.
The programmers' standard roadblock remover? google -> "stackoverflow" + problem
Comment · 12 years ago
Start here: http://bit.ly/uwintrodatascience. It's the first bullet.
Comment · 12 years ago
You'll find R resources here: https://github.com/datasciencemasters/go/blob/master/r-resou...
The OSDSM maintains a focus on python resources.
Comment · 12 years ago
I took six months off for the OSDSM. I am very aware that it was a luxury to do so. I had to take loans, move out of my apartment, and I studied 10 hours per day.
If you weren't working 10 hours per day, 6 days per week for six months (1440 hrs), and you took 6 hours per week to do the same, it would take you more than 4 years to finish a similar curriculum.
Managing your time is still the hardest part of self-study. It always will be. Most people pay institutions to structure their lives with a workload, deadlines, time off, expectations, and consequences (positive and negative). Having the time for self-study is a luxury few people have; most people won't have the time, money, and opportunity to give up for a "classic liberal education," and by extension they won't have time for self-study, either (one such conversation on the topic: http://www.newyorker.com/culture/culture-desk/loud-nathan-he...). That's part of why new forms of education like The OSDSM are so necessary. Such curriculums fit exceptional cases that are less and less the exception.
Comment · 12 years ago
The truthful meta-answer which you probably don't want to hear: We only have a statistical likelihood (of dubious confidence) of what will transpire in the future. We have no knowledge of the future. None. Same is true of your knowledge of your future problems. Use some basic bayesian logic and guess in an educated manner, like with most prediction problems.
More practical advices: Find out what problems you're interested in working on, then google, read books, engage people working on those problems to tell you more about how they solve them. Another great place to start is to look at your business' biggest inefficiencies and informational gaps, and determine which of them could potentially be addressed with prediction or statistical inference. This used to be called Business Intelligence. Even a coffee shop can benefit from understanding simple seasonality.
Comment · 12 years ago
It's worthwhile to explain why many people find the OSDSM useful.
The topics listed do indeed require depth to acquire an adequate proficiency to understand and command them. The OSDSM is most useful for people who don't know where to start, which is often the hardest part. The hardest things are often the most trivial in hindsight.
Typical interaction:
Q: I want to be a Data Scientist. What do you think I should study first?
A: Build a basic proficiency in linear algebra, programming in python, and statistics. Then take cursory classes in the subjects in the OSDSM that interest you. You'll figure out what depth is most meaningful to you from there.
Q: What does a Data Scientist actually need to know?
A: Totally and completely depends on what you want to do. There are people who crunch click logs all day, people who comb the tendrils of search algorithms, and yet others who seek terrorists and criminals in statistical signs among the bits and bytes. See my last answer for relevant guidance.
e.g. https://medium.com/@clarecorthell/the-brief-multi-tweet-enum...
Comment · 12 years ago
Try CodeAcademy and CS106a (how I learned programming at Stanford) https://www.udemy.com/cs-106a-programming-methodology
I've started cataloging some of the best beginning resources here. Main benefit is that people can battle out what the best resources are in PRs. https://github.com/datasciencemasters/go/blob/master/basic-p...
Comment · 12 years ago
Sure, you can learn to use a technology and still be braindead or poor at analysis.
This is an applied curriculum with a focus on specific technologies that enable an analysis-bent people to leverage their brains with technology. That's why data work is so beautiful -- it's space to demonstrate unquantifiables like curiosity, diligence, creativity, and grit.
The quality of your projects is likely a good metric for your aptitude for data work, which is why I strongly advise working on a personal project.
I'd love to get more pull requests with more materials that teach analysis!
Comment · 12 years ago
Working with a technology that is more highly documented is better when you're starting out. Especially when you're teaching yourself and StackOverflow is your TA. Python is very appropriate for someone who's new to data work.
Otherwise, asking what technology to use is like asking what mode of transportation to use to get to a destination -- it's not the point. The important part is that you arrive. Some days walking over the mountain is the least sensible method, other days high seas make taking the boat around it impossible. The tool that gets the job done is the best tool.