logan__dev

Tutorial on how to scrape job offers from LinkedIn using Puppeteer and RxJS

Hi all! Over the past 17 months, I have been collecting data on job offers from various job boards such as Glassdoor, LinkedIn, StackOverflow, Dice, and many others. In total, I have gathered approximately 14 million unique job offers for developers. I have written a blog article where I explore this data and rank the top 8 most in-demand programming languages.

Please note that this analysis is based solely on job offers and not on surveys conducted with actual developers! It is an objective study that considers ONLY the job offers that were found.

I hope you find it interesting!

Hi!

Over the past 17 months, I have been collecting data on job offers from various job boards such as Glassdoor, LinkedIn, StackOverflow, Dice, and many others. In total, I have gathered approximately 14 million unique job offers for developers. I have written a blog article where I explore this data and rank the top 8 most in-demand programming languages.

Please note that this analysis is based solely on job offers and not on surveys conducted with actual developers! It is an objective study that considers ONLY the job offers that were found.

I hope you find it interesting!

Hi all!

For 1 year I have been scraping job portals like Linkedin, Glassdoor, Dice etc. and selecting the dev related jobs from it. After that time, I have a database of more than 10 Million dev job offers.

For this blog/study, I have selected a smaller dataset of job offers, including those that:

Had Salary information (with a minimum of $5K and maximum of $1M)

Have been found more than 1 consecutive days. (This excludes 1 day offers that may be posted wrongly)

Also for the language categorization, only the TITLE of the job offer has been analyzed. This means that for example, a title of "Backend developer" would be discarded, since it does not contain any language or stack valid on it. Analyzing only the title also filters out offers that require many languages and are fuzzy.

Hope you like the salary distribution chart and the article also, if there are any doubts about the study let me know in the comments!

Note: I advertise that the blog post has "minimal", "non-intrusive" ads. Understand that this may help keep my work into the future, thanks!

Thanks for the feedback, I would definetly do it

Hmm could try something, Glassdoor, Indeed and Dice have a lot of them, so I could filter out them or put less weigth on their job offers. What also needs to be solved, is, what is considered to be a 'Python' job for example. If python is a on a job 'tag', I count it, but many offers have many tags with many different languages and frameworks, and that is no good for the study. Maybe in future, I would just count what languages/stack is specified on the title. Just saying. Thanks for your comments, I would try to improve the next study I make.

Not enough data. There are so few Clojure job offers with salary data. But, it may be yes.