eshvk

You are right, my mistake.

> H1B:

The problem now is that the visas seem to be harder to obtain and have a longer delay. I know people who were stuck waiting for multiple months to do a transfer.

> L1:

My understanding is that this doesn't provide a path to a green card.

One thing I have observed is that engineering roles at national labs tend to have a lot of paperwork needed for an interview. I can understand a faculty role requiring recommendation letters, a long application form to fill up but an engineering role requiring all that makes it rather annoying.

Not sure if Oak Ridge does this tho. :)

> In the broader sense, I guess my question is what are the next steps to build on this basic recommendation system? What is a good way to serve recommendations based on user behavior in near-real time and how do these systems take feedback to improve the models.

In the past, I have helped build Lambda architectures where we use a batch model to build a content vector space, build estimates of users in batch, update those in realtime (using PubSub/Kafka) based on user feedback.

Other online mechanisms could be to use Contextual Bandits: e.g. use context in terms of user interactions with the several arms of the bandits being recommendation choices etc. This interaction data can be used to continuously improve your policy. Of course, the key benefit over a Matrix Factorization setup where the interaction matrix is continuously rebuilt over time based on new data, is the in built exploration which minimizes regret.

Not really sure what your central question then is: The general theme for building a recommender system architecture is:

1. Decide whether you are okay with a batch approach or an online learning approach or a hybrid.

2. Start simple with a batch approach (similar to what you are doing):

a) Get features ready from your dataset (assuming you have interaction data) : Pre-processing via some big data framework (Map Reduce, Data flow etc)

b) Build a vector space and nearest neighbors datastructures.

c) Stick both into a database optimized for reads

d) Stick a service in front of it and serve.

Once you are happy with 2, you can try out variations involving either online updates to your recommender system which involves changes to the type of database you might want to optimize. etc

There are roles called Machine Learning Engineers/Research engineers. The work does require enough background in ML; as in it is non trivial to implement an algorithm, consider approximations while scaling without knowing how the math works.

Although, you might be able to get away with not knowing recent ML work by having a solid education in linear algebra, probability and statistics.

I work in this field. I can tell you that the maths involved is an easier (and more exciting) learning curve. A lot of software engineering already involves the tools necessary for getting good af math: example SVG transformations and matrix algebra, learning how to manipulate abstraction in operating systems etc.

I'd suggest an approach where you use your (presumably good) software engineering skills to join an engineering gig in a data science team. Then, gradually pick up tools, slowly pivoting into fully fledged ML/AI. It is definitely possible as a career path.

Spotify extensively uses Scala for data infrastructure and ML feature engineering.

> I work in Sweden. I don't think I have seen ever anyone fired. What I have seen is some of the best hiring processes. The process looks at the candidate values, and skills. And there is a discussion about what we expect from her, and what she expects from the job.

I have worked for a few years in a company that is based in Sweden but also has a very large U.S. presence. Couple of points:

1. The recruiting process is involved in Sweden is more involved than in the U.S, agreed. However it is not perfect. False positives do enter the system. Folks there have started getting junior candidates intern/contract as a way of evaluating them. This unfortunately has its own bias.

2. Yes, I think it is impossible to fire people in Sweden. What is typically done in those rare situations is to shuffle you in such a way that you don't do any work. Imagine a head of a team suddenly head in title but with no reports.

3. On the U.S. side, interviews are way easier but then at will employment takes care of that in those rare situations.

There is/was a option available for F1 Visa students called OPT (optional practical training). It could in theory run up ~24 months depending on the company having everify or not.

> Probably will start with either linear algebra or maybe set theory?

This can work. I spent a year or so before grad school at work, I brushed up by spending a summer drilling through a Linear Algebra book.

A new thing I have been trying out recently is find an application, find something I am confused by, dig deeper through the layers of mathematics till I uncover the basics of the confusion, then re-learn and work up.

So BFS vs DFS. The latter seems more like up my alley. YMMV

I think the poster assumed that data science/computational work could provide the mathematical background the OP is looking for.

Because Anaconda is a package of tools that include numpy,scipy (python numerical and scientific computing) and a notebook (a "live editing" utility).

While this might help, I wonder if this is too roundabout a way to learn math.

> Sure I have a new found love for math, but I'm not going to be accepted as a mathematician ever without the rigor of a formal education

I think there are two separate things here:

1. Being a research mathematician requires a degree of expertise which is easier to come by with formal education.

2. Knowing how to use APIs and make correct distributional assumptions; despite the bullshit fed by our industry, it is not easy or non trivial to design a completely idiot proof API. So having know-how of how the math works under the hood is helpful even if you are going to just use the API.

Also things like qualitative user testing, a/b testing would go into every single pixel of his design. I am not particularly sure if the most important separation of the music listening population is into hoarders vs nomads. You'd evaluate that, make a decision. Things might get muddled there.

> My theory is that they have a few people manually curate a list with good songs for a while to train an AI, but when they finally turn control over to the AI it falls on its face and starts adding shitty cover band music to the list.

A curated playlist (not discover weekly or release radar for example) is editorially curated and driven by a human. Doesn't involve an AI for now.