awaythrowact

Fair. I agree Arrow is still more of a vision than anything else.

> it didn't even provide a shape attribute

I suspect this has to do with the project's focus. I think they aspire to be a back-end to DataFrame libraries, which are generally 2d. I think they (correctly) are ceding the "n-dimensional tensor computation" space to the current incumbents.

What is your point? (I honestly do not understand.)

Blackbear's comment was that writing libraries in C allows those libraries to be deployed broadly in many compute environments.

Jakob's reply (as I understood it) was that outside of the big Deep Learning libraries, this has not really happened. There is no C implementation of Pandas that allows for redeployment in other non-python compute contexts.

My point was that, with Arrow, this type of cross platform compatibility is coming to python dataframe libraries. You can prototype Dask code that runs on your laptop, then deploy it to a production Spark cluster, knowing the same Arrow engine is underpinning both. Or at least that's the vision. Obviously Arrow is still relatively young. But the point is, it's far from certain that the long-term global optimum for the ecosystem isn't sticking with "all libraries are written in C".

What are your favorite open source Python packages written in Rust?

> for library writers, how many people might use it can be a deciding factor on investing the time

Yes. Especially given that a high percentage of library development these days is being done by industry, not just academia. (At least in machine learning.) They're not writing packages for fun or for academic citations, they want users.

Makes sense. I don't use TF Eager, but I do use Jax, and Jax lets you arbitrarily compose JITed and non-JITed code, which made me think that might be a viable pattern. I guess I wondered if there might be something like "nonstatic_jit(foo)" that would do "julia style" compiling on function foo, in addition to "jit(foo)" that compiles foo to optimized XLA ops. Probably impractical. Thanks.

If the next machine learning killer-app model requires autodiff'ed dynamic control flow, do you think Google/Facebook will build that capability into XLA/TorchScript? Seems like if NLP SOTA requires dynamic control flow, Google will build it? Maybe they let you declare some subgraph as "dynamic" to avoid static optimizations? But maybe the static graph assumption is so deeply embedded into the XLA architecture, they'd be better off just adopting Julia? (I honestly don't know the answer, asking your opinion!)

Always happy to help. Especially the last day or so - I've been waiting on some long training loops so it's been a pleasant diversion.

To be 100% honest with you, there's pretty much 0% chance of me adopting Julia in the next 12 months. I evaluated it before embarking on my current project, but ended up going with Python, and now I have several thousand lines of Python code that work fine, and I'm not going to rewrite it all in the near future. At some point in the medium term, I'll re-evaluate Julia. But until then, I don't want to lead anyone on any wild goose chase. Even if you solved this problem, and all my problems, I'm just not in a position to switch right now. So for that reason I think I'll hold off on issuing a community wide call for help. But I'm cautiously optimistic that at some point in the future I'll be writing Julia professionally.

A lot of this is also probably cultural rather than language features. The first thing they teach any Python data science boot camp initiate is: never "from numpy import *", always "import numpy as np" and yet in Julia "using" appears more common than "import"...

I also wouldn't read too much into my one example. It was initially meant just as an illustrative point, but somehow I was so bad at explaining it that it took tons of comments for me to get my minor point across. I do think the MLJ guys are properly on the right track, and that should work fine for most people who don't mind Macros. Maybe I'm in the minority in hating Macros.

The more commonly cited issues around compile time caching, tooling, etc. are boring to list off yet again, but probably the right areas of focus for the community, in terms of both impact and ease of implementation.

More generally, I really do think you're better off talking to people like @systemvoltage, who have actually given Julia more of a chance than I have. If I worked for Julia Computing I'd be reaching out to him and begging to get his brain dump on all the challenges he faced. In any business, it's always easier to reduce churn and learn from your (unhappy) customers, then it is to convert new prospects, whether that business is programming languages or investment banking.

Best of luck. Sincerely.

> since `fit` would've likely come from the same source

No.

As described in the great-great-great-great-great-great-great-great-great-parent comment, the problem I am describing is that of trying to combine models from multiple software authors. You may not have that problem. It may not be a common problem among Julia's academic user base. I do have that problem.

Thanks for reading.

Just to be clear, I like Julia, and think it has advantages over Python. I'm writing all this as someone who is cheering for Julia to break out of its HPC niche. Thanks.

@cbkeller

I know in Julia I can just precede every function call and object instantiation with "modulename." and solve the namespace problem that way. What I want to do instead, what I do in python, is bind one namespace of function methods to each object, so that as I code, I don't have to remember which module each each object came from. That is the appeal to me of "model.fit" over "module.fit(model)".

EDIT: This is not some "shave a few seconds off coding time" quality of life issue. This is a mission critical requirements in many enterprise contexts.

Scenario A: Model Development Team trains a model, serializes it, and sends the serialized model object to Production. You want Production to have to lookup which software Package the Model Development Team used for each model, just so they know which "predict" function to call? No, "predict" should just be packaged with the model object as a method, along with "retrain", "diagnose", etc.

Scenario B: I want to fit a dozen different types of models, from multiple packages, on the same data, to evaluate how they each do, and build build some sort of ensemble meta model. So I need to loop over each model object in my complicated ensembling logic. You want me to also programmatically keep track of which module each model comes from?

These are important, happens-every-day use cases in the enterprise.

I think the best solution for this in Julia is to package the model state with all the "method" functions in one struct. Again, this is what MLJ does. This is the closest Julia gets to OOP. But then you either need a lot of boilerplate code, or a Macro to unpack it all for you behind the scenes.

Never made it that far. This was a feature I use all the time in Python ML development (both consuming open source packages via an OOP interface and also writing in-house model classes) that I consider essential for my productivity and that Julia was missing.

<edit>I'm also nervous of relying on the recourse of asking package maintainers to edit their variable names to improve compatibility with the random third package I want to use; maybe the culture is different in Julia but in Python that's a good way to get laughed out of the issue tracker :) </edit>

If I were you I would maybe ask people like @systemvoltage who took the plunge and wrote a big project in Julia only to find they had trouble maintaining the project. Maybe one reason he can't upgrade without breaking everything is because of namespace collisions amongst his many dependencies? If not that, it's something like that.

I very frequently run into namespace collisions like that. I think they are quite common in large codebases.

I am aware of the ability to do eg "import TensorFlow; model = TensorFlow.model; TensorFlow.fit(model,data)"

As I mentioned previously, I find Python's OOP "model.fit" syntax to be better, for a variety of reasons.

Thank you for your engagement. Have a nice day.