wooorm
Joined 12 years ago
Karma
44
Submissions
31
Member since
Jul 31
Recent submissions
Comment · 11 years ago
alex is open to suggestion. See http://alexjs.com/#contributing on how to contribute. So if you have a word to add or phrasing to remove let us know!
Comment · 12 years ago
Perhaps you should ;) If, I’d be interest to know how it goes!
Comment · 12 years ago
It’s a very interesting idea. Would it work accurate enough when scaled to 160+ languages?
Comment · 12 years ago
I pushed a fix, incorporating your suggestions, and your examples in the specs.
Thanks a lot!
Comment · 12 years ago
It’s an interesting thought. I might fiddle on it, but I’m not sure it would work in practice (d’oh). Thanks!
Comment · 12 years ago
I agree the task is neither impossible nor useless. There’s work to do. Short passages should be supported. I do however think franc does a good job, and adds support for some languages which before today have never (I think) been supported. Franc, certainly, “attempt”s to fix language detection, which I would argue is an AI-complete problem.
Comment · 12 years ago
Thanks ;)
Comment · 12 years ago
By `correct language` I mean the language you expect, by `second` and `third` I mean `2.` and `3.` in the previously mentioned demo: http://wooorm.github.io/franc/). I think we’re talking about the same thing!
Anyway, yeah, franc is for language detecting, but it’s optimised for many languages and works best at longer text. It’s a trade-off. For less languages and shorter texts, check out https://github.com/shuyo/ldig
Comment · 12 years ago
Ha! Some very nice examples, I have to say :)
Anyway, You’re completely right. Italian is `und` due to LTE 10 characters, the others are slightly off due to short input too, but the demo (http://wooorm.github.io/franc/) shows the correct languages in the second or third place though!
Comment · 12 years ago
No full-frequency data is kept, only 300 top-trigrams are identified. A quick through the source also reveals wooorm/trigrams, and wooorm/udhr, as sources!
Comment · 12 years ago
Oh you’re right. I think I have a fix in mind, will work on it. Thanks so much!
Comment · 12 years ago
I’ll investigate this, but I think I excluded the preamble’s for trigram creation. Sure, the words will be a bit similar, but it’ll be a lot of work to compile 380 fixtures from other sources.
I’ll investigate that too. But it’s lots of work, this already was, give me some time :)
Comment · 12 years ago
Thanks! Currently, the UDHRs are crawled, and I’d rather not include exceptions and maintain their plain-text and XML/JSON versions by hand. If you’re into growing the language, I suggest contacting the Office of the High Commissioner of Human Rights of the UN, and the Unicode project, or fork wooorm/udhr and add support, I’ll merge :)