EarlyOom
Joined 4 years ago
Karma
205
Submissions
29
Member since
Mar 22
Recent submissions
Comment · 1 year ago
Shocking how poor frontier models perform on simple visual tasks. Best-in-domain tool calling will
Become the norm
Comment · 1 year ago
VLM Run | Member of Technical Staff, ML Systems | Full-time | Hybrid Bay Area, CA | https://vlm.run | 150k-220k / yr + Equity
VLM Run is a first-of-its-kind API dedicated to running Vision Language Models on Documents, Images, and Video. We’re building a stack from the bottom-up for ‘Visual’ applications of language models that we believe will make up > 90% of inference needs in the next 5 years.
Hybrid from Bay Area, CA
Looking for experience in any of the following:
* ML Domains: Vision Language Models, LLMs, Temporal/Video Models
* Model Training, Evaluation, and Versioning platforms: WnB, Huggingface
* Infra: Python, Pytorch, Pydantic, CUDA, Torch.compile
* Devops: Github CI, Docker, Conda, API Billing and Monitoring
Comment · 1 year ago
This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on your dataset (we're working on this) to push accuracy beyond what you get from an unspecialized frontier model.
Comment · 1 year ago
You can try out some of our schemas with Ollama if you want: https://github.com/vlm-run/vlmrun-hub (instructions in Readme)
Comment · 1 year ago
VLMs are able to take context into account when filling in fields, following either a global or field specific prompt. This is great for e.g. unlabeled axes, checking a legend for units to be suffixed after a number, etc. Also, you catch lots of really simple errors with type hints (e.g. dates, addresses, country codes etc.).
Comment · 1 year ago
You can! it works with Ollama https://github.com/vlm-run/vlmrun-hub
At the end of the day its just schemas. You can decide for yourself if its work upgrading to a larger, more expensive model.
Comment · 1 year ago
We can do bounding boxes too :) we just call it visual grounding https://github.com/vlm-run/vlmrun-cookbook/blob/main/noteboo...
Comment · 1 year ago
We convert to a JSON schema, but it would be trivial to convert this to yaml. There are some minor differences in e.g. tokens required to output JSON vs yaml which is why we've opted for our strategy.
Comment · 1 year ago
OCR seems to be mostly solved for 'normal' text laid out according to Latin alphabet norms (left to right, normal spacing etc.), but would love to see more adversarial examples. We've seen lots of regressions around faxed or scanned documents where the text boxes may be slightly rotated (e.g. https://www.cad-notes.com/autocad-tip-rotate-multiple-texts-...) not to mention handwriting and poorly scanned docs. Then there's contextually dependent information like X-axis labels that are implicit from a legend somewhere, so its not clear even with the bounding boxes what the numbers refer to. This is where VLMs really shine: they can extract text then use similar examples from the page to map them into their output values when the bounding box doesn't provide this for free.
Comment · 1 year ago
Would love to chat! reach out [email protected]
Comment · 1 year ago
That's one of our main focuses, yes: https://docs.vlm.run/api-reference/v1/fine-tuning/post-finet...