ajhai
Joined 17 years ago
Karma
1,092
Submissions
234
Member since
Jul 15
Recent submissions
Comment · 2 months ago
Anthropic has just announced extending the limits through August 31st with plans to make the new limits permanent.
Comment · 1 year ago
https://x.com/ajhai/status/1899528923303809217 something I have been working on for a few months now.
Comment · 1 year ago
It is inference latency most of the time. These VLA models take in an image + state + text and spit out a set of joint angle deltas.
Depending on the model being used, we may get just one set of joint angle deltas or a series of them. In order to be able to complete a task, it will need to capture images from the cameras, current joint angles and send them to the model along with the task text to get the joint angle changes we will need to apply. Once the joint angles are updated, we will need to check if the task is complete (this can come from the model too). We run this loop till the task is complete.
Combine this with the motion planning that has to happen to make sure the joint angles we are getting do not result in colliding with the surroundings and are safe, results in overall slowness.
Comment · 1 year ago
Building a wheeled robot with arms to help automate household chores - https://x.com/ajhai/status/1891933005729747096
I have been working with LLMs and VLMs to automate browser based workflows among other things for the last couple of years. Given how good the vision models have gotten lately, the perception problem is solved to level where it opens up a lot of possibilities. Manipulation is not generally solved yet but there is a lot of activity in the field and there are promising approaches to solve (OpenVLA, π0). Given these, I'm trying to build an affordable robot that can help around with household chores using language and vision models. Idea is to ship capable enough hardware that can do a few things really well with the currently available models and keep upgrading the AI stack as manipulation models get better over time.
Comment · 2 years ago
I've tested Q4 on M1 and it works though the quality may not likely be the same as you'd expect as others have pointed out on the issue.
Comment · 2 years ago
You can already run these models locally with Ollama (ollama run llama3.1:latest) along with at places like huggingface, groq etc.
If you want a playground to test this model locally or want to quickly build some applications with it, you can try LLMStack (https://github.com/trypromptly/LLMStack). I wrote last week about how to configure and use Ollama with LLMStack at https://docs.trypromptly.com/guides/using-llama3-with-ollama.
Disclaimer: I'm the maintainer of LLMStack
Comment · 2 years ago
You can actually do this with LLMStack (https://github.com/trypromptly/LLMStack) quite easily in a no-code way. Put together a guide to use LLMStack with Ollama last week - https://docs.trypromptly.com/guides/using-llama3-with-ollama for using local models. It lets you load all your files as a datasource and then build a RAG app over it.
For now it still uses openai for embeddings generation by default and we are updating that in the next couple of releases to be able to use a local model for embedding generation before writing to a vector db.
Disclosure: I'm the maintainer of LLMStack project
email: ajay [at] trypromptly [dot] com
twitter: @ajhai
Building https://trypromptly.com and https://makerdojo.io