fantispug

> Practical/applied stuff seems to require levels of compute, training data, etc. that make it unapproachable for a home hacker for everything beyond basic exercises like a handwritten digit classifier.

Compute is only an issue if you want to train large foundational models from scratch. You can easily train non-neural models on CPU, and fine-tune smaller models (say <4B) on a commodity GPU.

For training data there's lots of datasets online (like Kaggle), but you can build your own and can label things yourself or get a language model to label them for you. Getting good data is the hardest part of ML so this is a good exercise.

Then do some hobby projects: classify whether your photos contain a photo of you or a particular friend/family member, predict the expense category from bank transaction exports, classify your emails into categories, extract book recommendations from Hacker News, ...

Multi layer perceptron. A tiny neural net.

Boosted trees over embeddings work surprisingly well too.

Their strength was the ability to keep at the very fast moving AI frontier.

When a new model is released they have been able to have a working implementation days later, not months when the model was redundant. And often it's much more usable than the one-off, unmaintained, research code written in a very fragile environment.

BLOOM was a decent fully open and very large model for its time, and a huge coordination effort. It was around the same time as another "joke", OPT, but that gave Meta the knowledge that let them build Llama. Huggingface didn't have the money to compete in that race, but I'd bet it helped them stay at the frontier of AI software.

Google could have owned this space; they had TensorFlow hub, but it was too hard to use and too locked down. Huggingface made it easy to run and train transformers, and got the timing just right.

I'm glad someone is trying to build truly open models (along with the great work at Allen AI), because the likes of Meta and Alibaba are only going to release weights as long as it suits their business.

Yes, this seems to be a common capability - Anthropic and Mistral have something very similar as do resellers like AWS Bedrock.

I guess it lets them better utilise their hardware in quiet times throughout the day. It's interesting they all picked 50% discount.

Meta's primary business is capturing attention and selling some of that attention to advertisers. They do this by distributing content to users in a way that maximizes attention. Content is a complement to their content distribution system.

LLMs, along with image and video generation models, are generators of very dynamic, engaging and personalised content. If Open AI or anyone else wins a monopoly there it could be terrible for Meta's business. Commoditizing it with Llama, and at the same time building internal capability and a community for their LLMs, was solid strategy from Meta.

I find this style changes the way I think about and write code transformations. It's also in shell pipelines, R's magrittr, and Clojure's thread macros, and can be emulated in some OO languages with methods that return the transformed object itself.

I have seen it work better than LSH.

Each time you embed a document you search for approximate nearest neighbours before adding it, so it is O(N) like MinHash. Vector indexes like HNSW and PQ have better performance/quality tradeoffs than SimHash LSH which is the analogue of MinHash for cosine distance.

The quality depends on what you mean by near duplicate and the embedding model you use. Current models work well, and if you have labelled data you can fine tune them to be better.

The main drawback is the additional cost of embedding all the documents, especially for longer documents. But this cost has dropped really quickly with smaller models, better optimisations, and faster hardware.

It covers a lot of the fundamentals in some detail (attention and transformers, decoding, transfer learning) that are underneath current cutting edge NLP; this is still a very good foundation likely to be good for several more years.

What might be missing is in-context learning, prompt engineering, novel forms of attention, RLHF, and LoRA (though it covers adaptors), but this is still changing rapidly and the details may be irrelevant in another year. If you have a look at a recent course like Stanford CS224N 2023 there's a lot of overlap.

> Maybe my data structure has a hash table of items, as well as a direct link to the largest item. When I say: "write the function to insert a new item in the list, and remember to update the largest item if it is larger than the current one", would Copilot do the right thing? Each step is easy in itself (adding an element to a hash, comparing an item to another one).

In general it won't solve all your problems, but it's helpful for automating simple things like this (but you still need to test edge cases). With this prompt in Python (which I'm more familiar with):

  from dataclasses import dataclass
  from typing import TypeVar
  
  T = TypeVar('T')
  
  @dataclass
  class MaxDict:
      items: dict[T, float]
      max_value_item: T

      def add_item(

It completed:

    def add_item(self, item: T, value: float):
        if value > self.items[self.max_value_item]:
            self.max_value_item = item
        self.items[item] = value

This was my second attempt; first I called it `max_item` and the completion did something about comparing the key.

With the prompt

  // Print the fibonnaci sequence, except the number 5.
  int main() {

I got the result:

    int a = 0;
    int b = 1;
    int c = 0;
    while (c < 100) {
        c = a + b;
        a = b;
        b = c;
        if (c != 5) {
            cout << c << endl;
        }
    }
    return 0;

This happened in mathematics about a decade ago when Wolfram Alpha came out. Lecturers started complaining that Wolfram Alpha could solve assignment problems with worked steps.

In both cases I think this is a real opportunity; we can let students get more quickly to bigger problems and systems thinking by leveraging these tools. It requires professors to start thinking innovatively about how to teach and assess these subjects.

These issues are often interactions between packages. I use TRAMP daily without freezing, but only when I disable company mode in shell modes. VSCode certainly requires much less of this tweaking, even with a variety of extensions installed.

The killer feature of racket is it is very easy to make Domain Specific Languages (including the teaching language) and related tooling. However last time I looked the library ecosystem didn't seem great; there were many libraries but few that were actively maintained.

Kaluza-Klein theory is an attempt to exhibit a transformation that charged particles travel on "straight" lines in a 5-dimensional space. The fifth dimension is supposed to be a very small circle. There are some experimental implications, but the circle is so small that we can't detect them (it's similar with string theories).

String theory tries to do the same thing with other forces; so it's not clear whether there's such a transformation. Forces are technically simpler than embedding extra dimensions that there are no evidence for.

I'm the same; I use evil-mode in emacs and find the friction of switching between them small.