jamesk_au

For example, is GPT-4’s list of ~100k tokens sufficient to understand and generate every non-obsolete word in the English language (per, say, a standard dictionary)? Or even every word in the training data?

If not, do we have examples of ordinary words that it is impossible for GPT-4 ever to understand or generate? What happens when it encounters those words and is unable to tokenize them; are they simply ignored (eg omitted from the input vector, or set to 0 or some sort of null token)?

Comment · 3 years ago

For a fun exploration of this, see the short story “Crystal Nights”, by Greg Egan: https://www.gregegan.net/MISC/CRYSTAL/Crystal.html

Comment · 3 years ago

I also had this experience, thank you for fixing it so quickly. The game is fun!

Comment · 9 years ago

Possibly relevant (from the Firebase FAQ[1]):

Why was my Realtime Database reported bandwidth lower than average between September 2016 and March 2017?

For our bandwidth calculations, we normally include SSL encryption overhead (based on layer 5 of the OSI model). However, in September 2016, we introduced a bug that caused our bandwidth reporting to ignore encryption overhead. This might have resulted in artificially low reported bandwidth and bills on your account for a few months.

We released a fix for the bug in late March 2017, returning bandwidth reporting and billing to their normal levels.

Does not excuse the issue with support.

[1] https://firebase.google.com/support/faq/

Comment · 9 years ago

This analysis from Ben Thompson (Stratechery[1]) just yesterday would be a great place to start:

"Imagine a Twitter app that, instead of a generic Moment that is little more than Twitter’s version of a thousand re-blogs, let you replay your Twitter stream from any particular moment in time. Miss the Oscars gaffe? Not only can you watch the video, you can read the reactions as they happen, from the people you actually care enough to follow. Or maybe see the reactions through someone else’s eyes: choose any other user on Twitter, and see what they saw as the gaffe happened.

What is so powerful about this seemingly simple feature is that it would commoditize “live” in a way that is only possibly digitally, and that would uniquely benefit the company: now the experience of “live” (except for the shock value) would be available at any time, from any perspective, and only on Twitter. That such a feature does not exist — indeed, that the company’s stated goal is to become more like old media, instead of uniquely leveraging digital — is as good an explanation for why the company has foundered as any."

[1] https://stratechery.com/2017/twitter-live-and-luck/

Comment · 9 years ago

Chris Lattner talked about some of these issues on the Accidental Tech Podcast a few weeks ago: http://atp.fm/205-chris-lattner-interview-transcript/

It was a great episode and is worth the listen for those interested.

On Swift adoption at Apple:

"The Swift team itself has specific goals they need to achieve before there can be truly, across-the-board adoption at Apple. ABI stability is the number-one thing [35:30] that prevents framework developers, for example, from adopting Swift. That's a really important thing. That's one of the reasons it's always a really high priority. Swift has been adopted by application developers and other things. The Dock is public. Swift Playgrounds app is public. The Music app in iOS is publicly known. So there are definitely some big adopters.

More broadly though, the big problem is that I think, I won't speak for everybody but many, many people doing [36:00] Objective-C development at Apple are chomping at the bit. They want to be using Swift. It's really just a matter of getting the technology problems solved and checking off the things that are holding people back. It's not about people dragging their feet and not wanting to use it."

On whether to adopt Swift now:

"I don't [1:14:30] think Objective-C is going to go away anytime soon. Apple still supports C and C++ and there's no obvious benefit of dropping Objective-C, and obviously they have a ton of Objective-C code themselves."

He also described Apple's approach to some of the strategic questions that arose early on, such as whether to just invest in making Objective-C better instead of introducing Swift, and the various trade-offs involved.

Comment · 9 years ago

"It is our goal to train a trillion-parameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware."

Comment · 10 years ago

In telling the other side of the story, the Dash developer published his (apparently secret) recording of a 7-minute long telephone call he had with a person at Apple who purports to be speaking on behalf of Phil Schiller.[1]

Leaving aside whatever inferences might be drawn from the fact that the developer saw fit to record and publish the recording in the first place, here's a brief summary of what was said:

- There was at least one other developer account "linked" to the Dash developer account. In this context, "linked" means that the accounts "shared the same details": they were enrolled in the Apple Developer Program "with the same credit card number", and "used the same test devices".

- Apple says that at least one of those other developer accounts "definitely had fraudulent activity": "It was not your direct account but it was a linked account." Warnings about fraudulent activity were sent to the linked account. No warnings were sent to the Dash account.

- The Dash developer asked: "Why didn't you notify me beforehand though, and let me know that an account that's linked to mine is doing fraudulent activity, so I can do something about it?" The answer is "because they were linked"; "we see them as the same entity". "If we have accounts that have enrolled in our program using the same credit card, they are the same legal entity; they are the same." So Apple believed they had notified the Dash developer because Apple believed the same person was behind the linked accounts. (Prudence would dictate notifying all accounts at risk of termination; we'll see what happens in future.)

- Apple's position is that no mistakes were made. The Dash developer account was linked to an account with fraudulent activity based on the facts known to Apple.

- The Dash developer says (in his blog post) that he "helped a relative get started by paying for her Apple's Developer Program Membership using my credit card" and "handed her test hardware that I no longer needed".

- Apple says they are "working with" the Dash developer to "unlink the accounts", which (I speculate) may involve some attempt to verify the Dash developer's claim that the linked account was used only by a relative and not by him. If the accounts are unlinked, there would be no reason for the Dash account to remain closed.

[1] https://blog.kapeli.com/dash-and-apple-my-side-of-the-story

Comment · 10 years ago

> If I had an algorithm that produced halfway decent results in an automated fashion, I wouldn't use it to get into the legal technology market. I'd set myself up as a litigation funder and make boatloads of money with accurate valuations of potential investments.

Legalist is a recent YC startup aiming to do exactly that:

http://www.newyorker.com/business/currency/what-litigation-f...

Comment · 10 years ago

"In some Australian cities, in the central business district, the traffic light push buttons don't work at certain times — for example, 7 am to 7 pm, Monday to Wednesday, and 7 am to 9 pm, Thursday to Saturday.

The rationale is that between these busy hours, the road traffic is relatively constant, and also, the pedestrian crossings are in continuous use. So the lights do their own thing.

However, the traffic light push buttons do work outside these hours, including all day Sunday."

Comment · 10 years ago

Here you go: https://news.ycombinator.com/item?id=11924009