Tokens: Why ChatGPT Can't Count the R's in 'Strawberry'

A language model never sees letters or words — it sees tokens, chunks of ~4 characters. Understand tokens and a lot of AI's quirks (and your bill) suddenly make sense.

Read Original

Related