How Does Most devs don't understand how LLM tokens work Work?

Most devs don’t understand how LLM tokens work because they think of them like letters on a page, but really, they're more like building blocks for sentences.

Imagine you're playing with LEGO bricks. Each brick is a token, and together they make up a bigger picture, just like words and phrases make up stories or messages. But instead of colors and shapes, each token holds part of a word or even a whole word.

When you write "Hello, world!", it's not one block, it's several tokens, like "Hello", ",", " ", "world", and "!", all separate pieces that come together to make the full sentence. An LLM (like ChatGPT) reads these tokens one by one, using them to predict what comes next, just like you might guess the next word in a story.

Some devs think of words as single blocks, but really, they're often split into smaller parts, it's like cutting a block in half and treating each part as its own brick. This makes things more complicated than they seem!