How GPT models work: for data scientists and ML engineers
During training, the model is exposed to a lot of text, and its weights are tuned to predict good probability distributions, given a sequence of input tokens. In the context of OpenAI GPT models, common and short words typically correspond to a single token, such as the word “We” in the image below.