Introduction & Why this articleWe have learned previously how a Large Language Model (LLM) is able to understand any input sentence & produces outputs using attention mechanism.  However, the process is more complex than it appears. When LLMs encounter text, they don’t see words; they see numbers. Each word in a sentence is translated into … Continue reading Tokenization: The language of AI World & its significance in controlling costs (1/2)