https://low-hands.github.io/posts/internship/llm/Recent content in LLM on Rui's BlogHugo -- gohugo.ioenMon, 02 Mar 2026 23:25:00 -0800https://low-hands.github.io/posts/internship/llm/llm-architecture/encoder-only/Mon, 02 Mar 2026 23:25:00 -0800https://low-hands.github.io/posts/internship/llm/llm-architecture/encoder-only/<h1 id="what-is-pre-training">What is Pre-training?</h1> <p>Initializing neural network model parameters through <strong>self-supervised learning</strong> (unlabeled data).</p> <p>The goal is to learn a <strong>universal language understanding capability</strong>.</p> <hr> <h1 id="bert">BERT</h1> <h2 id="bert-pre-training-tasks">BERT Pre-training Tasks</h2> <h3 id="masked-language-modeling-mlm">Masked Language Modeling (MLM)</h3> <p>A certain percentage (15%) of tokens in the input sequence are replaced, and the model is tasked with predicting what those original words were.</p> <h4 id="the-8-1-1-rule">The 8-1-1 Rule</h4> <ul> <li><strong>80%</strong> of the selected tokens are replaced with <code>[MASK]</code>: Acts like a cloze test, enabling the model to learn bidirectional semantic context.</li> <li><strong>10%</strong> are replaced with a <strong>random token</strong> from the vocabulary: Since the model doesn&rsquo;t know which tokens are random, this improves error-correction capabilities and forces the model to rely on the global context to generate the correct vector representation.</li> <li><strong>10%</strong> remain as the <strong>original token</strong>: This anchors the model&rsquo;s representations toward the actual &ldquo;true&rdquo; embeddings.</li> </ul> <h4 id="optimization-objective">Optimization Objective</h4> <p>$$\mathcal{L}<em>{MLM} = - \sum</em>{i \in m} \log P(x_i | \tilde{X}; \theta)$$ Where $\tilde{X}$ is the corrupted input sequence, $m$ is the set of chosen masked positions, and $\theta$ represents the model parameters.</p>