Posts on Rui's Blog

Diffusion LLM paper reading

Sat, 14 Mar 2026 00:16:00 -0800

1 Residual Context Diffusion Language Models

Method

Traditional dLLMs discard the [MASK] at unpredicted positions in each round and reset them as new blank [MASK]s. To avoid wasting computational power, this method retains these discarded semantics and transforms them into vectors for the next round of prediction.

Algorithm

Generate the residual vector by performing a weighted sum of the model’s predicted probability and the word embeddings:

$\Delta_i^{(t_k)} = \sum_{j=1}^V p_{i,j}^{(t_k)} E_{j,:}$

Encoder-only

Mon, 02 Mar 2026 23:25:00 -0800

What is Pre-training?

Initializing neural network model parameters through self-supervised learning (unlabeled data).

The goal is to learn a universal language understanding capability.

BERT

BERT Pre-training Tasks

Masked Language Modeling (MLM)

A certain percentage (15%) of tokens in the input sequence are replaced, and the model is tasked with predicting what those original words were.

The 8-1-1 Rule

80% of the selected tokens are replaced with [MASK]: Acts like a cloze test, enabling the model to learn bidirectional semantic context.
10% are replaced with a random token from the vocabulary: Since the model doesn’t know which tokens are random, this improves error-correction capabilities and forces the model to rely on the global context to generate the correct vector representation.
10% remain as the original token: This anchors the model’s representations toward the actual “true” embeddings.

Optimization Objective

$$\mathcal{L}{MLM} = - \sum{i \in m} \log P(x_i | \tilde{X}; \theta)$$ Where $\tilde{X}$ is the corrupted input sequence, $m$ is the set of chosen masked positions, and $\theta$ represents the model parameters.

Yan:Foundational Interactive Video Generation

Mon, 09 Feb 2026 00:16:00 -0800

Abstract

Introduction

The first paragraph primarily discusses the applications and deficiencies of Interactive Generative Video (IGV). AIGC is evolving from generating text and images to video synthesis, and has now progressed to IGV. IGV requires dynamic reactions to user input, with applications ranging from virtual simulation to embodied intelligence. However, the deficiencies of current methods lie in the lack of high visual fidelity, sustained temporal coherence, and rich interactivity. Generated content also remains static after creation, failing to adapt in real-time. (Not high-definition, not coherent, not interactive).

LSTM

Tue, 27 Jan 2026 16:49:00 -0800

pending

RNN

Mon, 26 Jan 2026 23:57:00 -0800

结构

纵向看，每一列相当于一个小的传统神经网络

x：输入层，长度为word embedding的维度

U：输入层到隐藏层的权重

s：隐藏层 = f(U * xt + W * s(t-1) + bias),即输入层的计算结果加上上一时刻的结果乘上一个权重W

V: 隐藏层到输出层的权重

o：输出 = softmax(V * st + bias)

注意

由公式可见，每个隐藏层s的值不仅依赖于当前时刻的输入，而且依赖于上一个时刻的的结果s(t-1)
U,V,W都是共享的
不一定每一个时刻都要有一个输出，也就是说不是每一个小神经网络都必须有输出，它可以是不完整的

举例

这是一个时间序列做预测的例子，一共有三天的数据，预测明天的数据

w1 = U = 1.8， w2 = W = -0.5，w3 = V = 1.1

x0 = 1, x1 = 0.5, x2 = 0.5 这里的输入只是数字而不是向量

Day0

s0 = f(1.8 * 1 + -0.5 * 0) = f(1.8) = 1.8

Tokenizer

Mon, 26 Jan 2026 22:20:00 -0800

sasa

Word Embedding

Mon, 26 Jan 2026 22:20:00 -0800

One-hot

每个token表示为一个长向量，每个向量的维度为词表的大小，向量中只有一个维度的值为1，一般为该词在词表中的索引

举例

假设用 one-hot对句子进行表示，对句子分词之后，我们可以得到[‘我‘，’爱‘，’北京‘，’天安门‘]，可以用one hot（独热编码）对单词进行编码。具体为：

“我”可以表示为[1,0,0,0]

“爱"可以表示为[0,1,0,0]

‘北京’可以表示为[0,0,1,0]

‘天安门’可以表示为[0,0,0,1]

问题

one-hot向量之间彼此正交，无法识别每个词之间的关系（无法计算相似度，都为0）
维度爆炸。维度和词表有关系，模型训练难度增大
向量稀疏，难以捕捉语义关系，因为数据点分布系数，距离度量失效。而且稀疏矩阵存储效率低

词嵌入

词嵌入（Word Embedding）是将人类语言中离散、抽象的文本词汇，通过算法映射为计算机可理解的「连续、稠密的低维实数向量」的文本表示技术

分布式表示

词嵌入将词表示为一种分布式的形式。分布式表示是将词转化成一个定长、稠密并且存在语义关系的向量，即分布相似的词，是有相似的语义的

Word2Vec

CBOW

Skip-gram

Leetcode 541 - Reverse String II

Mon, 19 Jan 2026 23:12:00 -0800

Leetcode 541. Reverse String II

Problem Link

Problem Description

Given a string s and an integer k, reverse the first k characters for every 2k characters counting from the start of the string.

If there are fewer than k characters left, reverse all of them.

If there are less than 2k but greater than or equal to k characters, reverse the first k characters and leave the other as original.

Leetcode 344 - Reverse String

Mon, 19 Jan 2026 09:00:00 -0800

Problem Link

Problem Description

Write a function that reverses a string. The input string is given as an array of characters s.

You must do this by modifying the input array in-place with O(1) extra memory.

Example 1:

Input: s = [“h”,“e”,“l”,“l”,“o”] Output: [“o”,“l”,“l”,“e”,“h”] Example 2:

Input: s = [“H”,“a”,“n”,“n”,“a”,“h”] Output: [“h”,“a”,“n”,“n”,“a”,“H”]

Approach

Opposite Direction Pointers: One pointer starts at the beginning, and the other starts at the end.

Both pointers move inward simultaneously with the same step size until they cross each other (left > right), at which point the traversal ends.

Leetcode 27 - Remove Element

Sun, 18 Jan 2026 22:21:00 -0800

Problem Link

Problem Description

Given an array nums and a value val, you need to in-place remove all instances of that value and return the new length of the array.

Do not use extra space for another array; you must do this by modifying the input array in-place with O(1) extra space.

The order of elements can be changed. You do not need to consider the elements beyond the new length.

Shortcodes Samples

Tue, 13 Jan 2026 01:07:25 -0800

This is a sample post intended to test the followings:

Default hero image.
Different shortcodes.

Alert

The following alerts are available in this theme.

This is sample alert with type="success".

This is sample alert with type="danger".

This is sample alert with type="warning".

This is sample alert with type="info".

This is sample alert with type="dark".

This is sample alert with type="primary".

This is sample alert with type="secondary".

Image

A sample image without any attribute.

A sample image with `height` and `width` attributes.

A center aligned image with `height` and `width` attributes.

A image with `float` attribute.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Cras egestas lectus sed leo ultricies ultricies. Praesent tellus risus, eleifend vel efficitur ac, venenatis sit amet sem. Ut ut egestas erat. Fusce ut leo turpis. Morbi consectetur sed lacus vitae vehicula. Cras gravida turpis id eleifend volutpat. Suspendisse nec ipsum eu erat finibus dictum. Morbi volutpat nulla purus, vel maximus ex molestie id. Nullam posuere est urna, at fringilla eros venenatis quis.

Markdown Samples

Tue, 13 Jan 2026 00:43:25 -0800

This is a sample post intended to test the followings:

A different post author.
Table of contents.
Markdown content rendering.
Math rendering.
Emoji rendering.

Markdown Syntax Rendering

Headings

The following HTML <h1>—<h6> elements represent six levels of section headings. <h1> is the highest section level while <h6> is the lowest.

H1

H2

H3

H4

H5

H6

Paragraph

Xerum, quo qui aut unt expliquam qui dolut labo. Aque venitatiusda cum, voluptionse latur sitiae dolessi aut parist aut dollo enim qui voluptate ma dolestendit peritin re plis aut quas inctum laceat est volestemque commosa as cus endigna tectur, offic to cor sequas etum rerum idem sintibus eiur? Quianimin porecus evelectur, cum que nis nust voloribus ratem aut omnimi, sitatur? Quiatem. Nam, omnis sum am facea corem alique molestrunt et eos evelece arcillit ut aut eos eos nus, sin conecerem erum fuga. Ri oditatquam, ad quibus unda veliamenimin cusam et facea ipsamus es exerum sitate dolores editium rerore eost, temped molorro ratiae volorro te reribus dolorer sperchicium faceata tiustia prat.

Posts on Rui's Blog

Diffusion LLM paper reading

1 Residual Context Diffusion Language Models

Method

Algorithm

Encoder-only

What is Pre-training?

BERT

BERT Pre-training Tasks

Masked Language Modeling (MLM)

The 8-1-1 Rule

Optimization Objective

Yan:Foundational Interactive Video Generation

Abstract

Introduction

LSTM

RNN

结构

注意

举例

Day0

Tokenizer

Word Embedding

One-hot

举例

问题

词嵌入

分布式表示

Word2Vec

CBOW

Skip-gram

Leetcode 541 - Reverse String II

Leetcode 541. Reverse String II

Problem Description

Leetcode 344 - Reverse String

Problem Description

Approach

Leetcode 27 - Remove Element

Problem Description

Shortcodes Samples

Alert

Image

A sample image without any attribute.

A sample image with height and width attributes.

A center aligned image with height and width attributes.

A image with float attribute.

Markdown Samples

Markdown Syntax Rendering

Headings

H1

H2

H3

H4

H5

H6

Paragraph

A sample image with `height` and `width` attributes.

A center aligned image with `height` and `width` attributes.

A image with `float` attribute.