This is a review of the course: 生成式AI時代下的機器學習(2025)】第五講：大型語言模型訓練方法「預訓練–對齊」(Pretrain-Alignment) 的強大與極限 from professor Hung-Yi Lee from NTU.
Link to the course video & slides.

1. Pre-training: Given large input data available, train the model
2. Supervised Fine Tuning (SFT, a.k.a Instruction Tuning): Given a smaller set of curated supervised data (input, ground-truth answer), fine-tune the pre-trained model.
3. RLHF: Reinforcement Learning with Human Feedback

Alignment means stage 2 & 3 where human interactions are involved. We hope the machine output aligns with humans’ expectations.

The performance gap is significant before and after the “Alignment” process.
You do NOT need large-scale data for alignment. (e.g.
Quality is the key! LIMA: Less Is More for Alignment