Rabdos Prep
06-11-2026
The Three Pillars
How It Connects
Pretraining, SFT, and RL fine-tuning all run on the same kernels (MatMul, attention, LayerNorm). The leverage point is RL fine-tuning with GRPO: sample many candidate kernels for a problem, score each one automatically (compiles, passes fuzz tests, how fast), and push the policy toward the best. The fuzzer is what makes the reward concrete and human-free.
Learning Sequence
Week 1: ML and RL theory (Transformers, RLHF, Proximal Policy Optimization, Group Relative Policy Optimization). Week 2: GPU systems and the fuzzer (CUDA Memory Model, compute-sanitizer, Fuzzing Axes). Week 3: Integration (Oracle Problem, Strides, Reward Design).
First-Principles ML
- Neural Networks
- Activation Functions
- Loss Functions
- Automatic Differentiation
- Backpropagation
- Gradient Descent
- Optimizers
- Training Loop
Transformers and Fine-Tuning
- Tokenization
- Attention
- Multi-Head Attention
- Transformer Block
- Supervised Fine-Tuning
- LoRA
- Direct Preference Optimization
- Data Curation and Contamination
- ML Evaluation
- Scaling Laws
Kernel Engineering
- Roofline Analysis
- GPU Memory Hierarchy
- Floating Point Data Types
- Tensor Cores
- CUDA Kernel Programming
- Triton
- CUTLASS and CuTe
- FlashAttention
- GPU Benchmarking
- TPU Architecture and Pallas
- MLIR
- XLA
- TorchDynamo and Inductor
- NCCL and All-Reduce
- KV Cache and PagedAttention
- Quantization