Courses
Course catalog

Robot Reinforcement Learning: From PPO to Self-Improving VLAs

Series 01 taught robots to imitate; Series 02 teaches them to improve. Derive TRPO→PPO and DQN→TD3→SAC, rebuild the PPO details that matter, then work through visual RL, the sample-efficiency race, and model-based RL from World Models to Dreamer 4. Finish with the robot recipe (parallel simulation, sim-to-real, real-world RL) and the frontier: RL post-training of VLAs and the PPO→GRPO bridge to LLMs.

  1. 01

    Foundations: from policy gradient to PPO

    1. MDPs, returns and Bellman equationsPlanned
    2. Policy gradients: REINFORCE, baselines and the log-derivative trickPlanned
    3. Actor-critic and GAE: estimating the advantagePlanned
    4. From TRPO to PPO: trust regions and the clipped objectivePlanned
    5. PPO implementation details that matter, and how to debug PPOPlanned
  2. 02

    Value-based and off-policy RL: DQN to SAC

    1. The DQN family: replay, target networks, double Q and RainbowPlanned
    2. DDPG and TD3: deterministic policy gradients and overestimationPlanned
    3. SAC: maximum-entropy RL and automatic temperature tuningPlanned
    4. Offline RL and offline-to-online: CQL, IQL, AWR, RLPDPlanned
    5. PPO vs SAC in the parallel-sim era: FastTD3, FastSAC, FlashSACPlanned
  3. 03

    Visual RL and sample efficiency

    1. RL from pixels: SAC-AE, RAD, DrQ, DrQ-v2Planned
    2. Representations and generalization: CURL, SPR, pretrained encoders, DMC-GBPlanned
    3. The sample-efficiency race: Atari 100k, high UTD, resets and BBFPlanned
    4. Scaling RL networks: SimBa, BRO and 1000-layer agentsPlanned
  4. 04

    Special topic: model-based RL and world models for control

    1. Model-based RL basics: Dyna, PETS and MBPOPlanned
    2. World Models and PlaNet: learning to dream in latent spacePlanned
    3. Dreamer v1–v3: learning behaviors by latent imaginationPlanned
    4. Planning with learned models: MuZero, EfficientZero V2, TD-MPC2Planned
    5. Dreamer 4 and generative world models: IRIS, STORM, DIAMONDPlanned
  5. 05

    RL on robots: parallel sim, sim-to-real and real-world learning

    1. Learning to walk in minutes: massively parallel sim + PPOPlanned
    2. Sim-to-real: domain randomization, privileged teachers, RMA and ASAPPlanned
    3. Humanoid motion tracking: DeepMimic to BeyondMimic and SONICPlanned
    4. Real-world RL: SERL, HIL-SERL and DayDreamerPlanned
  6. 06

    Frontier: RL post-training for VLAs and LLMs

    1. RL for diffusion and flow policies: DPPO, DSRL, FPOPlanned
    2. RL fine-tuning VLAs in simulation: SimpleVLA-RL, RL4VLA, πRLPlanned
    3. VLAs that learn from experience: π*0.6/RECAP, RL Token, GR-RLPlanned
    4. World models as RL environments: WMPO, VLA-RFT, Ctrl-WorldPlanned
    5. From PPO to GRPO: RL for robots vs RL for LLMsPlanned

Join the waitlist

Get new lessons, cards and the full roadmap poster when they ship. No spam.

Sign-ups open soon. Until then, write to contact@dpintelli.com.