MikeDegany/ppo-LunarLander-v3

PPO Agent Playing LunarLander-v3

This is a trained model of a PPO agent playing LunarLander-v3, implemented from scratch (CleanRL-style).

Mean reward over 10 evaluation episodes: 14.62 +/- 120.00

Some tips about implementing ppo for LunarLander from scratch

• PPO on LunarLander is non-monotonic. A run can reach a great policy (reward 200+) mid-training and then destabilize and get worse by the end. Saving/evaluating only the final checkpoint is a real risk, not a hypothetical one — it's exactly what caused the bad video here. The fix: evaluate periodically during training and keep the best checkpoint seen, not just the last one.
• A single video is one sample, not the mean. If actions are sampled stochastically (probs.sample()) rather than greedily (argmax), even a genuinely decent policy can crash in any individual episode. A good mean eval reward does not guarantee a good-looking replay video.
• Variance can dominate over hyperparameters. I tried ~10 combinations (different seeds, larger batch sizes via more parallel envs, KL early stopping, truncation bootstrapping) and none reliably reached the "solved" ~200 mean reward — some "tuned" configs did consistently worse than the plain defaults. Don't assume a bigger/tuned run is automatically better; verify with real eval numbers over enough episodes (10 is noisy; ~30+ gives a more honest picture of the variance).
• Correctness fixes (e.g. bootstrapping value estimates on time-limit truncation) matter for principle but aren't a silver bullet for training instability — don't expect one flag to fix a high-variance policy.

Hyperparameters

exp_name: ppo
seed: 2
torch_deterministic: True
cuda: False
capture_video: False
env_id: LunarLander-v3
total_timesteps: 1000000
learning_rate: 0.00025
num_envs: 4
num_steps: 128
anneal_lr: True
gae: True
gamma: 0.99
gae_lambda: 0.95
num_minibatches: 4
update_epochs: 4
norm_adv: True
clip_coef: 0.2
clip_vloss: True
ent_coef: 0.01
vf_coef: 0.5
max_grad_norm: 0.5
target_kl: None
eval_episodes: 20
eval_freq: 0
bootstrap_truncation: False
repo_id: MikeDegany/ppo-LunarLander-v3
save_path: /home/UNT/md0708/RL/ppo_model.pt
batch_size: 512
minibatch_size: 128