README.md
4.5 KB · 164 lines · markdown Raw
1 ---
2 library_name: stable-baselines3
3 pipeline_tag: reinforcement-learning
4 tags:
5 - stable-baselines3
6 - reinforcement-learning
7 - deep-reinforcement-learning
8 - ppo
9 - LunarLander-v3
10 - custom-reward
11 - reward-shaping
12 - actor-critic
13 ---
14
15 # PPO LunarLander flip, recover and land agent
16
17 This repository contains a Stable-Baselines3 PPO
18 actor-critic agent trained on a customised
19 `LunarLander-v3` environment.
20
21 ## Learned task
22
23 The curriculum teaches one policy to:
24
25 1. complete a full rotation in a fixed direction;
26 2. recover upright and arrest angular motion;
27 3. enter the landing zone;
28 4. reduce descent speed and land safely.
29
30 Reward configuration version: `v5-soft-touchdown-refinement`.
31
32 ## Changes in this upload
33
34 1. Continued from the selected Phase D checkpoint.
35 2. Added an altitude-dependent vertical-speed target.
36 3. Added a near-ground quadratic descent-overspeed penalty.
37 4. Added a dedicated in-zone crash penalty.
38 5. Reduced risky late horizontal corrections using a deadband.
39
40 ## Reward design
41
42 The shaped reward includes:
43
44 - one-off rotation-progress and flip-completion rewards;
45 - an upright post-flip recovery reward;
46 - horizontal guidance towards the landing zone;
47 - an altitude-dependent vertical-speed target;
48 - attitude and angular-speed control;
49 - a near-ground descent-overspeed penalty;
50 - distinct penalties for off-zone landings and in-zone crashes.
51
52 | Parameter | Value |
53 |---|---:|
54 | `required_rotations` | 1 |
55 | `rotation_direction` | 1 |
56 | `upright_tolerance_radians` | 0.3 |
57 | `recovery_angular_velocity_tolerance` | 0.3 |
58 | `pre_flip_original_reward_weight` | 0.15 |
59 | `post_flip_original_reward_weight` | 3 |
60 | `pre_flip_center_weight` | 25 |
61 | `pre_flip_horizontal_speed_weight` | 15 |
62 | `pre_flip_shaping_weight` | 1 |
63 | `pre_flip_shaping_clip` | 10 |
64 | `flip_corridor_half_width` | 0.35 |
65 | `flip_entry_penalty_weight` | 600 |
66 | `rotation_progress_bonus` | 300 |
67 | `flip_completion_bonus` | 500 |
68 | `recovery_bonus` | 250 |
69 | `flip_landing_bonus` | 2000 |
70 | `landing_without_flip_penalty` | 600 |
71 | `no_flip_terminal_penalty` | 500 |
72 | `failed_landing_penalty` | 900 |
73 | `outside_zone_landing_penalty` | 1200 |
74 | `in_zone_crash_penalty` | 1500 |
75 | `post_flip_shaping_weight` | 2 |
76 | `post_flip_shaping_gamma` | 0.999 |
77 | `post_flip_shaping_clip` | 30 |
78 | `post_flip_center_weight` | 100 |
79 | `post_flip_horizontal_speed_weight` | 50 |
80 | `post_flip_vertical_speed_weight` | 100 |
81 | `post_flip_angle_weight` | 70 |
82 | `post_flip_angular_speed_weight` | 40 |
83 | `post_flip_leg_contact_weight` | 30 |
84 | `landing_zone_half_width` | 0.2 |
85 | `post_flip_zone_excess_weight` | 180 |
86 | `post_flip_target_vx_gain` | 0.6 |
87 | `post_flip_max_target_vx` | 0.4 |
88 | `post_flip_horizontal_deadband` | 0.08 |
89 | `post_flip_target_vy_high` | -0.45 |
90 | `post_flip_target_vy_near_ground` | -0.12 |
91 | `near_ground_height` | 0.6 |
92 | `safe_touchdown_vertical_speed` | 0.14 |
93 | `near_ground_overspeed_weight` | 120 |
94
95 ## Evaluation
96
97 Deterministic evaluation over
98 100 fixed-seed episodes:
99
100 | Metric | Value |
101 |---|---:|
102 | Mean shaped reward | -413.81 |
103 | Mean original reward | -671.00 |
104 | Full-rotation rate | 15.0% |
105 | Recovery rate | 9.0% |
106 | Recovery given a flip | 60.0% |
107 | Safe-landing rate | 0.0% |
108 | Flip-and-land rate | 0.0% |
109 | Terminal in-zone rate | 44.0% |
110 | In-zone crash rate | 13.0% |
111
112 ## Architecture
113
114 - Algorithm: PPO
115 - Policy: MLP actor-critic
116 - Actor hidden layers: `[128, 128]`
117 - Critic hidden layers: `[128, 128]`
118 - Observation dimensions: `11`
119 - Discrete actions: `4`
120
121 ## Training configuration
122
123 | Parameter | Value |
124 |---|---:|
125 | Phase timesteps | 10000000 |
126 | Parallel environments | 32 |
127 | Learning rate | 5e-05 |
128 | Rollout steps per environment | 1024 |
129 | Batch size | 64 |
130 | Optimisation epochs | 4 |
131 | Gamma | 0.999 |
132 | GAE lambda | 0.98 |
133 | Entropy coefficient | 0.005 |
134 | PPO clip range | 0.1 |
135 | Training seed | 42 |
136
137 ## Replay
138
139 - Seed: `20000`
140 - Original reward: `-669.61`
141 - Shaped reward: `-294.96`
142 - Rotations completed: `1.00`
143 - Flip completed: `False`
144 - Recovery completed:
145 `False`
146 - Landed safely: `False`
147 - Outcome: `episode_ended_without_flip`
148
149 [![Flip, recovery and landing replay](replay.gif)](replay.mp4)
150
151 [Open the full MP4 replay](replay.mp4)
152
153 ## Repository files
154
155 - `ppo-LunarLander-v3-flip-128x128.zip`: selected PPO model
156 - `flip_landing_reward_wrapper.py`: custom environment wrapper
157 - `training_config.json`: PPO training settings
158 - `reward_config.json`: reward configuration
159 - `episode_results.csv`: fixed-seed evaluation episodes
160 - `results.json`: machine-readable evaluation summary
161 - `config.json`: compact model metadata
162 - `replay.gif`: model-card preview
163 - `replay.mp4`: full replay
164