Robust Outer-Loop Control of Quadrotors via Reinforcement Learning

This paper presents a reinforcement learning (RL)–based outer-loop controller for quadrotor UAV trajectory tracking and its real-world experimental validation. The proposed approach integrates RL into a standard cascaded flight-control architecture by replacing the conventional PID outer loop while retaining the onboard attitude and body-rate PID controllers. This hierarchical design preserves reliable inner-loop stabilization while leveraging RL to address nonlinear dynamics, coupling effects, and modeling uncertainty in translational motion. The controller is trained entirely in a physics-based simulation using Proximal Policy Optimization (PPO) and transferred directly to a Crazyflie quadrotor without additional tuning. Performance is evaluated through real-world figure-8 trajectory tracking experiments with varying time scales to impose increasing dynamic demands. Compared to a conventional PID outer-loop controller operating under identical conditions, the RL-based controller consistently reduces effective phase delay and achieves lower position and velocity tracking errors, particularly for aggressive trajectories. The results demonstrate robust sim-to-real transfer and highlight the potential of learning-based outer-loop control as a drop-in enhancement to classical quadrotor flight controllers.