Mix-Spectrum for Generalization in Visual Reinforcement Learning

Visual Reinforcement Learning (RL) trains agents on policies using images showing the potential for real-world applications. However, the limited diversity in the training environment often results in overfitting with agents underperforming in unseen environments. To address this issue, image augmentation is utilized in visual RL to increase data diversity, but the effectiveness is limited due to the potential to alter the semantic information of the image. Therefore, we introduce Mix-Spectrum, a straightforward yet highly effective frequency-based augmentation method that maintains the semantic consistency of data and enhances the agent's focus on semantic information. The proposed method combines two existing methods: mixing amplitudes of original and reference images, and Random Convolution. Through this synergistic combination of established methods, our approach not only maintains the advantages of each method but also introduces a novel characteristic that enhances performance. Furthermore, the proposed method stands out for adaptability when integrated with any visual RL algorithm, whether off-policy or on-policy. Through extensive experiments on the DMControl Generalization Benchmark (DMControl-GB) and Procgen, our method demonstrates superior performance compared to existing frequency-based, normalization-based, and image augmentation methods in zero-shot generalization. In DMControl-GB, our method improved by 35.5% over the baseline and 15.2% over the second-best. In Procgen, it achieved 15.2% and 10.1% improvements, respectively.