Ruijie Zheng

I am a first-year Ph.D. student in Computer Science at the University of Maryland, College Park, where I am fortunate to be advised by Prof. Furong Huang and Prof. Hal Daumé III. Before that, I obtained my Bachelor’s degree in Computer Science and Mathematics also from the University of Maryland, College Park. My research spans a variety of topics in reinforcement learning (RL), including model-based RL, transfer RL, adversarial RL, etc. My long-term goal is to make RL agents more robust and efficient.

In model-based RL, my work provided a novel insight into probabilistic environment model ensemble, which is commonly used in model-based RL algorithm. Based on the insight, we can substitute the ensemble with a single model and Lipschitz regularized value function to make the learning algorithm much more computationally efficient. For transfer-RL, I have worked on transferring domain knowledge under the drastic change of observation spaces (e.g., from vector-based observation to image-based observation). For adversarial RL, I have worked on observation attacks for the deep RL policy, as well as designing efficient algorithms to improve the agent’s robustness under attack. In addition, I have also worked on communication attacks in multi-agent reinforcement learning and developed a certifiable defense mechanism.

selected publications

ICLR 2023

Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function

Zheng, Ruijie, Wang, Xiyao, Xu, Huazhe, and Huang, Furong

In International Conference on Learning Representations 2023

Spotlight Presentation at NeurIPS 2022 DeepRL Workshop

Abs HTML

Probabilistic dynamics model ensemble is widely used in existing model-based reinforcement learning methods as it outperforms a single dynamics model in both asymptotic performance and sample efficiency. In this paper, we provide both practical and theoretical insights on the empirical success of the probabilistic dynamics model ensemble through the lens of Lipschitz continuity. We find that, for a value function, the stronger the Lipschitz condition is, the smaller the gap between the true dynamics- and learned dynamics-induced Bellman operators is, thus enabling the converged value function to be closer to the optimal value function. Hence, we hypothesize that the key functionality of the probabilistic dynamics model ensemble is to regularize the Lipschitz condition of the value function using generated samples. To validate this hypothesis, we devise two practical robust training mechanisms through computing the adversarial noise and regularizing the value network’s spectral norm to directly regularize the Lipschitz condition of the value functions. Empirical results show that combined with our mechanisms, model-based RL algorithms with a single dynamics model outperform those with ensemble of the probabilistic dynamics models. These findings not only support the theoretical insight, but also provide a practical solution for developing computationally efficient model-based RL algorithms.
ICLR 2023

Certifiably Robust Policy Learning against Adversarial Multi-Agent Communication

Sun, Yanchao, Zheng, Ruijie, Hassanzadeh, Parisa, Liang, Yongyuan, Feizi, Soheil, Ganesh, Sumitra, and Huang, Furong

In International Conference on Learning Representations 2023

Abs HTML

Communication is important in many multi-agent reinforcement learning (MARL) problems for agents to share information and make good decisions. However, when deploying trained communicative agents in a real-world application where noise and potential attackers exist, the safety of communication-based policies becomes a severe issue that is underexplored. Specifically, if communication messages are manipulated by malicious attackers, agents relying on untrustworthy communication may take unsafe actions that lead to catastrophic consequences. Therefore, it is crucial to ensure that agents will not be misled by corrupted communication, while still benefiting from benign communication. In this work, we consider an environment with N agents, where the attacker may arbitrarily change the communication from any C<\fracN-12 agents to a victim agent. For this strong threat model, we propose a certifiable defense by constructing a message-ensemble policy that aggregates multiple randomly ablated message sets. Theoretical analysis shows that this message-ensemble policy can utilize benign communication while being certifiably robust to adversarial communication, regardless of the attacking algorithm. Experiments in multiple environments verify that our defense significantly improves the robustness of trained policies against various types of attacks.
ICLR 2022

Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL

Sun, Yanchao, Zheng, Ruijie, Liang, Yongyuan, and Huang, Furong

In International Conference on Learning Representations 2022

Best Paper Award in NeurIPS 2021 Workshop of Safe and Robust Control of Uncertain Systems

Abs HTML

Evaluating the worst-case performance of a reinforcement learning (RL) agent under the strongest/optimal adversarial perturbations on state observations (within some constraints) is crucial for understanding the robustness of RL agents. However, finding the optimal adversary is challenging, in terms of both whether we can find the optimal attack and how efficiently we can find it. Existing works on adversarial RL either use heuristics-based methods that may not find the strongest adversary, or directly train an RL-based adversary by treating the agent as a part of the environment, which can find the optimal adversary but may become intractable in a large state space. This paper introduces a novel attacking method to find the optimal attacks through collaboration between a designed function named "actor" and an RL-based learner named "director’". The actor crafts state perturbations for a given policy perturbation direction, and the director learns to propose the best policy perturbation directions. Our proposed algorithm, PA-AD, is theoretically optimal and significantly more efficient than prior RL-based works in environments with large state spaces. Empirical results show that our proposed PA-AD universally outperforms state-of-the-art attacking methods in various Atari and MuJoCo environments. By applying PA-AD to adversarial training, we achieve state-of-the-art empirical robustness in multiple tasks under strong adversaries.
ICLR 2022

Transfer RL across Observation Feature Spaces via Model-Based Regularization

Sun, Yanchao, Zheng, Ruijie, Wang, Xiyao, Cohen, Andrew E, and Huang, Furong

In International Conference on Learning Representations 2022

Abs HTML

In many reinforcement learning (RL) applications, the observation space is specified by human developers and restricted by physical realizations, and may thus be subject to dramatic changes over time (e.g. increased number of observable features). However, when the observation space changes, the previous policy will likely fail due to the mismatch of input features, and another policy must be trained from scratch, which is inefficient in terms of computation and sample complexity. Following theoretical insights, we propose a novel algorithm which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task to use as a model-based regularizer. Our algorithm works for drastic changes of observation space (e.g. from vector-based observation to image-based observation), without any inter-task mapping or any prior knowledge of the target task. Empirical results show that our algorithm significantly improves the efficiency and stability of learning in the target task.