Why Are My Q-Learning Values Exploding? A Tale of Inflated Rewards and Floating Point Limits.

Q-Learning Values Exceeding Threshold

In an attempt to implement Q-Learning, an issue arose where the state-action values exceeded the limits of a double precision floating point variable. The initial implementation attributed this problem to the use of agent.prevState instead of a state-action tuple. However, the root cause was identified as the calculation of prevScore.

Understanding the Issue

Q-Learning updates the value of Q(s, a) based on the formula:

Q(s, a) = Q(s, a) + (LearningRate * (prevScore + (DiscountFactor * reward) - Q(s, a)))

The crucial aspect is that prevScore represents the reward for the previous state-action, not the Q-value. In the initial implementation, prevScore contained the Q-value of the previous step instead of the reward itself, resulting in inflated values that exceeded the floating point limit.

Resolution

By revising prevScore to hold the true reward for the previous step, the learning process behaved as intended. The maximum value after 2M episodes reduced significantly, and the model exhibited reasonable behavior during gameplay.

The Role of Reward

It's important to note the influence of the reward function in reinforcement learning. The goal is to maximize expected total reward. If a reward is given for every time step, the algorithm will favor prolonging the game, leading to excessively high Q-values. In this example, introducing a negative reward for each time step encouraged the agent to aim for victory, bringing the Q-values within appropriate bounds.

The above is the detailed content of Why Are My Q-Learning Values Exploding? A Tale of Inflated Rewards and Floating Point Limits.. For more information, please follow other related articles on the PHP Chinese website!