Several protection algorithms strive to reduce the number of vehicles trapped in the dilemma zone. These algorithms use some arbitrary policies such as terminating the green when only one vehicle is present in the dilemma zone and the dilemma zone has not cleared after a certain period of time. The research proposes a control agent that is able to develop and adapt an optimal policy by learning from the environment. The agent incorporates a Markovian traffic state estimation into its learning process. A novel approach is presented for controlling traffic signals so that the number of vehicles trapped in the dilemma zone is reduced in an optimal fashion according to changes in traffic states. A comparison between the proposed optimal policy and the emerging detection-control system two-stage policy was conducted, and it was found that the policy based on reinforcement learning reduced the number of vehicles caught in the dilemma zone by up to 32%.