Machine Learning with Phil posted this tutorial to apply experience replay to the actor critic algorithm.

It seems smart, but it turns out that it doesn’t work.

Despite the fact that the replay memory is critical to the success of the deep Q learning algorithm, it completely breaks the actor critic network.