Deep Reinforcement Learning Doesn’t Work Yet, by Alex Irpan

Source: Deep Reinforcement Learning Doesn’t Work Yet, by Alex Irpan

here are some of the failure cases of deep RL.

Deep Reinforcement Learning Can Be Horribly Sample Inefficient

If You Just Care About Final Performance, Many Problems are Better Solved by Other Methods

Reinforcement Learning Usually Requires a Reward Function

Reward Function Design is Difficult

Even Given a Good Reward, Local Optima Can Be Hard To Escape

Even When Deep RL Works, It May Just Be Overfitting to Weird Patterns In the Environment

Even Ignoring Generalization Issues, The Final Results Can be Unstable and Hard to Reproduce

The way I see it, either deep RL is still a research topic that isn’t robust enough for widespread use, or it’s usable and the people who’ve gotten it to work aren’t publicizing it. I think the former is more likely.

My feelings are best summarized by a mindset Andrew Ng mentioned in his Nuts and Bolts of Applying Deep Learning talk – a lot of short-term pessimism, balanced by even more long-term optimism.