We present a novel application of reinforcement learning to grasp synthesis in a physically based virtual environment. We use pre-defined controllers and learn a policy for choosing among them. Advice from a human is incorporated to guide exploration. Our approach is straightforward and has success in learning stable grasps that generalize across objects. Furthermore, the appearance of the resulting simulated trajectories resemble natural grasping motions.