Hmm, this part really shouldn't be that hard. I found the climbing a breeze by either control method. Even the mop in the diner was harder. I suggest whatever method you're using, either slow down or speed up. I suspect you're probably being too methodical (slow), and the game is punishing you for it. Really it's just a quick slide and flick, while saying "Wingardium Lev..." wait, no, not the last part. But the movement should be quite quick. If you go fast enough, the game doesn't realize how badly you've done it.
