h264: move resolution change check further up.

      This prevents some variables from being changed in case of a
rejected resolution change.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>