avutil/mem: Fix flipped condition

      Fixes return code and later null pointer dereference

Found-by: Laurent Butti <laurentb@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>