lavf: compute probe buffer size more reliably.

      The previous code computes the offset by reversing the growth
of the allocated buffer size: it is complex and did lead to
inconsistencies when the size limit is reached.

Fix trac ticket #1991.