This also allows for removing some of the buf_size checks and using the sample count for some of the decoding loops.