Unfortunately rendering to MP4 is a bit unstable due to the library used and the MP4 format will probably be removed in a future version due to this reason. May I recommend to please render to PNG or EXR instead, since those two are far more stable and provide a better consistent quality?
Thanks.