Ah, ok. The deserialization is actually fine. I was running with device attached to Visual Studio and it took 149 seconds to deserialize. When not attached to VS, it takes 1.8 seconds. Sorry, my bad.
(With your optimizations it takes 1.7 seconds to deserialize).