Sorry for the very late reply (I was on a small vacation) and thanks for the follow up!
Indeed, even having the asset itself selected does produce this same problem. Once again it is due to how Unity calls OnAfterDeserialize on every frame when a “Unity Object” is selected and currently in the inspector, something that I will have to possibly hack-around to avoid.