Hmm. When you slow down the game by Time.timeScale, rigibodies will indeed frame jump since they are updated in a different framerate (ProjectSettings/Time/FixedTimeStep).
One way, is to set the interpolation mode on the Rigidbody component inspector to either “Interpolate” or “Extrapolate” instead of “None”.
Would that resolve the issue in your case?