Good question. I personally feel that your vote should be out at all times barring extenuating circumstances. It forces you to take a stance, no matter how weak. Honestly, that's another thing that bugs me. Your random vote DID have a serious component to it that you are disavowing at this point. That probably contributes to the feeling of "Why are you backtracking".So, if I remove my *random* vote (which has no weight anyway, by definition), but I don't place it anywhere else, is this better or worse than if I hadn't ever placed a random vote in the first place?
As to actually answer your question, it's probably worse. Generally if you don't place a random vote in the first place, you have a meta for or are developing a meta of not doing so for theory reasons. I don't like it but I'm not likely to vote you for it. The fact you placed one and withdrew it because people were getting on you for it rubs me the wrong way. So worse.