I've noticed that I keep coming back to reedit my own questions because I am not satisfied with the clarity and the apparent focus of the question. "Luckily", my questions haven't drawn too many answers, but when should I give up fine-tuning my question and just open a new concise, well-phrased question?
My rule of thumb is that editing without changing the applicability of existing answers is generally fine. I think you get up to 10 edits before the post is automatically converted to CW, so just keep that limit in mind, but if you don't mind the post getting wikified (and thus losing your ability to gain rep from it), there's no reason to hard-cap your number of edits.
However, I would say that making an edit which changes the meaning of the question substantially, such that an answer which was previously valid becomes incomplete or invalid (or vice versa), is generally not appropriate. Occasionally it's necessary, but in such cases it's polite to leave a comment on any answers that are invalidated by the edit, so that their authors have the opportunity to come back and fix them up.
To be honest, I have no idea, because I've never actually encountered this problem in practice. I do know that if you edit your question enough, it immediate gets converted to Community Wiki. For an explanation of the consequences of Community Wiki, see this post on Meta Stack Overflow. In brief, it means that anyone can edit the post, and upvotes on the post do not earn or lose reputation.
My advice is to pace yourself. There are often improvements to questions required to make them Q&A appropriate, but guard against expanding the focus of the question, and let a good question be what it is.
A good question is one that has objective and useful answers. Certainly include what you've tried yourself in posing a question; that helps potential answers avoid the obvious and address your problem at a suitable level of exposition.
Sometimes good information about the proper context of a problem will come out in comments, either on the Question itself or on Answers. Folding this back into the Question statement may be beneficial to future readers, but it's the sort of thing I'd batch up and do once or twice.
Don't overlook the possible value of editing a Question's title. This can be a really important way for searchers to spot that you've asked something they want to ask about (only they shouldn't, since you already did).