There appears to be unspoken agreement in evolutionary psychology that when people experience unpleasant states, which is seen as the exception to the rule, the existence of these states must be explained in terms of inclusive fitness. Physical pain, for example, is described as a mechanism to alert the mind to something that needs attention, like an open wound.
In the same spirit one may ask: Why regret?
There also appears to be widespread agreement among psychologists, evolutionary or otherwise, that people see themselves in too positive a light. The standard evopsych explanation for this goes like as follows: Presenting yourself in an unrealistically positve light increases inclusive fitness; so does seeing others realistically. This creates an arms race type of situation in which the ability to detect dishonesty about themselves in others increases together with abilities to convincingly present oneself dishonestly. A great way of achieving the latter aim is to be deluded about your own qualities: Just as a polygraph has problems detecting it when people tell falsehoods they believe to be true, so does the human lie detector. (Robin Hanson adds depth.) The rest of this post assumes this explanation is valid.
Sometimes people wake up in the morning and regret what they did the night before because the vomit on the carpet really stinks. Sometimes people wake up in the morning and regret what they did the night before because one does not fuck one's best friend's girlfriend. Cleaning up the vomit is unpleasant, yet sooner or later people will do just that because it beats bearing the cost of leaving the vomit on the carpet. Regretting fucking your best friend's girlfriend, and then regretting it some more, is unpleasant, yet if you do enough regretting, you can interpret this as evidence that you're not the type of person who fucks his best friend's girlfriend despite the evidence showing pretty conclusively that you are. Hence, regretting is the price you pay for maintaining a positive image of yourself.
Prediction: Holding damage done constant, the more opportunities a wrongdoer is given for defining the behaviour in question as untypical, the less regret is going to ensue.
Let's add two assumptions: (1) People act as though they were anticipating regret (whether or not they consciously are) and adjust their behaviour in an approximately utility-maximizing manner; (2) the more untypical the situation is defined as being, the easier it is to define the behaviour in the situation as untypical.
Another prediction: The easier it is to define a situation as untypical on the basis of its qualities, the more wrongdoing is going to ensue.
And another: Holding the features of the situation constant, a person is more likely to define it as untypical the more likely it is s/he's going to do something not in line with his/her self-image (an illustration).
While I'm at it: The more likely it is that a person will do something not in line with his/her self-image, the more likely it is that s/he will change the objective features of the situation so that it gets easier to define it as untypical.
That's the cynical view, and there is anecdotal evidence to support it. Yet there is also evidence in favour of the optimistic view that regret motivates you to change your behaviour, just as your memory of cleaning up the vomit may motivate you to cut down on your drinking. When will regret merely serve self-protection, and when will it lead to a change in behaviour? I'm afraid that in response to this, I only have a triviality to offer: Depending on the interplay between personality and the evidence at hand, there will come a point at which the illusion that regret itself is sufficient to show that the self-image may be maintained becomes unsustainable. This insight, the sensation of which is rather unpleasant, will motivate behaviour change.
In case the beginning of the post has created expectations of an answer in terms of inclusive fitness, I'll say this: There comes a point at which the benefits of keeping a positive self-image without behaviour modification are outweighed by its costs.
Well, at least it's an answer.
P.S.: If your reaction to this post is, "Oh, he's talking about cognive dissonance", you may want to read this short article (low-quality pdf).
P.P.S.: Speaking of psychologists - yeah, I, too, would guess that psychologists have written about this. But if I posted only after searching world history's writings for something that may be of relevance to a post of mine, I might as well stop blogging altogether. Specific references welcome, of course.