It's better to regret something you HAVE done (idea)

The Irish saying that deep_thought quotes above is actually from the Book of Common Prayer. It is a part of the General Confession, normally said during Morning and Evening Prayer, among other times. From the 1928 BCP;

We have left undone those things which we ought to have done;
And we have done those things which we ought not to have done;
And there is no health in us.