That notion and many of its permutations caused me to question. Since then I’ve never fully accepted that premise, but neither have I been able to wholly accept the premise that desire leads to fullfilment.
Below is an excerpt from a talk by Charles Harper given at the Singularity Summit:
The Transformation of Desire
People use power to pursue ends they desire. Therefore the increase of personal power calls for the transformation of personal desire. Science, however, knows next to nothing of the transformation of desire. Monks, hermits, fasters, counterculturals – the athletes of the spirit; these are the sorts of people who work on and know about the transformation of desire. A wise approach towards the development of superintelligence probably should include serious consideration on how to transform desire so that enhanced powers are not abused to serve un-enhanced desires. The transformation of desire for humans involves what in virtue ethics is called “habitus” – the formation of habituated character through devoted, willful practice within a space of real freedom. Virtue is not a matter of either knowledge or “programming.” And it also often is not limited to only individual lives. It occurs in group contexts such as families, teams, monastic orders, communities. Also, people who engage in the transformation of desire often are involved in worship and prayer. They seek inspiration and transformative power from God. In view of such issues, what would be the “transformation of desire” for a superintelligence?