The utility function is not up for grabs
The constraints of decision theory are only there to help you win; they don't specify what constitutes a win. Rationality in and of itself cannot constrain what you want, except insofar as what you thought you wanted was different from what you actually wanted (or just plain inconsistent). Hence the saying: the utility function is not up for grabs.
- Newcomb's Problem and Regret of Rationality
- Something to Protect
- Rationality is Systematized Winning