Difference between revisions of "Acausal trade"

From Lesswrongwiki
Jump to: navigation, search
m (See also)
 
(134 intermediate revisions by 5 users not shown)
Line 1: Line 1:
In '''acausal trade''', agents P and Q cooperate as the result of the following process: P simulates or otherwise analyzes agent Q, and so learns that Q provably does something that P wants, if P does what Q wants; and Q symmetrically learns the same about P.
+
In '''acausal trade''', two agents each benefit by predicting what the other wants and doing it, even though they might have no way of communicating or affecting each other, nor even any direct evidence that the other exists.
 +
 
 +
== Background: Superrationality and the one-shot Prisoner's Dilemma ==
 +
 
 +
This concept emerged out of the much-debated question of how to achieve cooperation on a one-shot Prisoner's Dilemma, where, by design, the two players are not allowed to communicate. On the one hand, a player who is considering the causal consequences of a decision ("Causal Decision Theory") finds that defection always produces a better result. On the other hand, if the other player symmetrically reasons this way, the result is a Defect/Defect equilibrium, which is bad for both agents. If they could somehow converge on Cooperate, they would each individually do better. The question is what variation on decision theory would allow this beneficial equilibrium.
 +
 
 +
Douglas Hofstadter (see references) coined the term "superrationality" to express this state of convergence. He illustrated it with a game in which twenty players, who do not know each other's identities, each get an offer. If exactly one player asks for the prize of a billion dollars, they get it, but if none or multiple players ask, no one gets it. Players cannot communicate, but each might reason that the others are reasoning similarly. The "correct" decision--the decision which maximizes expected utility for each player, ''if'' all players symmetrically make the same decision--is to randomize a one-in-20 chance of asking for the prize.
 +
 
 +
Gary Drescher (see references) developed the concept further, introducing an ethical system called  "acausal subjunctive morality." Drescher's approach relies on the agents being identical or at least similar, so that each agent can reasonably guess what the other will do based on facts about its own behavior, or even its own "source code." If it cooperates, it can use this correlation to infer that the other will probably also cooperate.
 +
 
 +
Acausal trade goes one step beyond this. The agents do not need to be identical, nor similar, nor have the same utility function. Moreover, they do not need to know what the other agents are like, nor even if they exist. In acausal trade, an agent may have to surmise the probability that other agents, with their utility function and proclivities, exist.
  
 
== Description ==
 
== Description ==
We have agents P and Q, possibly separated in space-time so that no interaction is possible. They might be in separate Everett Branches; or they might be outside each other's light cones, e.g., they might be very distant in an expanding universe; or Q might be a billion years in P's future on Earth. Alternatively, they might be able to communicate, as in ordinary trade.
+
We have two agents, separated so that no interaction is possible. The separation can be simply because each is not aware of the location of the other; or else each may be prevented from communicating with or affecting the other.  
 +
 
 +
In an asymmetrical example, one agent may be in the other's future.  
  
P and Q each can do something the other wants, and value that thing less than the other does. This is the usual condition for trade. However, ''acausal'' trade can happen even if  P and Q are completely separated and have no opportunity to affect each other causally. One example: An FAI wants the well-being of ''all'' humans, even those it cannot be in causal contact with, because they are in its past.
+
Other less prosaic thought experiments can be used to emphasize that interaction may be absolutely impossible. For example, agents that are outside each other's light cones, or in separate parts of an Everett multiverse. And abstracting away from those scenarios, we can talk of counterfactual "impossible possible worlds" as a model for probability distributions.
 +
 +
In truly ''acausal'' trade, the agents cannot count on reputation, retaliation, or outside enforcement to ensure cooperation. The agents cooperate because each knows that the other can somehow predict its behavior very well. (Compare Omega in [[Newcomb's problem]].) Each knows that if it defects (respectively: cooperates), the other will (probabilistically) know this, and defect (respectively: cooperate).
  
In ''acausal'' trade, P and Q cannot count on the usual enforcement mechanisms to ensure cooperation -- there is no  expectation of future interactions or an outside enforcer.
+
Acausal trade can also be described in terms of (pre)commitment: Both agents commit to cooperate, and each has reason to think that the other is also committing.
  
P and Q cooperate because each knows that the other can somehow predict its behavior very well, like Omega in [[Newcomb's problem]].
+
== Prediction mechanisms  ==
 +
For acausal trade to occur, each agent must infer there is some probability that an agent, of the sort that  will acausally trade with it, exists.
  
Each knows that if it defects (respectively: cooperates), the other will know this, and defect (respectively: cooperate), and so the best choice is to cooperate, since, as usual in trade, Cooperate/Cooperate is better for both sides than Defect/Defect.  
+
The agent might be told, exogenously (as part of the scenario), that the other exists. But more interesting is the case in which the agent surmises the probability that the other exists.
  
This can also be described as P and Q provably precommitting to cooperate, or a form of mutual reflective convergence.
+
A superintelligence might conclude that other superintelligences would tend to exist because increased intelligence [[Basic_AI_drives|is an convergent instrumental goal]] for agents. Given the existence of a superintelligence, acausal trade is one of the tricks it would tend to use.
  
== Prediction mechanisms ==
+
To take a more prosaic example, we humans realize that humans tend to be alike: Even without knowing about specific  trading partners, we know that there exist other people with similar situations, goals, desires, challenges, resource constraints, and mental architectures.
This scenario does not specify how P and Q can predict each other.
 
  
# They might have each other's source code.  
+
Once an agent realizes that another agent might exist, there are different ways that might might predict the other agent's behavior, and specifically that the other agent can be an acausal trading partner.  
# They might be able to simulate each other, or to predict the other's behavior analytically. The simulation may be approximate, and the predictions probabilistic, to avoid tractability problems, and in particular the problem of fully simulating something of the same complexity of oneself.
+
# They might know or surmise each other's  mental architectures (source code).
# In particular, P and Q might have identical or similar mental architectures, so that the each one knows that its own mental processes approximately simulate the other's. See Gary Drescher's (2006, references below) "acausal subjunctive cooperation."
+
# In particular, they might know that they have identical or similar mental architecture, so that each one knows that its own mental processes approximately simulate the other's.
# But acausal trade may be possible for two agents that know nothing at all about each other, except for this: Each knows that the other is an RPOP ([[Really powerful optimization process]], an agent with an extremely high level of intelligence). Each RPOP comes to the conclusion that since reliable acausal trade is a powerful optimization tool and so supported by the best possible optimization architecture, both it and the other RPOP ''must'' also behave according to the principles of acausal trade.
+
# They might be able to simulate each other (perhaps probabalistically), or to predict the other's behavior analytically. (Even we humans simulate each other's thoughts to guess what the other would do.)
 +
# More broadly, it is enough to know (probabilistically) that the other is a powerful optimizer, that it has a certain utility function, and that it can derive    utility from resources. Seen mathematically, this is just an optimization problem: What is the best possible algorithm for an agent's utility function? Cooperate/Cooperate is optimal under certain assumptions, for if one agent could achieve optimal utility by defecting, then, symmetrically, so could the other, resulting in Defect/Defect which generates inferior utility.
  
== An example of acausal trade with simple resource requirements ==
+
== Decision Theories  ==
At its most abstract, the agents in this model are simply optimization algorithms. Such an agent can run another as subalgorithms if that is useful in improving the optimization power of the first agent.  
+
Acausal trade is a special case of [[Updateless decision theory]] (or a variant like Functional Decision Theory, see references). Unlike better-known  variations of [[Decision theory]], such as [[Causal decision theory]], acausal trade and UDT take into account the agent's own algorithm as cause and caused.
  
Let T be an algorithm for which time is most valuable in optimizing its utility function; while for S, space is most valuable.
+
In Causal Decision Theory, the agent's algorithm (implementation) is treated as uncaused by the rest of the universe, so that though the agent's ''decision'' and subsequent action can make a difference, its internal make-up cannot (except through that decision). In contrast, in UDT, the agents' own algorithms are treated as causal nodes, influenced by other factors, such as the logical requirement of optimality in a utility-function maximizer. In UDT, as in acausal trade, the agent cannot escape the fact that its decision to defect or cooperate constitutes strong Bayesian evidence as to what the other agent will do, and so it is better off cooperating.
  
T and S must both deal with subjective uncertainty as to whether they may be instantiated in a universe where time is in abundance (a universe which will not end for a long time), or a space-rich universe (an extremely large universe). This uncertainty is sometimes expressed in terms of multiverse theory, but only requires that we specify subjective uncertainty, i.e., that the algorithms should be able to optimize in either kind of universe, even if the designer does not know in advance which universe the algorithm will be in.
+
== Limitations and Objections  ==
 +
Acausal trade only works if the agents are smart enough to predict each other's behavior, and then smart enough to acausally trade. If one agent is stupid enough to defect, and the second is smart enough to predict the first, then neither will cooperate.  
  
T can prove about S that it will "do a favor" for T, i.e., that S will run T as a subalgorithm, if S finds itself instantiated in a  time-rich universe. S can prove that T will "reciprocate" -- i.e., T will run S if it finds itself in a space-rich universe, in acausal trade for S's proven (possibly probabilistically proven) commitment.  
+
Also, as in regular trade, acausal trade only works if the two sides are close enough in power that the weaker side can do something worthwhile enough for the stronger.
  
T and S also estimate that the other is likely to exist. This is because super-powerful intelligences (optimizers) are an attractor in any universe that has a general intelligence at all, since any general intelligence [[Intelligence explosion|will want to self-improve]] in order to better optimize towards its goals. Given that superintelligences exist in other possible universes, T and S will realize that one efficient technique for a super-optimizer is acausal trade, thus implying the existence of the superintelligent acausal trading partner.
+
A common objection to this idea: Why shouldn't an agent "cheat" and choose to defect? Can't it "at the last moment" back out after the other agent has committed? However, this approach takes into account only the direct effect of the  decision, while a sufficiently intelligent trading partner could predict the agent's choice, including that one, rendering the "cheating" approach suboptimal.
  
Algorithms T and S are so structured because, as we have described, acausal trade is a useful technique in certain conditions for optimizing towards each agent's own  goals: Acausal trade is one of the tricks a good optimizer can use to optimize its utility function.
+
Another objection: Can an agent care about (have a utility function that takes into account) entities with which it can never interact, and about whose existence it is not certain? However, this is quite common even for humans today. We care about the suffering of other people in faraway lands about whom we know next to nothing. We are even disturbed by the suffering of long-dead historical people, and wish that, counterfactually, the suffering had not happened. We even care about entities that we are not sure exist. For example: We might be concerned by news report that a valuable archaeological artifact was destroyed in a distant country, yet at the same time read other news reports stating that the entire story is a fabrication and the artifact never existed. People even get emotionally attached to the fate of a fictional character.
  
== Acausal trade with complex resource requirements ==
+
== An example of acausal trade with simple resource requirements ==
In the example above, resource requirements are very simple: time and space. It may be that in the limit of increasing intelligence, time and space are the only relevant resources. But for an agent G with complex and arbitrary goals, like humans, this type of calculation will not allow H, a possible trading partner for G, to consider the possibility of acausal trade to be likely enough that the trading partner H would to do any acausal favors directly for the agent G.  
+
At its most abstract, the agents are simply optimization algorithms. As a toy example, let T be a utility function for which time is most valuable as a resource; while for utility function S, space is most valuable, and assume that these are the only two resources.  
  
However, in a multiverse, all agents and their hypothetical trading partners exist in many near-identical instances. If an agent H decides to randomly sample a tiny fraction of all possible types of acausal trading partner G, and the almost-identical counterparts of H elsewhere in the multiverse do likewise, then in aggregate the near-copies of H will sample all of the major types of acausal trading partners G, in rough proportion to their frequency, if the models are good. Then all these near-identical instances of the agent H produce stuff that the sampled trading partners G would want to exist, if the potential partners G are running a symmetric algorithm that would produce stuff that the agents H want (if they sampled them in return). The computational effort is shared out across the ensemble of H, but this still makes the goods which  are given and received come into rough balance, as long as one samples representatively.
+
We will now choose the best algorithms for optimizing T. To avoid anthropomorphizing, we simply ask which algorithm--which string of LISP, for example--would give the highest expected utility for a given utility function. Thus, the choice of source code is "timeless": We treat it as an optimization problem across all possible strings of LISP. We assume that computing power is unlimited. Mathematically, we are asking about argmax T.
  
Thus, each agent H is producing stuff only for a tiny fraction of potential trading partners G, and cannot expect to trade one-on-one with any given instance of the trading partner G. But because there are a huge number of near-copies of that agent H, each instance of trading partner H has a good chance that one of the ensemble H will do what G wants and vice versa.
+
We specify that there is a probability that either agent will be run in an environment where time is in abundance, and if not, some probability that it will be run in a space-rich universe.
  
== Related Concept: Super-rationality ==
+
If the algorithm for T is instantiated in a space-rich environment, it will only be able to gain a small amount of utility for itself, but S would be able to gain a lot of utility; and vice versa.
  
The question of why to cooperate on a one-shot Prisoner's Dilemma has long been debated. On the one hand, defection always produces a better individual result that cooperation. On the other hand, this leads to an equilibrium of defect-defect, which is bad for both agents. If they can somehow converge on mutual cooperation, they will do better than the defect-defect equilibrium
+
The question is: What algorithm for T provides the most optimization power, the highest expected value of utility function T?
  
Douglas Hofstadter (see references) coined the term "super-rationality" to express this state of convergence. Gary Drescher (see references) developed the concept further, introducing the term "acausal subjunctive morality" for an ethical system of behavior based on this mechanism. Eliezer Yudkowsky has discussed this one-shot Prisoner's Dilemma at [http://lesswrong.com/lw/tn/the_true_prisoners_dilemma/ The True Prisoner's Dilemma].
+
If it turns out that the environment is space-rich, the agent for T may run the agent (the algorithm) for S, increasing the utility for S, and symmetrically the reverse. This will happen if each concludes, that the optimum occurs when the other agent has the "trading" feature. Given that this is the optimal case, the acausal trade will occur.
  
Acausal trade takes this form of cooperation several steps beyond Hofstadter's and Drescher's mechanisms: Agent R does not simply cooperate with agent S, doing what S wants and knowing that S will do what R wants. Rather, agent R  may also have to surmise the potential ''existence'' of S; and also surmise ''what'' S would want R to do, and what S will do for R in (acausal) exchange.
+
== Acausal trade with complex resource requirements ==
 +
In the  toy example above, resource requirements are very simple. In general, given that agents can have complex and arbitrary goals requiring a complex mix of resources, an agent might not be able to conclude that a specific trading partner has a meaningful chance of existing and trading.
  
== Relevance to Friendly AI ==
+
However, an agent can analyze the distribution of probabilities for the existence of other agents, and weight its actions accordingly. It will do acausal "favors" for one or more trading partners, weighting its effort according to its subjective probability that the trading partner exists. The expectation on utility given and received will come into a good enough balance to benefit the traders, in the limiting case of increasing super-intelligence.
Acausal trade may be a way to get the cooperation of a future AI. If we know that the AI would want us to behave a certain way, and we can prove that it will do good things for us, once it arises, if we will have done do what it wants before it comes into existence, and that it can prove the symmetrical statement -- that we do what it  wants, if we've figured out and proven that it may have this behavior -- then we can trade with it, even though it does not yet exist.  
 
  
This approach rests on being able to prove certain facts about human behavior; or to put it another way, for humans to be able to provably commit to behavior.
+
== Ordinary trade ==
 +
 +
Even ordinary trade can be analyzed acausally, using a perspective similar to that of [[Updateless decision theory]]. We ask: Which algorithm should an agent have to get the best expected value, summing across all possible environments weighted by their probability? The possible environments include those in which threats and promises have been made.
  
 
==See also==
 
==See also==
* "AI Beliefs" blog: [http://aibeliefs.blogspot.com/2007/11/non-technical-introduction-to-ai.html "AI deterrence"]
+
* [http://aibeliefs.blogspot.com/2007/11/non-technical-introduction-to-ai.html?a=1 "AI deterrence"]
* Stuart Armstrong, [http://lesswrong.com/lw/1pz/the_ai_in_a_box_boxes_you/ "The AI in a box boxes you"]
+
* [http://lesswrong.com/lw/1pz/the_ai_in_a_box_boxes_you "The AI in a box boxes you"]
* Alexander Kruel, [http://kruel.co/2012/07/27/the-acausal-trade-argument/ The Acausal Trade Argument].
+
* [https://slatestarcodex.com/2017/03/21/repost-the-demiurges-older-brother/ A story] that shows acausal trade in action.
* Jaan Tallinn, [http://fora.tv/2012/10/14/Jaan_Tallinn_Why_Now_A_Quest_in_Metaphysics Why Now? A Quest in Metaphysics]. Singularity Summit 2012 talk.
+
* [http://slatestarcodex.com/2018/04/01/the-hour-i-first-believed/ Scott Alexander] explains  Acausal Trade. (Most of that article is tongue-in-cheek, however.)
 +
* "[http://www.nickbostrom.com/papers/porosity.pdf Hail Mary, Value Porosity, and Utility Diversification]," Nick Bostrom, the first paper from academia to rely on the concept of acausal trade.
 +
* [http://intelligence.org/files/TowardIdealizedDecisionTheory.pdf Towards an idealized decision theory], by Nate Soares and Benja Fallenstein discusses acausal interaction scenarios that shed light on new directions in decision theory.
 +
* [https://ie.technion.ac.il/~moshet/progeqnote4.pdf Program Equilibrium], by Moshe Tennenholtz. In: Games and Economic Behavior.
 +
* [https://arxiv.org/abs/1401.5577 Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic], by Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire and Eliezer Yudkowsky
 +
* [https://arxiv.org/abs/1602.04184 Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents], by Andrew Critch
 +
* [https://link.springer.com/article/10.1007/s11238-018-9679-3 Robust Program Equilibrium], by Caspar Oesterheld. In: Theory and Decision.
 +
* [https://foundational-research.org/multiverse-wide-cooperation-via-correlated-decision-making/ Multiverse-wide Cooperation via Correlated Decision Making], by Caspar Oesterheld
  
 
== References ==
 
== References ==
*  [http://www.gwern.net/docs/1985-hofstadter Douglas Hofstadter's Superrationality essays, published in ''Metamagical Themas''] ([http://lesswrong.com/lw/bxi/hofstadters_superrationality/ LW discussion])
+
*  [http://www.gwern.net/docs/1985-hofstadter Hofstadter's Superrationality essays, published in ''Metamagical Themas''] ([http://lesswrong.com/lw/bxi/hofstadters_superrationality/ LW discussion])
* [[Gary Drescher]], ''Good and Real'', MIT Press, 2006.
+
* Jaan Tallinn, [http://fora.tv/2012/10/14/Jaan_Tallinn_Why_Now_A_Quest_in_Metaphysics Why Now? A Quest in Metaphysics].
 +
* [[Gary Drescher]], ''Good and Real'', MIT Press, 1996.
 +
* [https://arxiv.org/abs/1710.05060|Functional Decision Theory], an updated Updateless Decision Theory
 +
[[Category:Concepts]]

Latest revision as of 07:55, 7 January 2020

In acausal trade, two agents each benefit by predicting what the other wants and doing it, even though they might have no way of communicating or affecting each other, nor even any direct evidence that the other exists.

Background: Superrationality and the one-shot Prisoner's Dilemma

This concept emerged out of the much-debated question of how to achieve cooperation on a one-shot Prisoner's Dilemma, where, by design, the two players are not allowed to communicate. On the one hand, a player who is considering the causal consequences of a decision ("Causal Decision Theory") finds that defection always produces a better result. On the other hand, if the other player symmetrically reasons this way, the result is a Defect/Defect equilibrium, which is bad for both agents. If they could somehow converge on Cooperate, they would each individually do better. The question is what variation on decision theory would allow this beneficial equilibrium.

Douglas Hofstadter (see references) coined the term "superrationality" to express this state of convergence. He illustrated it with a game in which twenty players, who do not know each other's identities, each get an offer. If exactly one player asks for the prize of a billion dollars, they get it, but if none or multiple players ask, no one gets it. Players cannot communicate, but each might reason that the others are reasoning similarly. The "correct" decision--the decision which maximizes expected utility for each player, if all players symmetrically make the same decision--is to randomize a one-in-20 chance of asking for the prize.

Gary Drescher (see references) developed the concept further, introducing an ethical system called "acausal subjunctive morality." Drescher's approach relies on the agents being identical or at least similar, so that each agent can reasonably guess what the other will do based on facts about its own behavior, or even its own "source code." If it cooperates, it can use this correlation to infer that the other will probably also cooperate.

Acausal trade goes one step beyond this. The agents do not need to be identical, nor similar, nor have the same utility function. Moreover, they do not need to know what the other agents are like, nor even if they exist. In acausal trade, an agent may have to surmise the probability that other agents, with their utility function and proclivities, exist.

Description

We have two agents, separated so that no interaction is possible. The separation can be simply because each is not aware of the location of the other; or else each may be prevented from communicating with or affecting the other.

In an asymmetrical example, one agent may be in the other's future.

Other less prosaic thought experiments can be used to emphasize that interaction may be absolutely impossible. For example, agents that are outside each other's light cones, or in separate parts of an Everett multiverse. And abstracting away from those scenarios, we can talk of counterfactual "impossible possible worlds" as a model for probability distributions.

In truly acausal trade, the agents cannot count on reputation, retaliation, or outside enforcement to ensure cooperation. The agents cooperate because each knows that the other can somehow predict its behavior very well. (Compare Omega in Newcomb's problem.) Each knows that if it defects (respectively: cooperates), the other will (probabilistically) know this, and defect (respectively: cooperate).

Acausal trade can also be described in terms of (pre)commitment: Both agents commit to cooperate, and each has reason to think that the other is also committing.

Prediction mechanisms

For acausal trade to occur, each agent must infer there is some probability that an agent, of the sort that will acausally trade with it, exists.

The agent might be told, exogenously (as part of the scenario), that the other exists. But more interesting is the case in which the agent surmises the probability that the other exists.

A superintelligence might conclude that other superintelligences would tend to exist because increased intelligence is an convergent instrumental goal for agents. Given the existence of a superintelligence, acausal trade is one of the tricks it would tend to use.

To take a more prosaic example, we humans realize that humans tend to be alike: Even without knowing about specific trading partners, we know that there exist other people with similar situations, goals, desires, challenges, resource constraints, and mental architectures.

Once an agent realizes that another agent might exist, there are different ways that might might predict the other agent's behavior, and specifically that the other agent can be an acausal trading partner.

  1. They might know or surmise each other's mental architectures (source code).
  2. In particular, they might know that they have identical or similar mental architecture, so that each one knows that its own mental processes approximately simulate the other's.
  3. They might be able to simulate each other (perhaps probabalistically), or to predict the other's behavior analytically. (Even we humans simulate each other's thoughts to guess what the other would do.)
  4. More broadly, it is enough to know (probabilistically) that the other is a powerful optimizer, that it has a certain utility function, and that it can derive utility from resources. Seen mathematically, this is just an optimization problem: What is the best possible algorithm for an agent's utility function? Cooperate/Cooperate is optimal under certain assumptions, for if one agent could achieve optimal utility by defecting, then, symmetrically, so could the other, resulting in Defect/Defect which generates inferior utility.

Decision Theories

Acausal trade is a special case of Updateless decision theory (or a variant like Functional Decision Theory, see references). Unlike better-known variations of Decision theory, such as Causal decision theory, acausal trade and UDT take into account the agent's own algorithm as cause and caused.

In Causal Decision Theory, the agent's algorithm (implementation) is treated as uncaused by the rest of the universe, so that though the agent's decision and subsequent action can make a difference, its internal make-up cannot (except through that decision). In contrast, in UDT, the agents' own algorithms are treated as causal nodes, influenced by other factors, such as the logical requirement of optimality in a utility-function maximizer. In UDT, as in acausal trade, the agent cannot escape the fact that its decision to defect or cooperate constitutes strong Bayesian evidence as to what the other agent will do, and so it is better off cooperating.

Limitations and Objections

Acausal trade only works if the agents are smart enough to predict each other's behavior, and then smart enough to acausally trade. If one agent is stupid enough to defect, and the second is smart enough to predict the first, then neither will cooperate.

Also, as in regular trade, acausal trade only works if the two sides are close enough in power that the weaker side can do something worthwhile enough for the stronger.

A common objection to this idea: Why shouldn't an agent "cheat" and choose to defect? Can't it "at the last moment" back out after the other agent has committed? However, this approach takes into account only the direct effect of the decision, while a sufficiently intelligent trading partner could predict the agent's choice, including that one, rendering the "cheating" approach suboptimal.

Another objection: Can an agent care about (have a utility function that takes into account) entities with which it can never interact, and about whose existence it is not certain? However, this is quite common even for humans today. We care about the suffering of other people in faraway lands about whom we know next to nothing. We are even disturbed by the suffering of long-dead historical people, and wish that, counterfactually, the suffering had not happened. We even care about entities that we are not sure exist. For example: We might be concerned by news report that a valuable archaeological artifact was destroyed in a distant country, yet at the same time read other news reports stating that the entire story is a fabrication and the artifact never existed. People even get emotionally attached to the fate of a fictional character.

An example of acausal trade with simple resource requirements

At its most abstract, the agents are simply optimization algorithms. As a toy example, let T be a utility function for which time is most valuable as a resource; while for utility function S, space is most valuable, and assume that these are the only two resources.

We will now choose the best algorithms for optimizing T. To avoid anthropomorphizing, we simply ask which algorithm--which string of LISP, for example--would give the highest expected utility for a given utility function. Thus, the choice of source code is "timeless": We treat it as an optimization problem across all possible strings of LISP. We assume that computing power is unlimited. Mathematically, we are asking about argmax T.

We specify that there is a probability that either agent will be run in an environment where time is in abundance, and if not, some probability that it will be run in a space-rich universe.

If the algorithm for T is instantiated in a space-rich environment, it will only be able to gain a small amount of utility for itself, but S would be able to gain a lot of utility; and vice versa.

The question is: What algorithm for T provides the most optimization power, the highest expected value of utility function T?

If it turns out that the environment is space-rich, the agent for T may run the agent (the algorithm) for S, increasing the utility for S, and symmetrically the reverse. This will happen if each concludes, that the optimum occurs when the other agent has the "trading" feature. Given that this is the optimal case, the acausal trade will occur.

Acausal trade with complex resource requirements

In the toy example above, resource requirements are very simple. In general, given that agents can have complex and arbitrary goals requiring a complex mix of resources, an agent might not be able to conclude that a specific trading partner has a meaningful chance of existing and trading.

However, an agent can analyze the distribution of probabilities for the existence of other agents, and weight its actions accordingly. It will do acausal "favors" for one or more trading partners, weighting its effort according to its subjective probability that the trading partner exists. The expectation on utility given and received will come into a good enough balance to benefit the traders, in the limiting case of increasing super-intelligence.

Ordinary trade

Even ordinary trade can be analyzed acausally, using a perspective similar to that of Updateless decision theory. We ask: Which algorithm should an agent have to get the best expected value, summing across all possible environments weighted by their probability? The possible environments include those in which threats and promises have been made.

See also

References