I sometimes forget that not everyone realizes how poorly understood open-source game theory is, until I end up sharing this example and remember how weird it is for folks to see for the first time. Since that’s been happening a lot this week, I wrote this post to automate the process.
Consider a game where agents can view each other’s source codes and return either “C” (cooperate) or “D” (defect). The payoffs don’t really matter for the following discussion.
First, consider a very simple agent called “CooperateBot”, or “CB” for short, which cooperates with every possible opponent:
def CB(opp): return C
(Here “opp” is the argument representing the opponent’s source code, which CooperateBot happens to ignore.)
Next consider a more interesting agent, “FairBot”, or “FB” for short, which takes in a single parameter $k$ to determine how long it thinks about its opponent:
def FB_k(opp): search all strings of length ≤ k ...for a proof that ..."opp(FB_k) = C" if a proof is found: return C else: return D
I claim this agent is interesting.
Warm-up question: What’s $FB_k(CB)$?
Real question: what’s $FB_k(FB_k)$, for very large $k$?
Got it yet? Here’s hint #2:
Okay, that’s enough hints. What does this all mean for the value of $FB_k(FB_k)$? Here’s the answer:
I hope you enjoyed that, and/or the paper linked in the answer. But don’t read the paper before you’ve thought about it yourself first! To me, the main importance of results like this are to show that open-source game theory can be very counterintuitive, and we could really use a lot more research to understand how it works before we start building a lot of code-based agents that are smart enough to read and write code themselves.