X: Representation and its consequences have been a disaster for the human race.

Y: True, but but anything before that probably couldn’t be called the human race.

Twitter

1.

I am nearly finished with a book-length treatment of the “surrogation” concept—that is, the substitution of a representation—be it a metonym, symbol, proxy, metric, or signal—for some represented whole, and the subsequent gamification of that representation. Several sections have been posted on this blog already, but I want to give an overview of how they fit into the big picture—the larger social and ecological situation which gives rise to surrogation, and its implication in fields from biology to economics to microsociology.

As soon as I began writing about surrogation, I saw it everywhere; it seemed to be a fundamental law of interaction between agents. I quickly realized that I would need to put boundaries around the concept, to prevent it from bleeding into everything else. Second, I needed to understand the basic social game which supported surrogation. It was as if I had tried to explore the dynamics of frequency-dependent selection without first having a theory of evolution.

The basic, atomic social dynamic I settled on is the selection game. I introduce it in longer form on the Pfeilstorch forum, in the post “Selection Games,” but will give a brief gloss here. The selection game is somewhere between ethology’s signaling theory, economics’ private vs. public information dichotomy, and the Goffmanian concept of an impression game. It consists of two agents, one a “selector,” the other a “selected.” The selector is making a decision or choice whose outcome is relevant to the goals of the selected party—for instance, a hiring committee to a job applicant, or a consumer to a shampoo brand. Most of the qualities which are of interest to the selector are hidden from immediate view—they either require lengthy, expensive monitoring, or are an unrepresentable intangible. Therefore, the selector must rely on public cues, proxies, and metonyms which testify in some way to these qualities. Because selection pressure is exerted on these representations, selected parties optimize toward the representation (a behavior sometimes called Goodhart’s Law).

This incentivization of representation-gaming is an inevitable outcome of surrogation. This is not a novel argument; in fact, the opposite. It has been discovered time and time again by different fields, working in isolation; what is important is bringing these discourses together, and bringing each’s unique insights to bear on the others. Worded alternatively, this is an information logistics project. In an era of information overload and hyper-specialization, what is need is curation and coherence. In my section “From measurement to degenerate play,” I compile many of these conceptual precedents, and explore the basic surrogation pipeline:

from the field of artificial intelligence, wireheading, underspecification, and nearest unblocked strategy; in philosophy, from C Thi Nguyen, the ideas of gamification and value capture; in statistics, those of overfitting, latent vs. manifest variables, proxy measures, operationalization, and heuristics; in sociology, goal displacement, legibility and Campbell’s law; in economics Goodhart’s law, the Lucas Critique, perverse incentives, and the distinction between private and public information; in information theory, joint entropy and mutual information; in metascience, Tom Griffiths’ idolatry and Feynman’s cargocult; in games studies, degenerate play; and finally, in Theory and everyday parlance, the concepts of fetish, masturbation, cobra effects, cheap play, surface compliance, spirit vs. letter, and winning by technicality.

2.

In single-layer selection games, such as that between prey and predator on the savannah, the selector always wishes to know the truth about its objects of selection: whether the red dart frog is actually poisonous or edible; whether a job applicant is in reality competent or incompetent. To the selectee, the truth is relevant only insofar as it constrains their ability to represent themselves in a way which will lead to a desirable outcome in the selection game (e.g., being selected in a dating game, or dodging selection in a military draft).

And yet—crucially—many selection games are not single-layer but many-layered and nested. Thus, in a selection game between a police officer and criminal, the police officer is not selected (e.g. for promotion or firing) based on the truth but on appearances. There is a third-party judge involved—such as the police officer’s commanding officer—whose perception of the police officer’s performance in his games with criminals, and not the reality of the officer’s performance, determines the selection outcomes of such games. In other words, the relationship between a criminal and police officer is the same as the relationship between a police officer and his superior. This relationship tiles itself up and down an institution—even at the top levels, bureaucrats answer to politicians, politicians to the public, CEOs to shareholders, shareholders to perceptions in the stock market, and so on. The more distant these relations of oversight—such as, at its most extreme, the relationship between a politician and voters—the more the necessity of relying on representations in making decisions, and the more room for surrogation.[1]

I believe this nested structure is best conceptualized through Karl Friston’s appropriation of the Markov blanket concept:

This theory holds, among other things, that boundaries are a precondition of life itself (and of complexity more generally). They are a prerequisite for maintaining homeostasis, that is, to control and regulate internal conditions which are, again, necessary to fulfilling its goals. In other words, boundaries are, first and foremost, a selection mechanism, with both a schema for admission and physical capacities for enforcing this preferential schema. They allow valuable resources—that is, goal-furthering materials, such as water, in body cells, or food supplies in a castle—to stay inside the boundaries, and assist the bounded entity in its goals. They keep undesired or harmful materials outside, either by preventing entry or expelling them. This includes other agents or sub-agents, who will attempt to improve his own lot by gaining access to the internally-hoarded resources of another bounded agent—either antagonistically, through theft or violence or deception, or cooperatively, in symbiosis.

from “The surrogation matrix“

The book, when it is finished, will begin with the building blocks of selection games, and culminate in a theory of nested Markov blankets, borrowing the concepts of base- and mesa-optimizers from artificial intelligence:

In their 2019 paper, Hubringer et al introduce the concept of mesa-optimization: a “framework that distinguishes what a system is optimized to do (its ‘purpose’), from what it optimizes for (its ‘goal’), if it optimizes for anything at all.” Mesa-optimizers are selected for by “base optimizers,” and “inner alignment” refers to an alignment between the base and mesa optimizer—for instance, natural selection is a base optimizer selecting for reproduction; organisms are subject to the base optimizations of natural selection even as they themselves may have goals which only partially align with the base optimizer’s goals. Modern non-reproductive sex is an example of a technologically-enabled uncoupling (birth control) between reward from the perspective of the basis system—natural selection—and the perspective of the mesa-optimizer—a human being.

ibid

It is this uncoupling which is crucial to understanding alignment problems—in other words, to solving coordination—be it in bureaucracies, politics, or AGI. Although selection games, and the prominent role of representations in them, are inescapable, by building an understanding of the macro-situation which governs such problems, we can begin building better and more robust selection criteria—and thereby instill these games with non-perverse incentives. The next step in my research is understanding the evolutionary cycles which govern these selection games—the feedback loops of corruption, or credentialism, which can wrest control of an institution. The Cultural Revolution in China, by bottlenecking political ability and corruption, rather than administrative skill, as criteria for survival survival, will be felt for many generations to come—since the selected-for, bottlenecked senior officials are now selectors for successors of their own.

Endnotes

[1] I’ve called these domains “opticratic,” because advancement and success are based not on real merit, but on the appearance thereof.