x-post from Carcinisation

In a recent paper, my collaborator Tom Rutten and I advanced a tentative theory of how contemporary visual artworks might interact with a predictive error minimization (or “predictive processing“) system in human viewers. The predictive processing model of cognition is a relatively recent figuration of the age-old problem of inference (how humans make predictions from patterns and pull patterns from data), originating in the work of computational neuroscientists like Friston, Rao, and Ballard in the 1990s but prefigured by Jeurgen Schmidhuber, whose theory of cognitive “compression” has been covered previously on this site and its neighbors.

I haven’t yet tried summarizing the paper’s ideas in an informal way, or arguing (beyond Twitter) for its usefulness as a theory. Here, I advance that argument both modestly and boldly.

As a brief overview, predictive error minimization is the hypothesis that our brains make hierarchical, top-down predictions about the sensory experiences it encounters; bottom-up information is propagated up the hierarchy as error signals, correcting and updating the top-down high-level hypotheses. It’s compelling in part because it explains so many otherwise-wooey research findings about how expectations and beliefs (themselves formed through previous experiences) actively shape our running experience of reality, from the effect of price on perceived quality to the distortions of post-traumatic stress disorder.

The modest version of our argument is this: If we take the predictive processing model as our most-accurate-yet description of how cognition works, we gain insight into how a piece of art or literary text (for example) interacts with our sensory and analytical systems. We gain a vocabulary for talking about what a “schema,” or cognitive knowledge structure, is, and what it means for that structure to learn from surprising events and prediction errors, and how expectations and beliefs at the structure’s highest level can radically distort how sensory experiences are processed at its lowest. We can begin talking about how different types, or periods, of art might enact themselves on, or manipulate, the schema differently, and how divergences between individuals’ schemas (their “calibrations”) can lead them to receive an artwork divergently in turn. While the basics of the art-mind interaction have already been sketched out in various aesthetic and hermeneutic theories, predictive hierarchical cognition 1) fills in missing theoretical gaps and 2) acts as secondary evidence for some of these theories while rejecting others. The paper’s further suggestion is that a key feature of 20th and 21st century artworks lies in their intent to the subvert their viewers’ schemas: they aim to propagate sensory and conceptual errors up the hierarchy, revising viewers’ high level priors about art or the world. This is not dissimilar from the hypothesized function of jokes by Dennet & Hurley et al.

To present our argument boldly: The steelman of the modernist and avant-garde project is basically “Art = a technology that activates, subverts, and bares our predictive schema” (i.e. the perceptual filter through which we understand the world). Unfortunately, right now the public factoring of that project is still relatively muddled and confused; there’s a real sense from many artists that they aren’t sure what they’re “supposed to be” doing, and from critics that the justificatory ground beneath their taste is perpetually receding. “Schematic subversion”—the subversion of viewers’ (cognitive) structure of expectations—offers both a coherent description of the gestalt 20th century art project as well as point to a path for honing art within that tradition—as a technology or instrument toward a coherent end with demonstrable human benefits.

We argue first that literary and artistic encounters can, and ought to be, conceptualized as designed temporal experiences which continually update, anticipate, and act either synchronously with or against the reader/viewer’s structure of expectations. Over the course of reading a book, our impression of a character, or his circumstance, or the direction of the plot (for instance) is constantly mutating as we take in new information. In literary theory, this is known as the hermeneutic circle; our understanding of a part informs our understanding of the whole which in turn informs our understanding of a part:

As a toy demonstration of how this temporal process of updating and revision works, we can use a garden-path sentence like “The old man the boats.” On first parse, one assumes “old” is an adjective modifying the noun “man”; when that interpretation “breaks” on us, we switch to parsing “old” as a plural noun and “man” as in its verb form. In our first parse, the high probabilistic likelihood of “the old man” signifying an elderly male may have made our initial interpretation so strong that the subsequent “the boats” cannot immediately amend it, and a second or third reading is required to verify that the dramatically unlikely minority sense-meaning is, in fact, the one intended.

In the case of post-1900 work, actively subverting audience expectation (somewhat like a garden-path sentence) replaces synchronicity as a dominant aim. This new, post-1900 kind of artwork should be understood as a technology analogous to psychedelics (see Friston’s work with Carhart-Harris on REBUS): designed to massage users’ hierarchical predictive structures, either to interrogate deep-rooted assumptions or else nudge them into a new “potentialities,” new perceptions or orientations toward reality. We divide the subverted expectations into two types: those set up by the outside world (e.g. the expectations one walks into an art encounter with, based on genre, or artist reputation, etc), and those created by the work itself (e.g. casting a character as initially malevolent, who is eventually revealed to be of strong moral character).

These new, post-1900 works are machines which create for participants temporal experiences characterized by a morphing set of expectations, beliefs, and predictions. The negotiation between participant and artwork generates the experience of “interestingness”—an enjoyable, dynamic interplay. The work “is not the work”; instead, processing and interpreting the work is what produces “the work.” This active interpretive process replaces traditional plot as the work’s underlying temporal logic & dynamic—the almond milk of narrative.

*

The conceptual carving we call “schema-subverting art” in our paper tries to be meaningful by describing a fundamental, intrinsic value of the 20th century art project which is distinct from previous age’s aesthetic value hierarchies. It therefore inevitably doubles as a theory of the 20th century’s art project, a project which must be (1) coherent across many schools & movements and yet (2) distinct from the pre-modern paradigm. “Schema-revising” or “schema-subverting” art drives a through-line from the likes of, at the 20th century’s start, Cubist painting, rip-it-up Dada, and modernism’s relentless “make it new”—through to the happenings, psychedelia, and conceptual anti-art of the century’s center—to finally, the justice-oriented and activist work of contemporary avant production. All of these periods—though distinct in their attitudes toward materiality, politics, technique, and representation—are united in their preoccupation with perception, disruption, and disrupting perception: in a phrase, schematic interrogation.

*

Artistic production and theory in the 20th century shifts its emphasis from the material to the experiental. Art begins to be understood, or conceptualized, as an emergent product of the interaction between participant and stimulus. (Soon after, in a reflection of the zeitgeist, structuralist hermeneutics begins moving into theories of “reader response”: that the meaning of a text is an emergent property of its interaction with the interpretive schema of its reader, instead of an intrinsic property of the text itself.)

John Dewey is an unsung figure in this sea change in visual art. Allan Kaprow, who started the participatory, multi-media Happenings movement in the 1960s treated Dewey’s Art As Experience like a Bible. Dewey was friends with Matisse and MoMA director Alfred Barr; his thinking touched John Cage and Fluxus, Rob Rauschenberg, Claes Oldenburg, Ray Johnson of Correspondence Art, and the Black Mountain College curriculum. Dewey’s ideas “reoriented” Bauhaus; Josef Albers befriends him and suddenly starts speaking like him: “Through  some kind of art experience… the student can come into realization of order in the world.” Dewey’s foil in the era is the (today more widely read) Clement Greenberg: a cult of flatness and an obsession with material. One philosophy wins. 

*

So art is an experience. It has a temporal dimension. It emerges from participant—reader, viewer, listener—meeting work-as-crafted-stimulus. From predictive processing it follows that it emerges from the interaction between work and the participant’s predictive schema. This is a thing art does, can do. When you boil away differences in the specific ways 20th century movements and schools have talked about such things—“make it new,” Russian defamiliarization, art that challenges us, art that changes how we see, art that lets us imagine alternate possibilities—you’re left with this shared phenomenon of subversion and revision, and there’s a reasonable way we can imagine that this art really does flash us with self-awareness of our perceptual structures (what we call an “effect idea” in the paper), and really can update our understanding of the world, possibly in extra-linguistic, underlying ways less accessible to non-fiction. This is the ideal: to find a niche, or purpose, which no other technology can comparably achieve. Visual art’s crisis, in part, comes from being outcompeted (by film, recorded music, and even the printing press) on many fronts it once excelled at.

*

Lastly, some terms from aesthetic and literary theory that make more sense in the light of predictive processing:

• Resonance: As Dominic Fox has pointed out over at Poetix, resonance is a kind of “fit” or accord between the worldview tacitly contained in an artwork, and the worldview of the engaging participant.

In an art encounter, our mind updates its inferential models about the world with respect to the work’s perceived accuracy, an assessment made by the model itself. The observer’s schema acts as a “check” or arbitrator on its own incorporation of the artworks’ worldview (dynamics and concepts); when the cartography of the work is too implausible in the eyes of an apprehending schema, it may be dismissed entirely. This is to say that concepts learned directly from personal experience, and indirectly from outside sources, are used to assess the likelihood of a work’s worldview as conveyed through its components. The artwork’s “truthiness” as estimated by the viewer can be understood as its precision, or reliability (Clark 2016). In this way the schema can be understood as a gatekeeper to its own revision: only stimulus surpassing some level of intelligibility and precision for a viewer will be able to interact with the viewer’s schema and incite revision. One consequence is that information which fits closely with an existing schema but poorly with a ground-truth reality is perceived by that schema as more, rather than less, likely to be the case.

• Ambiguity: A situation in which there is no dominant, top-down hypothesis about what is being perceived or communicated. All communications and artworks have ambiguity, but certain works or modes of communication use that ambiguity intentionally to bring about an effect. Works become more ambiguous over time as the inherent, local, “indexical” context in which they are embedded is lost, or overwritten. This makes them effectively “lost” to us; the experience of encountering them as they were intended to be encountered is no longer possible, and they are often defanged as a result.
• Defamiliarization. I’ll quote our paper at length here, because I believe it’s the best explanation I can put forward.

A well-known phenomenon in psychology is the cessation of full awareness of familiar stimuli. The phenomenon has been called “compiling” by Herbert A. Simon, “tacit dimensionality” by philosopher Michael Polanyi, the “ready-to-hand” by Martin Heidegger, and “driving on autopilot” in casual parlance (Ekman 2013). In the terms of the Russian Formalist school [of aesthetics], it is the difference between recognition and seeing, where to recognize is to perceive in a minimal, peripheral way. Under the PEM framework, we can understand this as schemas “explaining away” well-integrated stimuli. Where high-novelty information, or significant clashes between our expectations and sense data, earns more awareness in this model, low novelty or schema-congruent sensory data is allocated less space in the consciousness field.

To the Formalists, a crucial aim of art was defamiliarization, where the everyday and banal (or “ready-to-hand,” or “pre-compiled”) is presented in a way which distorts it into newness. Audiences appreciate a fresh sight of what was previously merely recognized. Often, the defamiliarized subject is not immediately recognizable for what it is; only when the mind connects the defamiliarly presented with the familiarly known, an analogic link is created between the two which upcycles into new models, or interpretations, of the familiar.

An adjacent concept to defamiliarization, taken from psychology, is cognitive disfluency, which describes effortful attention to a stimulus (in contrast to automatic or effortless attention). Disfluent experiences, such as the art encounter, have been found to “improve syllogistic reasoning and reduce reliance on heuristics” (Hurley, Dennett, and Adams 2011). Tellingly, both awkward situations and avant artistic encounters are characterised by the disruption of automaticity in favor of more disfluent hermeneutic or affective states.