The System Cannot See Itself - The Deeper Thinking Podcast

The System Cannot See Itself

The Deeper Thinking Podcast is digitally narrated. 

For those drawn to systems thinking, institutional power, artificial intelligence, and the difficult question of how reality can correct the models imposed upon it.

#SystemsThinking #ArtificialIntelligence #InstitutionalDesign #FeedbackLoops #AlgorithmicGovernance #PoliticalPhilosophy

A hospital introduces a scheduling system intended to reduce missed appointments. It works. Attendance improves. Then something more troubling begins to happen. The people classified as unreliable receive fewer choices and shorter confirmation windows, making attendance harder. Their subsequent failures return as evidence that the original classification was correct.

In this episode, we explore the central paradox of systems thinking: the systems we build do not merely observe reality. They enter it, alter behaviour, redistribute opportunity, and gradually teach the world to resemble the models through which it is being judged.

A credit score changes the price of credit. A school ranking changes where families move. A productivity measure changes how work is performed. A crime map changes where police are sent, influencing which offences are detected and what the next map will show. A recommendation system alters what people encounter, then records their altered behaviour as evidence of preference.

The map acquires hands.

Drawing on ideas associated with cybernetics, feedback, reflexivity, and complex systems, the episode asks what happens when measurement ceases to report the world and begins reorganising it.

The danger is not simply that models can be inaccurate. A model may predict successfully while producing the conditions that make its prediction appear true. It may be calibrated precisely to a definition of success that excludes dignity, discretion, uncertainty, or the harms imposed on those with the least power to contest it.

This becomes especially urgent with artificial intelligence. When no individual can reconstruct the entirety of a judgement, a score or classification can begin to feel less like an argument and more like an event. The system has seen something. The output arrives surrounded by the authority of scale and complexity.

Yet transparency alone is not enough. A perfectly explainable system can still impose a category that distorts the life placed inside it. What matters is contestability: whether the person affected can challenge the account of reality on which the decision depends.

A mature system must remain corrigible from below.

Reflections

This episode traces the movement from systems thinking as an escape from isolated blame to systems thinking as a new fantasy of mastery. It asks how institutions might retain the power to act without claiming possession of the whole.

Here are some other reflections that surfaced along the way:

• A model of a human system does not merely represent behaviour; it can reorganise the conditions under which behaviour occurs.
• Predictions can become interventions, and interventions can return as evidence that the prediction was correct.
• No model discovers its own purpose. Someone decides what counts as success, failure, cost, risk, and acceptable harm.
• A system can be exquisitely calibrated and still be calibrated to the wrong thing.
• Once budgets, promotions, and reputations become attached to a measure, the measure no longer reports the work. It reorganises the work.
• The problem is not simplification itself, but whether a simplification remains answerable to what it excludes.
• Transparency matters, but the ability to challenge and alter a decision matters more.
• Discretion, redundancy, local judgement, and appeal may look like inefficiency while functioning as routes through which reality re-enters the system.
• Sometimes friction is information.
• Systems do not absolve responsibility. They relocate it.
• People act within systems, and systems continue through people.
• A free society must preserve the capacity of those within it to interrupt its descriptions.
• A healthy system is not one that eliminates disturbance, but one capable of learning from it.
• Stewardship means intervening without pretending to possess the whole.
• Surprise is the moment reality exceeds the account we have made of it.
• Knowledge does not lose its power when it recognises its position. It loses its alibi.

Why Listen?

• Understand why human systems cannot be analysed as though observers stand outside them
• Explore how feedback loops allow predictions and classifications to reshape behaviour
• Examine why apparently neutral metrics often contain hidden political and ethical decisions
• Consider how Goodhart’s law helps explain why measures deteriorate when institutions organise themselves around them
• Learn why explainability alone cannot protect people from the authority of automated decisions
• Reconsider institutional friction, discretion, and appeal as sources of knowledge rather than obstacles to efficiency
• Explore the difference between controlling a system and stewarding one

Listen On:

• YouTube
• Spotify
• Apple Podcasts

Support This Work

If this episode stayed with you and you’d like to support the ongoing work, you can do so here: Buy Me a Coffee 

Further Reading

• Ludwig von Bertalanffy. General System Theory: Foundations, Development, Applications. George Braziller, 1968.
• Norbert Wiener. Cybernetics: Or Control and Communication in the Animal and the Machine. MIT Press, 1948.
• W. Ross Ashby. An Introduction to Cybernetics. Chapman & Hall, 1956.
• Donella H. Meadows. Thinking in Systems: A Primer. Chelsea Green Publishing, 2008.
• Stafford Beer. Brain of the Firm. Allen Lane, 1972.
• James C. Scott. Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed. Yale University Press, 1998.
• Cathy O’Neil. Weapons of Math Destruction. Crown, 2016.
• Virginia Eubanks. Automating Inequality. St. Martin’s Press, 2018.

Further Reading Relevance

• Ludwig von Bertalanffy: Established general systems theory as a way of understanding organised wholes whose behaviour cannot be explained by examining components in isolation.
• Norbert Wiener: Developed cybernetics around communication, feedback, regulation, and the consequences of systems acting upon information about their own behaviour.
• W. Ross Ashby: Explored adaptation, regulation, and the limits placed on a controller by the complexity of the system it attempts to govern.
• Donella Meadows: Made systems thinking accessible while emphasising feedback loops, delays, leverage points, and the importance of remaining humble before complex systems.
• Stafford Beer: Examined how organisations can remain viable through distributed intelligence, recursive structure, and responsiveness rather than centralised command alone.
• James C. Scott: Showed how administrative systems simplify societies into legible categories, often destroying forms of knowledge that do not fit the governing model.
• Cathy O’Neil: Demonstrated how mathematical models can reproduce inequality while appearing objective, scalable, and technically authoritative.
• Virginia Eubanks: Documented how automated decision systems classify, punish, and constrain vulnerable people through apparently neutral administrative processes.

The system cannot see itself from outside. There is no outside from which to look.

 #SystemsThinking #TheSystemCannotSeeItself #Cybernetics #FeedbackLoops #ComplexSystems #ArtificialIntelligence #AlgorithmicGovernance #AlgorithmicBias #InstitutionalDesign #InstitutionalPower #PoliticalPhilosophy #TechnologyEthics #AIethics #GoodhartsLaw #Reflexivity #Contestability #PublicPolicy #Governance #Stewardship #Metrics #Automation #DecisionSystems #HumanAgency #InstitutionalResponsibility #TheDeeperThinkingPodcast