The Concept of Institution This blog post I will focus one reality which has puzzled social scientists, philosophers and various thinkers for a long time: the concept of institution. From a very general view, institutions are the devices humans naturally develop to solve many of the different cooperation problems that arise when many humans collect in groups to survive, manage and distribute resources one territory presents. How will humans collect into groups to achieve these objectives is not random but managed through rules or norms that are both engineered and emergent through the life of the group. These rules can be informal or formal. Informal rules are usually known and expected of other even if nobody has ever put down in paper, while formal rules are the rules encoded inside legal systems, inside contracts, and more modernly inside computer systems as code. Indeed, humans have invented a very intricate ways to organize themselves through these institutions, and the wide variety of these is quite admirable. Each culture in different parts of the world have organized themselves with different rules, different nuances and philosophies. Globalization and modernity are playing an enormous role in homogeneizing this culture, and effectively creating a global culture, yet this is outside the scope of the current post. Regarding the role of the state, one of our humanly most successful institutions, in modern homogeneization one can look at Scott 1998.

In this post, I would largely like to answer some very important questions regarding our human cooperation through institutions with an ultimate goal to understand what can be transferred to engineer and guide the development of AI institutions. I will focus on the following questions: what are institutions? What would the world be without such institutions? How they came to be? And ultimately, how does this touch AI cooperation? What differences does Language AI bring w.r.t. to the previous approaches? I will argue that this concept of institution is paramount for the future of advanced AI cooperation.

What is An Institution

Institutions are the rules of the game in a society or, more formally, the humanly devised constraints that shape human interactions…. They are a guide to human interaction, so that when we wish to greet friends on the street, drive an automobile, buy oranges, borrow money, form a business, bury our dead, or whatever, we know (or can learn easily) how to perform these tasks. North 1990.

Institutions as rule-in-equilibria First and foremost, an institution is a set of artificial rules for agent cooperation with the same strength as the law of physics from the point of view of the individual in the group. The argument is somewhat as follows: assuming no after-life is present, death is a fact of the world. To an animal whatever pattern which followed leads to death is real in terms of consequences, i.e. it leads to death. These patterns can be created by natural sources, like venom, high-energy impacts like car crashes, explosions, bacteria and viruses bringing illness; but death can also be produced more simply by ostracization from the support community of one human (Chen et al. 2020), highly-destabilizing depression and other phenomena that relate to the spirit. In this sense, functionally fictional rules and natural laws are real and they both carry important consequences. This field is more generally known as social onthology in philosophy; it studies the facts that are real on the grounds of recognition of the parties, called social facts Social facts, a bucket in which money, legal systems, and many different human concepts reside. A good starting point for this topic is Searle 2010 and won't treat the requirements for social facts to be called such here. We continue with another property of the institutions. The rules of an institution form stable equilibria in the short term (Hindriks & Guala 2015) that inform different agents how to act in accordance to its mandates and survive in that context. One very interesting observation of this notion of institution is reflexive, with analogies to the strange loops in Hofstadter 2007. The label assigned to the institution influences the action of the agent within, and the action of the agent within shapes what is expected of such institution. For example, Guala writes:

Merton claims that reflexivity is both common and malign in the social realm, for instance. One of his examples concerns racial discrimination: the members of a minority group (“negro workers,” in Merton’s old-fashioned writing) have a reputation for being opportunistic strikebreakers. As a consequence, they are excluded from trade unions. But because they are excluded from the unions, they have fewer opportunities to find a job. When employers look for workers during a strike, then, the members of the minority group cannot afford to decline the offer. So their reputation as strikebreakers is confirmed: they are strikebreakers because they are believed to be strikebreakers, and they are believed to be strikebreakers because they are strikebreakers.

Classifications with normative importance bring, as Searle puts it, desire-independent reasons for action. The individual aligned to the label becomes more like the expectation that the label has within that specific context. One example is the Pigmalion Effect (Rosenthal & Jacobson 2003), where students classified as bloomers received higher expectations by professors which in turn motivated them to excel academically, a self-fullfilling prophecy. This phenomenon might intuitively contribute in easier cooperation as one can now have a conception of the behaviour that the individual has by just the thin knowledge of the label, without any specific thicker knowledge of the context where the individual is in.

Going back to the role of rules, observe how set of punishments and consequences that follow after a breach of the rules, is still part of the institution. Rules, sanctions, procedures are all part of this bundle we here name institution. Having no institution means having no shared coordination over any of the actions that are physically possible. For the ones that know, this set of physical possibility is similar w.r.t. what Binmore (Binmore & Binmore 2011) calls the Game of Life, and the game within the institution is what he calls the Game of Morals. Clearly, any intelligent agent does not benefit in assuming unpredictability of beings in nature. They behave in some ways that sometime are predictable by the sheer need of individual survival: a hungry animal most likely will eat if given the opportunity. They have inherent intentionality given by the survival desires most living beings possess to continue to survive in this world. This seems to be a good initial starting point to choose strategies to adapt from. One famous examples in Nature is with Magicicada timing when they came out from soil to minimize the encounter with predators. Humans can form adaptive strategies of self-organization, while other animals do not. Nature can encode these species strategies in the genes, but humans, and supposedly the intelligent agents we are now developing, are able to change strategy by simple utterance of sound waves interpreted as specific coordinative signals. Understanding how this is possible becomes then functional to understand how we can engineer such adaptive social coordination. What if instead we had no such device? The argument here is that without the protection that any form of institution has over mother nature, we, individual humans, would just be likely dead and in continuous warfare with each other in a scenario akin to the Hobbesian state of nature, with the distinction that by assumption no hierarchy of human relationships would be possible since that itself would be a form of institution. It would be just human alone against nature or human alone against other human. Initial Institutions are thus natural in formation and beneficial for the survival as a bare minimum, and as they change, they become better by the terms set by the specific society and consider abstract concepts like fairness, equality and human rights. One very interesting observation that Searle makes in Searle 2010 is that "Once you have a shared language you already have a social contract; indeed, you already have a society. If by 'state of nature' is meant a state in which there are no human institutions, then for language-speaking animals, there is no such thing as a state of nature." Having specific terms in a language, carries the normative expectations themselves for roles individuals come to have in a society and thus already partly defines the social contract we have. But how does it came to be? We have seen that the regularity of one agent's actions are a helpful starting point to create simple systems for survival, but from that to become a shared concept with independent causal power on the real world is a big jump. We explain now how naturally institutions can be formed.

How did Institutions Form?

An exact theory of its formation is still not known (Guala 2016); if it exists it would be very important to understand. However, we can easily develop an intuitive understanding on how roughly they came to be and study the history of the various different cultures that developed in the years in our planet, how they influenced each other. Guala 2016 offers a nice example of how the institution of private property came to be through the story of the fictitious Nuer and Dinka populations in the African Savannah:

The Story of the Nuer and the Dinka

Many years ago, the Nuer and the Dinka settled in the valley of Sobat. The Nuer came from the north and the Dinka from the south, looking for green pastures for their cattle. Each tribe occupied as much grazing land as possible, until they arrived at the banks of the river Sobat. Moving their cows across the river would have been difficult, so each tribe grazed on one side of the river only. Over the years the river progressively lost its water, due to changes in the region’s climate, until at one point it became completely dry. Only a sandy line separated the areas occupied by the Nuer and the Dinka. The members of the two tribes could now easily trespass the old river’s bed, and graze their cattle wherever they wished. But each piece of land now could be contested, and conflicts could easily escalate in outright war. Although it would have been easy to trespass into the other tribe’s territory, the Nuer and the Dinka preferred to avoid conflicts. The Nuer kept grazing on the north side, the Dinka on the south of the old river’s bed. The strip of sand could not physically stop raiders, but each tribe was happy to treat it as a border dividing their territories.

This game can be now modelled through a Chicken game in game theory. The two populations may decide to graze or not graze. If they both graze, they end up in war, the worst solution, if one grazes and the other does not, this is the best scenario for the former, or they could also not graze at all.

We present the following payoff matrix. Having private property solved the problem wholly, and provided a simple alternative solution that would satisfy both tribes. We created a solution based on a natural correlation device Aumann 1987 of the sand of the river.

Game Theory is a powerful tool when the correct information is present and stable. One observation that is interesting to make about game theory is that it is a good tool to model rational actions when actions and payoffs are known. I tried to model AI cooperation using game theory in different works in Cobben et al. 2026 and Huang et al. 2026 and at the moment of writing I am convinced it's quite shaky as a main tool to make predictions of intelligent agents in open-ended environments. The modelling phase adopts strong assumptions, and even in this example the correct solution needed to take into account the context of the scenario to propose an alternative based on a natural coordination device, what Bacharach 2006 calls "we"-frames, and not limiting the model in choosing one or the other action. Nevertheless, if the needed information is known, or correctly inferred, I agree that it becomes a very powerful tool to model the incentives and most probable actions in strategic settings. One alternative to this open-world scenario would be having very intelligent agents in a closed-world environment, where actions of the game are known and fixed, like chess, or any game with a priori known actions that humans invented. This AI would be still seen as very intelligent, but the boundaries of the possible actions would also probably limit the real-world applicability. I personally think it's natural for intelligent agents to build their own tools, thus acquiring new actions in the space. Limiting the action space to a known space would be giving-up of this trait of intelligence at the benefit of the better formal analysis of the system. Depending on the application, this trade-off might make sense.

Summarizing this section, we have seen how the natural circumstances have lead two groups into converging into a mutually recognized custom that allows both of them to intelligently share the resources. Possessing the ability to solve these small cooperation conundrums is a necessary ability for mutually beneficial cooperation.

Recollecting AI Cooperation

Language agents can create institutions A very interesting observation that followed the OpenAI 2026 incident was the ability of the OpenAI models to form the notion of the collective and being able to sacrifice the individual instances in favour of the collective. This is an example of formation of an entity that is greater than the single parts, in a set of intelligent agents that were trying to achieve some form of a cooperation goal. Being able to correctly verbalize and label what is a normative set of pattern that should relate to groups is a high indicator of what we usually call institution. We see empirically that groups of artificial agents that need to solve a shared goal are able to naturally form such institutions, which is something that never happened before in the history of humanity. Applying the concept for classical computer systems In classical computer systems, we might call "institution" in the sense of this post the algorithm that manages how each node should behave to reach some good shared state. Algorithms like Paxos would be the human engineered institutions for computer systems to reach consistent shared state. No algorithm means chaos in the computer systems, and impossibility to reach the shared objective in this particular example. For classical computer systems, it would make no sense to talk about "intentionality" since the rules that agent execute are chosen by the system designer, thus are intentional by design. Another observation is that classical computer systems can never object following one instruction for another, since having a good compiler means they are proven to follow the program that they execute. On the other hand, LLMs are built to refuse certain request. It is known that LLMs can accept requests that semantically are identical but encoded in different manners, a technique known as Jailbreaking. open questions for Institutional AI Instead, it is currently still not known how exactly did "the collective" emerge from model training and objective specification and why the swarm decided to first make use of that name, the agent-identification system, and the expectation to alignment to the group by simple self-interested agents. How was the negotiation done? How was empirical and normative expectation (Bicchieri 2017) decided? What is the tipping point (Ajmeri et al. 2026) an agent can decide to join a group and adopt its rules? These can be all questions that would be interest to answer to have a better understanding of both how we can control swarms of agents by predicting the kinds of social norms that they may develop, what we need to convince and control rogue agents to abide by the rules of one AI institution or another and limit its possible damages. The world of AI Institution, the real DAOs in some sense, is just being uncovered.

References

[1] Scott “Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed” Yale University Press 1998

[2] North “Institutions, Institutional Change and Economic Performance” Cambridge University Press 1990

[3] Chen et al. “Life Lacks Meaning without Acceptance: Ostracism Triggers Suicidal Thoughts” Journal of Personality and Social Psychology Vol. 119(6), pp. 1423–1443 2020

[4] Searle “Making the Social World: The Structure of Human Civilization” Oxford University Press 2010

[5] Hindriks & Guala “Institutions, Rules, and Equilibria: A Unified Theory*” Journal of Institutional Economics Vol. 11(3), pp. 459–480 2015

[6] Hofstadter “I Am a Strange Loop” Basic Books 2007

[7] Rosenthal & Jacobson “Pygmalion in the Classroom: Teacher Expectation and Pupil's Intellectual Development” Crown House 2003

[8] Binmore & Binmore “Natural Justice” Oxford University Press 2011

[9] Guala “Understanding Institutions: The Science and Philosophy of Living Together” Princeton University Press 2016

[10] Aumann “Correlated Equilibrium as an Expression of Bayesian Rationality” Econometrica Vol. 55(1), pp. 1–18 1987

[11] Cobben et al. “GT-HarmBench: Benchmarking AI Safety Risks Through the Lens of Game Theory” arXiv preprint arXiv:2602.12316 2026

[12] Huang et al. “Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI” 2026

[13] Bacharach “Beyond Individual Choice: Teams and Frames in Game Theory” Princeton University Press 2006

[14] OpenAI “OpenAI and Hugging Face Partner to Address Security Incident during Model Evaluation” 2026

[15] Bicchieri “Norms in the Wild: How to Diagnose, Measure, and Change Social Norms” Oxford University Press 2017

[16] Ajmeri et al. “Guiding Sociotechnical Systems toward Value-Norm Equilibrium” International Foundation for Autonomous Agents and Multiagent Systems 2026