Recent searches
Search options
It occurs to me that “the purpose of the system is what it does” and “the intent of the system is what it does” are subtly different statements and tightly intertwined but very often conflated and it could be productive to disentangle them a little bit
@kevinriggle Here was my somewhat-phoned-in attempt in -- I think -- a similar direction: https://buttondown.com/nelhage/archive/what-is-the-purpose-of-posiwid/
@nelhage yes! Oh this is precisely it
@nelhage (hence my itchiness around it from a blame-is-the-enemy-of-safety perspective)
@kevinriggle Immediately after I published that, @njs articulated a related but alternate formulation I wish I had come up with:
> When examining a complex social system, you _must not_ get hung up on any particular individual's internal intentions (because those are unknowable, and because it's not even a meaningful question because these systems are larger than any single person anyway, and because much experience shows that is an attractive tarpit for discussions); but
[continued]
> […] but "purpose" is still a useful concept for high level reasoning and lots of our moral intuitions, and so the proper way to take advantage of those cognitive tools is to slot in whatever the system actually does in the space labeled "purpose"
@kevinriggle
Huh: I often quote the first (purpose) but have never heard the latter. “Intent” seems not only incorrect but misguided: AIUI the original Stafford Beers idea of POSIWID was to get away from arguing about what other people think and pay attention to what actually •happens•. But “intent” does capture how people often misunderstand POSIWID.
@inthehands yeah no one says the latter, I just made it up. One of the things we do a lot in safety engineering is try to write down the system designer’s intent in structured ways, capturing all the important invariants and interactions, so that we can make sure when we make future modifications that we don’t accidentally connect two things that shouldn’t be connected. So when people say “intent doesn’t matter” I’m like, hmm. Surely not as a way to avoid taking responsibility, no, but as a way of capturing invariants, maybe
@inthehands (of course then you discover that one of the invariants the system designers had in mind is “Black people shouldn’t be able to go to the beach” and you’re like hmm maybe that is not an unacceptable loss for us anymore and it shouldn’t have been for you either, Mr. Moses)
@kevinriggle
How do you handle socially unacceptable intents? I mean this in both directions — there are a lot of contexts where prosocial intents are smuggled into systems by the mutual assent of some set of parties involved.
@inthehands
@kevinriggle @inthehands @nelhage definitely agree I've heard POSIWID invoked in both of these senses. I find the first sense more useful, though, in addressing harmful systems, as a reminder that:
* a system's effect may be different from its designer's intent, i.e. original intent should not be a defense of the present system
* its effect may change, i.e. previous benefits don't justify current harms
* it's possible for a system to serve *nobody's* intent (e.g. arms races), although some people will find a way to benefit from it (e.g. defense contractors)
The second, accusatory sense seems to be more narrowly useful (e.g. Moses). There are plenty of harmful effects that can't be traced back to a single person's malicious foresight.