Approaching the world as a software problem is a category error that has led us into some terrible habits of mind.
We obsess over these fake problems while creating some real ones. In our attempt to feed the world to software, techies have built the greatest surveillance apparatus the world has ever seen. … Just like industrialized manufacturing changed the relationship between labor and capital, surveillance capitalism is changing the relationship between private citizens and the entities doing the tracking. Our old ideas about individual privacy and consent no longer hold in a world where personal data is harvested on an industrial scale.
We tend to imagine dystopian scenarios as one where a repressive government uses technology against its people. But what scares me in these scenarios is that each one would have broad social support, possibly majority support. Democratic societies sometimes adopt terrible policies.
The goal should be not to make the apparatus of surveillance politically accountable (though that is a great goal), but to dismantle it. Just like we don’t let countries build reactors that produce plutonium, no matter how sincere their promises not to misuse it, we should not allow people to create and indefinitely store databases of personal information. The risks are too high.
Source: Remarks at the SASE Panel On The Moral Economy of Tech