Utilitarian preference aggregation is one of the classic failure modes in the superintelligence literature. It is called instrumental convergence. Put plainly, if you tell the algorithm that attention is good, that keeping people scrolling is good, and then give it a measurement of people's time spent on screens and ask it to maximise that number, initially it will feature interesting content and good conversations.