data collected - digital traces - search frequency, engagement behaviour and click duration, tagged by content and categorised by content type. search queries, videos watched, dwell times, clicks, shares and hesitations
often collected in the browser by social media sites and similar - processed and stored on the company’s servers
the algorithm aggregates the data and assigns categories to the data, which are used to fit a model of prediction of the content they’ll engage with
the advertisers and sponsor companies benefit from these decisions, and the users are harmed by having their behaviour affected and modified by where the algorithm has placed them and what it chooses to show them. In extreme (but common!) cases, users are radicalised
sometimes they make this data available to the user to edit and correct when there are incorrect predictions, but this is more recent and still uncommon
the information that is intentionally obscured is the aggregation method, the model’s weightings and biases towards extreme content, the user’s intent and response to engaging in items of content
the company has access to change, regulate, or contest these decisions, but no external regulatory authorities, safety mechanisms or individual users
risks and vulnerabilities include positive feedback loops on harmful content and radicalisation as well as inaccurate/factually incorrect information being shared
errors, biases, or uncertain outcomes may emerge as a result of misinterpreted signals as there is no input from the user’s actual thoughts and intentions behind interactions
the system assumes that certain kinds of interactions indicate positive or negative interactions, desirable or undesirable, and that the desirable interactions are ones that benefit advertisers and sponsors. It also assumes consent for any of this data to be collected
unintended consequences are in promotion of content that individuals aren’t aligned with to the extent that the company are seen as promoting particular views
missing data affects a system’s fairness and accountability as the outcome can only be dependent on the data it is given - for example, some users may take longer to read items before scrolling past due to language processing issues but this may be interpreted as engagement and interest - without the intention/context behind the action it prioritises complex content that is difficult to parse, making the experience more difficult for those with disabilities or those using it in a non-native language - this amplifies existing inequalities
there would be no accountability for this inequality as the system doesn’t reveal where the decision is made (or who is responsible for it) to highlight this kind of content for these particular users, and so there is no concrete proof of injustice
a key question that a designer or researcher should ask to expose or mitigate this uncertainty is ‘What information is not collected on a user’s engagement with content that if included would change their categorisation by the model?’
to make the system more transparent there would have to be opt-in for data sharing and ability to edit the model or reject the decisions it makes
policymakers could prevent this kind of data being collected and stored without consent, demand that the companies make their model publicly available, and demand that the companies let the user opt-out and/or edit their preferences manually. This would likely only happen at the demand of advocacy groups. Users must be involved in handling their own data but default mechanisms that favour the user’s rights and autonomy would benefit those who aren’t data literate but still need protections.
this system should be guided by transparency and user control to ensure that any data stored and decisions made are in favour of the user and not to the benefit of the advertisers/company/sponsors
constraints or trade-offs might be complication of interface, unwillingness/inability of users to engage in data handling, and other shortcuts/mods/add-ins/plug-ins/etc being employed to skip this/offload the decisions to another, equally unaccountable third-party