Recommendation algorithms are everywhere these days, suggesting everything from music to movies to restaurants to potential romantic partners. And for the most part, they’re pretty good at predicting what we like.
But how do these algorithms work? Broadly speaking, an algorithm is a specific set of instructions that, when followed, should always result in a desired outcome. In the case of recommendation algorithms, the desired outcome is to predict what a user will like, based on their past behavior.
To get a better sense of how this works, let’s go back to 2006, when Netflix launched a competition to improve its movie recommendation software, CineMatch. The company offered a $1 million reward to anyone who could increase the success rate of CineMatch by 10%.
As part of the competition, Netflix publicly shared data that included 100 million ratings of 17,770 movies from 480,189 customers. Through online forums, the various competitors shared their progress and ideas on how to improve their algorithms.
One competitor, who went by the pseudonym Simon Funk, came up with the idea of using a math technique called singular value decomposition (SVD) to sort Netflix’s data to automatically find similarities among movies users liked. These factors, identified by the algorithm, were then used to make more accurate recommendations.
SVD is now a cornerstone of most recommendation algorithms. In fact, some of the oldest recorded algorithms in history are written on clay tablets in Babylonia. And the word algorithm is derived from the name of the Persian mathematician abu-Jaʽfar Muhammad ibn Mūsā al’Khwārizmī, who wrote a book on algorithms in the 9th century.
In the almost 20 years since the Netflix Prize competition, recommendation algorithms have only gotten more sophisticated. TikTok, which has arguably the most powerful recommendation algorithm, collects and uses data on the actual watch time of millions of videos from 1.5 billion active monthly users. The app has said publicly that its algorithm takes into account likes, comments, shares, and video watch time; however, investigations from journalists and experts have found that watch time is the key to the app’s eerily psychic algorithm.
Every second you spend on TikTok gives the algorithm more data on the type of content that is most likely to keep you on the app — you don’t even have to hit the like button anymore.
And while these algorithms can be helpful, they are also having unintended real-world implications. YouTube’s algorithm has been known to promote extremist content. Facebook’s algorithm once prioritized angry emoji reactions, resulting in the spread of misinformation. TikTok itself has been found to suppress content made by LGBTQ+ and disabled users.
So, are we right to always blame the algorithm? In the season finale of the Quartz Obsession podcast, Quartz reporter Bruce Gil tells host Gabriela Riccardi about the origins of the algorithms shaping what we do online. You can listen to the episode here:
https://player.fm/series/quartz-obsession/how-algorithms-are-shaping-what-we-do-online
Or, read the transcript here:
https://qz.com/2190899/how-algorithms-are-shaping-what-we-do-online/
