Musk open sourced Twitter recommendation algorithm, GitHub reached 10,000 stars in seconds, and promised to evolve every 24-48 hours

Musk’s “open source commitment” finally came true.

Just now, Musk officially announced:

Most (Twitter’s) recommendation algorithms will be open source today, and the rest will follow.

#And Musk immediately showed the address of the source code of the Twitter recommendation algorithm on GitHub.

In just a few hours, tens of thousands of Stars were obtained:

In addition, Musk also said:

Push The recommendation algorithm based on user suggestions will be updated every 24 to 48 hours.

#At this point, the true face of this big algorithm that claims to make recommendations from 500 million tweets every day has finally been revealed.

What does Twitter’s recommendation algorithm look like?

According to reports, Twitter’s recommendation system is based on a set of core models and functions that can extract potential information from tweets, users and interaction data.

The role of these models is to answer important questions in the Twitter network, such as, "What is the probability that you will interact with another user in the future?" or, "What communities are there on Twitter and the trending tweets in them are What?"

If you can answer these questions accurately, you can let Twitter provide more relevant recommendations.

Twitter’s recommendation system consists of three parts:

• Get the best tweets from different recommendation sources. This process is called candidate sources.
• Use a machine learning model to rank each tweet.
• Apply heuristics and filters, such as filtering tweets from blocked users, NSFW content, and seen tweets.

The service responsible for building and delivering the "For You" timeline is called "Home Mixer".

"Home Mixer" is built on top of "Product Mixer", a custom Scala framework that helps build dynamic flows of content.

This service serves as the software backbone, connecting different candidate sources, scoring functions, heuristics and filters.

Candidate Sources

Twitter uses multiple candidate sources to retrieve recent and relevant tweets for users.

For each request, Twitter attempts to extract the best 1,500 tweets from a pool of hundreds of millions of tweets from these sources.

Find candidates from people you follow (internal network) and people you don’t follow (external network).

Today, the "Recommended for You" timeline is made up of an average of 50% internal network tweets and 50% external network tweets (although this can vary by user).

In-Network Source

In-Network Source is the largest candidate source and aims to provide the most relevant and latest tweets to the users who follow it.

It uses a logistic regression model to efficiently rank tweets from people you follow, based on their relevance. The top ranked tweets are then sent to the next stage.

The most important component of ranking internal network tweets is the Real Graph.

Real Graph is a model used to predict the likelihood of interaction between two users. The higher the Real Graph score between the user and the tweet author, the more of their tweets will be included.

Internal network sources have recently become the subject of research by Twitter. Twitter recently stopped using the Fanout service, a 12-year-old service it used to serve internal network tweets from each user's tweet cache.

Out-of-Network Sources

Finding relevant tweets outside a user’s network is a tricky problem.

Because if you don’t follow the author, how do you know if a tweet is relevant to you?

To this end, Twitter has taken two approaches to solve this problem.

One is the Social Graph.

This method is to estimate what you will find relevant by analyzing the activities of people you follow or have similar interests.

Mainly traverse the participating graph and follow the steps below to answer the following questions:

• What messages have the people I follow recently posted on Twitter?
• Who likes similar tweets like me and what else have they liked recently?

The team generated candidate tweets based on the answers to these questions and used a logit model to rank the resulting tweets.

This type of graph traversal is essential for external network recommendations; the team developed GraphJet, a graph processing engine that maintains a real-time interaction graph between users and tweets, to perform these traversals.

While this heuristic for searching Twitter engagement and attention networks has proven useful, the embedded spatial approach has become a greater source of tweets from external networks.

The second one is Embedding Spaces.

The embedded spatial approach aims to answer a more general question about content similarity - what tweets and users share my interests?

Embeddings work by generating digital representations of user interests and tweet content. The similarity between any two users, tweets or user-tweet pairs in this embedding space can then be calculated.

This similarity can be used as a surrogate for correlation as long as accurate embedding is generated.

One of Twitter’s most useful embedding spaces is SimClusters.

SimClusters uses a custom matrix factorization algorithm to discover communities anchored by a group of influential users (145,000 communities, updated every three weeks).

Users and tweets are represented in community spaces and can belong to multiple communities. Communities range in size from a few thousand users in a personal friend group to hundreds of millions in news or pop culture:

…

More with For related details, please click on the link at the end of the article~

One More Thing

Twitter’s algorithm is open source, and quick netizens...have begun to summarize how to become a big V based on the algorithm :

The ratio of the number of people you follow to the number of people following you is very important. Subscribing to TwitterBlue plays a certain role...

Reference link:
[1]​​​https://www.php.cn/link/ce2b9a26bddc32cba5af69372ee14c00​​
[2]​​​https://www.php.cn /link/e55bc0255c752d1cb05da10c0f1f5026​​​
[3]​​​https://www.php.cn/link/10fe8dc69a0964edc16fed1a1bd55716​​​
[4]​​​https://www.php.cn/link/51f4efbfb3e18f4ea053c4d3d282c4e2​​

The above is the detailed content of Musk open sourced Twitter recommendation algorithm, GitHub reached 10,000 stars in seconds, and promised to evolve every 24-48 hours. For more information, please follow other related articles on the PHP Chinese website!