python - How to get the number of forwarded Weibo posts and the number of original Weibo posts of a user on Sina Weibo

Question

1. Problem description
I am doing analysis related to Weibo trolls, and I need to obtain the ratio of forwarded Weibo posts and the ratio of original Weibo posts by Weibo users, that is, the number of users forwarding other people’s Weibo posts/the number of all Weibo posts.

After reviewing the literature, I found that many people’s papers contain this type of data, but I don’t know how to obtain it.

I found that in both the mobile version and the PC version of Weibo, you can use filtering to find a person's forwarded Weibo, but when I filter like this in the mobile version, the returned results are empty.

As for the results obtained in the computer version, the sum of the number of forwarded Weibo and the number of original Weibo is obviously much smaller than the total number of Weibo.

2. Help needed
(1) Is there a problem with Sina Weibo’s search engine?
(2) Can I get the number of a person’s original Weibo posts and forwarded Weibo posts through some kind of search method? Both API and crawler are available. (Do not use the total number of Weibos - the number of original Weibos, because it also includes invisible Weibos).

Answer 1

I open sourced a distributed Weibo crawler. The project was originally used to analyze popular Weibo reposts (related to my own work). Later, I modified it many times, added many new functions, and made it open source. Regarding the first question, is there a problem with Weibo’s search engine? I think it may be intentional, because for example, the Weibo system will only limit the return of 5 pages of data for user followings and user fans, such as searching by location. Users will only return tens of thousands of data. It may also be because there are some Weibo posts that you do not have permission to view, but the results it retrieves are ones that you do have permission to view.
The second question you mentioned is that invisible Weibo cannot be obtained. The crawler can only obtain Weibo data that can be seen by ordinary users. I have also implemented this in the code. Because if it is a private Weibo, it will definitely not be displayed by the Weibo system, and the API should not be available. Because Weibo is such a big company, it still has to protect user privacy.
The reason why your search through the search interface on the mobile terminal returns nothing may actually be because Weibo did this deliberately. If you investigate carefully, you will find that the information on the PC side is much richer than the Weibo information on the mobile side. Therefore, if you want to do a crawler and require comprehensive data, you must start from the PC side.