Mobike crawler source code analysis

The first two articles analyzed why I grabbed Mobike’s interface and As a result of data analysis, this article directly provides executable source code for learning

Statement:
This crawler is only for learning and research purposes, please do not use it for illegal purposes. . Any legal disputes caused by this will be your own responsibility.

If you don’t have the patience to read the article, please post directly:

git clone https://github.com/derekhe/mobike-crawler
python3 crawler.py

Please don’t forget to give it a star and ##!

#Directory structure

• \analysis - jupyter for data analysis

• \influx-importer - import to influxdb, but I didn’t do it well before

• \

  modules - Agent module

• \web - Real-time graphical The display module was just to learn

  react. The effect can be found here

• crawler.py - crawler core code

• importToDb.py - Import into postgres database for analysis

• sql.sql - Create table sql

##start.sh - Continue Running script
Idea

The core code is placed in crawler.py, the data is first stored in the

sqlite

3 database, and then after deduplication Export to a csv file to save space. Mobike’s

API

returns a bicycle in a square area. I can capture the entire area by moving it piece by piece. Large area data. left,

top

,right,bottom defines the crawling range, which is currently the Chengdu City Ring Expressway. Within and the square area south to Nanhu. offset defines the crawling interval. It is currently based on 0.002 and can be used within 15 minutes on the DigitalOcean 5$ server. Fetch it once.

    def start(self):
        left = 30.7828453209
        top = 103.9213455517
        right = 30.4781772402
        bottom = 104.2178123382

        offset = 0.002

        if os.path.isfile(self.db_name):
            os.remove(self.db_name)

        try:
            with sqlite3.connect(self.db_name) as c:
                c.execute('''CREATE TABLE mobike
                    (Time DATETIME, bikeIds VARCHAR(12), bikeType TINYINT,distId INTEGER,distNum TINYINT, type TINYINT, x DOUBLE, y DOUBLE)''')
        except Exception as ex:
            pass

Then 250 threads are started. As for you asking me why I didn’t use coroutines, humming~~I didn’t learn it at the time~~~It’s actually possible, maybe it’s more efficient. High.

Since the data needs to be deduplicated after crawling, in order to eliminate duplicate parts between small square areas, the last group_data is the core API for doing this. The code is here. For the API interface of the mini program, just create a few

variables

, it is very simple.

        executor = ThreadPoolExecutor(max_workers=250)
        print("Start")
        self.total = 0
        lat_range = np.arange(left, right, -offset)
        for lat in lat_range:
            lon_range = np.arange(top, bottom, offset)
            for lon in lon_range:
                self.total += 1
                executor.submit(self.get_nearby_bikes, (lat, lon))

        executor.shutdown()
        self.group_data()

Finally, you may want to ask if frequent IP grabbing is not blocked? In fact, Mobike has IP access speed restrictions, but the way to crack it is very simple, which is to use a large number of proxies. I have an agent pool, and there are basically more than 8,000 agents every day. Get this proxy pool directly in ProxyProvider and provide a pick

function

to randomly select the top 50 proxies. Please note that my proxy pool is

updated

every hour, but the jsonblob proxy list provided in the code is just a sample, and most of it should be invalid after a while. . A proxy scoring mechanism is used here. Instead of selecting agents directly at random, I sorted the agents according to their scores. Each successful request will add points, while an erroneous request will lose points. In this way, the agent with the best speed and quality can be selected in a short time. You can save it and use it next time if necessary.

    def get_nearby_bikes(self, args):
        try:
            url = "https://mwx.mobike.com/mobike-api/rent/nearbyBikesInfo.do"

            payload = "latitude=%s&longitude=%s&errMsg=getMapCenterLocation" % (args[0], args[1])

            headers = {
                'charset': "utf-8",
                'platform': "4",
                "referer":"https://servicewechat.com/wx40f112341ae33edb/1/",
                'content-type': "application/x-www-form-urlencoded",
                'user-agent': "MicroMessenger/6.5.4.1000 NetType/WIFI Language/zh_CN",
                'host': "mwx.mobike.com",
                'connection': "Keep-Alive",
                'accept-encoding': "gzip",
                'cache-control': "no-cache"
            }

            self.request(headers, payload, args, url)
        except Exception as ex:
            print(ex)

In actual use, select the proxy through proxyProvider.pick() and then use it. If there are any problems with the proxy, directly use proxy.fatal_error() to lower the score, so that this proxy will not be selected in the future.

class ProxyProvider:
    def init(self, min_proxies=200):
        self._bad_proxies = {}
        self._minProxies = min_proxies
        self.lock = threading.RLock()

        self.get_list()

    def get_list(self):
        logger.debug("Getting proxy list")
        r = requests.get("https://jsonblob.com/31bf2dc8-00e6-11e7-a0ba-e39b7fdbe78b", timeout=10)
        proxies = ujson.decode(r.text)
        logger.debug("Got %s proxies", len(proxies))
        self._proxies = list(map(lambda p: Proxy(p), proxies))

    def pick(self):
        with self.lock:
            self._proxies.sort(key = lambda p: p.score, reverse=True)
            proxy_len = len(self._proxies)
            max_range = 50 if proxy_len > 50 else proxy_len
            proxy = self._proxies[random.randrange(1, max_range)]
            proxy.used()

            return proxy

Okay, that’s basically it~~~Study the other codes yourself~~~

The above is the detailed content of Mobike crawler source code analysis. For more information, please follow other related articles on the PHP Chinese website!