Capture the 'IC Trading Network' supplier program_PHP tutorial-PHP Tutorial-php.cn

Home

Backend Development

PHP Tutorial

Capture the 'IC Trading Network' supplier program_PHP tutorial

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jul 13, 2016 pm 05:48 PM

modify php supplier crawl program

/**
* Capture the "IC Trading Network" supplier main program
* author Lee.
* Last modify $Date: 2012-2-6 10:44:32$
* Note: This program is executed according to the encoding GB2312, because the "IC Trading Network" website is encoded in GB2312, and the database must be consistent
*/
class ic {
    private $key; // 型号
    private $pageNum; // 页码

    /**
* Entry program
*/
    public function go($key) {
        $this->key = $key;
        $this->pageNum = $this->getPageNum();
        $this->getInfo();
    }

    /**
* Get the supplier url link array
* @return ArrayObject
*/
    private function getInfo() {
        if ($this->pageNum==1) { # 处理只有一页的情况
            $arr = $this->shopUrlMatchReArr($this->getContent());
            $this->isAddSuccess($arr);
        } elseif ($this->pageNum>1) { # 多页
            for ($i=1; $i<=$this->pageNum; $i++) {
                $arr = $this->shopUrlMatchReArr($this->getContent($i));
                $this->isAddSuccess($arr);
            }
        }
    }

    /**
* Print whether the addition was successful
* @param ArrayObject $arr
* @return string
*/
    private function isAddSuccess($arr) {
        foreach ($arr as $k=>$v) {
            if ($this->execAdd($this->getInfoByShopUrl($v))) {
                echo 'Add Success!!';
            } else {
                echo 'Add Faild!!';
            }
        }
    }

    /**
* Execute adding to database
* @param ArrayObject $infoArr
* @return Number The number of affected rows
*/
    private function execAdd($infoArr) {
        $mysqli = $this->getDb();
        if (!emptyempty($infoArr['company'])) {
            if (!$this->isExists($mysqli, $infoArr)) {
                $num = $mysqli->query("INSERT INTO ic(company,address,phone,mobile,fax,zip,person,qq,msn,email,website,regDate,shopUrl) VALUES ('{$infoArr['company']}','{$infoArr['address']}','{$infoArr['phone']}','{$infoArr['mobile']}','{$infoArr['fax']}','{$infoArr['zip']}','{$infoArr['person']}','{$infoArr['qq']}','{$infoArr['msn']}','{$infoArr['email']}','{$infoArr['website']}','{$infoArr['regDate']}','{$infoArr['shopUrl']}')");
                return $num;
            } else {
                return false; # 表示数据已经存在
            }
        } else {
            return false;
        }
    }

    /**
* Connect to database
*/
    private function getDb() {
        $mysqli = new mysqli('localhost', 'root', '1715544', 'weiku');
        $mysqli->query('SET NAMES GB2312');
        return $mysqli;
    }

    /**
* Check if the company already exists
* @param Resource $mysqli
* @param ArrayObject $infoArr
* @return bool
*/
    private function isExists($mysqli, $infoArr) {
        $mysqli->query("SELECT company FROM ic WHERE company = '{$infoArr['company']}'");
        if ($mysqli->affected_rows) {
            return true;
        } else {
            return false;
        }
    }

    /**
* Format string
* @param string $str
* @return string
*/
    private function formatString($str) {
        return trim($str);
    }

    /**
* Grab information
* @param $url
* @return ArrayObject
*/
    private function getInfoByShopUrl($url) {
        $re = $this->getUrlInfo($url);
        if (stristr($re, '')) $re = preg_replace('/.*/Usi', '', $re);
        preg_match_all('/(.+)/Usi', $re, $companyArr);
        preg_match_all('/地址:(.*)/Usi', $re, $addressArr);
        preg_match_all('/电话：(.*)/Usi', $re, $phoneArr);
        preg_match_all('/手机：(.*)/Usi', $re, $mobileArr);
        preg_match_all('/传真：(.*)/Usi', $re, $faxArr);
        preg_match_all('/邮编：(.*)/Usi', $re, $zipArr);
        preg_match_all('/联系人：(.*)/Usi', $re, $personArr);
        preg_match_all('/QQ：(.*)/Usi', $re, $qqArr);
        preg_match_all('/MSN：(.*)/Usi', $re, $msnArr);
        preg_match_all('/Email：(.*)/Usi', $re, $emailArr);
        preg_match_all('/网址：(.*)/Usi', $re, $websiteArr);
        preg_match_all('/注册日期：(.*)/Usi', $re, $regDateArr);
        $infoArr = array(
            'company'=>$this->formatString($companyArr[1][0]),
            'address'=>$this->formatString($addressArr[1][0]),
            'phone'=>$this->formatString($phoneArr[1][0]),
            'mobile'=>$this->formatString($mobileArr[1][0]),
            'fax'=>$this->formatString($faxArr[1][0]),
            'zip'=>$this->formatString($zipArr[1][0]),
            'person'=>$this->formatString($personArr[1][0]),
            'qq'=>$this->formatString($qqArr[1][0]),
            'msn'=>$this->formatString($msnArr[1][0]),
            'email'=>$this->formatString($emailArr[1][0]),
            'website'=>$this->stripATags($this->formatString($websiteArr[1][0])),
            'regDate'=>$this->formatString($regDateArr[1][0]),
            'shopUrl'=>$url
        );
        return $infoArr;
    }

    /**
* Get the supplier url array based on the page
* @param string $re
* @return ArrayObject
*/
    private function shopUrlMatchReArr($re) {
        preg_match_all('/.+/Usi', $re, $arr);
        $arr = $this->formatUrlArr(array_unique($arr[1]));
        return $arr;
    }

    /**
* * Format array
* @param Array $arr
* @return ArrayObject
*/
    private function formatUrlArr($arr) {
        $newArr = array();
        foreach ($arr as $key=>$value) {
            if ($this->isExistsHttp($value)) {
                $newArr[$key] = $value;
            }
        }
        return $newArr;
    }

    /**
* Format QQ
* @param string $str
* @return string
*/
    private function formatQqMsn($str, $e='QQ') {
        if (emptyempty($str)) return '';
        preg_match_all('/alt="'.$e.':(.+)"/Usi', $str, $arr);
        if (count($arr[1])==1) return $arr[1][0];
        $newStr = null;
        foreach ($arr[1] as $value) {
            $newStr .= $value . ' ';
        }
        return rtrim($newStr, ' ');
    }

    /**
     * 去掉网址的 A 标签
     * @param string $site
     * @return string
     */
    private function stripATags($site) {
        $site = preg_replace('/(.+)/', '1', $site);
        return $site;
    }

    /**
* Check if the url has http
* @param string $url
* @return bool
*/
    private function isExistsHttp($url) {
        if (stristr($url, 'http://')) {
            return true;
        } else {
            return false;
        }
    }

    /**
* Get page content
* @param Number $page
* @return string
*/
    private function getContent($page=1) {
        $re = file_get_contents($this->getUrl($this->key, $page));
        return $re;
    }

    /**
* Get page number
* @return Number
*/
    private function getPageNum() {
        $i = null;
        $re = $this->getContent();
        preg_match_all('/共(.+)页/Usi', $re, $arr);
        $i = $arr[1][0];
        return $i;
    }

    /**
* Get URL link
* @param string $str
* @param int $page page number
* @return string
*/
    private function getUrl($str, $page=1) {
        return "http://www.ic.net.cn/partsearch/searchinstock.asp?newtype=1&area=&Page={$page}&partnumber={$str}&mfg=&DateCode=&QTY=&PRICE=&Exact=&orderby=inputdate&qty_filter=50&usertype2=1&pack=";
    }

    /**
* Get page content
* @param string $url
* @return string
*/
    private function getUrlInfo($url) {
        $re = file_get_contents($url);
        return $re;
    }
}

/*
程序运行思路：根据“IC 交易网”的IC搜索功能，输入型号进行搜索，然后抓取供应商信息

数据库结构
CREATE TABLE `ic` (
    `id` mediumint(8) unsigned NOT NULL auto_increment,
    `company` varchar(500) NOT NULL,
    `address` varchar(500) default NULL,
    `phone` varchar(500) default NULL,
    `mobile` varchar(500) default NULL,
    `fax` varchar(300) default NULL,
    `zip` varchar(300) default NULL,
    `person` varchar(500) default NULL,
    `qq` varchar(300) default NULL,
    `msn` varchar(300) default NULL,
    `email` varchar(500) default NULL,
    `website` varchar(300) default NULL,
    `regDate` varchar(500) default NULL,
    PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=gb2312
*/

$i = new ic();
$arr = array_unique(array('MAX3232', 'AML8613', 'MT6225A', 'OM8373PS/N3/A', 'PT7313', 'MAX8212ESA', 'TL431', 'S3C2440', 'TMS320F2812PGFA', 'PCM1704', 'AN6717', 'CA3162E', 'CA3161E', 'LM393N', 'DS18B20', 'SHT10', 'AML8613', 'AN6717', 'LM393N', 'CA3161E', 'CA3162E', 'PCM1704', 'STK392-040', 'K1667', 'MAX232', 'STM32F103', 'LM358'));
foreach ($arr as $v) {
    $i->go($v);
}
?>
/**
* Capture the "IC Trading Network" supplier main program
* author Lee.
* Last modify $Date: 2012-2-6 10:44:32$
* Note: This program is executed according to the encoding GB2312, because the "IC Trading Network" website is encoded in GB2312, and the database must be consistent
*/
class ic {
private $key; // 型号
private $pageNum; // 页码

/**
* Entry program
*/
public function go($key) {
  $this->key = $key;
  $this->pageNum = $this->getPageNum();
  $this->getInfo();
}

/**
* Get supplier url link array
* @return ArrayObject
*/
private function getInfo() {
  if ($this->pageNum==1) { # 处理只有一页的情况
   $arr = $this->shopUrlMatchReArr($this->getContent());
   $this->isAddSuccess($arr);
  } elseif ($this->pageNum>1) { # 多页
   for ($i=1; $i<=$this->pageNum; $i++) {
    $arr = $this->shopUrlMatchReArr($this->getContent($i));
    $this->isAddSuccess($arr);
   }
  }
}

/**
* Print whether the addition was successful
* @param ArrayObject $arr
* @return string
*/
private function isAddSuccess($arr) {
  foreach ($arr as $k=>$v) {
   if ($this->execAdd($this->getInfoByShopUrl($v))) {
    echo 'Add Success!!';
   } else {
    echo 'Add Faild!!';
   }
  }
}

/**
* Execute adding to database
* @param ArrayObject $infoArr
* @return Number Number of affected rows
*/
private function execAdd($infoArr) {
  $mysqli = $this->getDb();
  if (!empty($infoArr['company'])) {
   if (!$this->isExists($mysqli, $infoArr)) {
    $num = $mysqli->query("INSERT INTO ic(company,address,phone,mobile,fax,zip,person,qq,msn,email,website,regDate,shopUrl) VALUES ('{$infoArr['company']}','{$infoArr['address']}','{$infoArr['phone']}','{$infoArr['mobile']}','{$infoArr['fax']}','{$infoArr['zip']}','{$infoArr['person']}','{$infoArr['qq']}','{$infoArr['msn']}','{$infoArr['email']}','{$infoArr['website']}','{$infoArr['regDate']}','{$infoArr['shopUrl']}')");
    return $num;
   } else {
    return false; # 表示数据已经存在
   }
  } else {
   return false;
  }
}

/**
* Connect to database
*/
private function getDb() {
  $mysqli = new mysqli('localhost', 'root', '1715544', 'weiku');
  $mysqli->query('SET NAMES GB2312');
  return $mysqli;
}

/**
* Check if the company already exists
* @param Resource $mysqli
* @param ArrayObject $infoArr
* @return bool
*/
private function isExists($mysqli, $infoArr) {
  $mysqli->query("SELECT company FROM ic WHERE company = '{$infoArr['company']}'");
  if ($mysqli->affected_rows) {
   return true;
  } else {
   return false;
  }
}

/**
* Format string
* @param string $str
* @return string
*/
private function formatString($str) {
return trim($str);
}

/**
* Grab information
* @param $url
* @return ArrayObject
*/
private function getInfoByShopUrl($url) {
  $re = $this->getUrlInfo($url);
  if (stristr($re, '')) $re = preg_replace('/.*/Usi', '', $re);
  preg_match_all('/(.+)/Usi', $re, $companyArr);
  preg_match_all('/地址:(.*)/Usi', $re, $addressArr);
  preg_match_all('/电话：(.*)/Usi', $re, $phoneArr);
  preg_match_all('/手机：(.*)/Usi', $re, $mobileArr);
  preg_match_all('/传真：(.*)/Usi', $re, $faxArr);
  preg_match_all('/邮编：(.*)/Usi', $re, $zipArr);
  preg_match_all('/联系人：(.*)/Usi', $re, $personArr);
  preg_match_all('/QQ：(.*)/Usi', $re, $qqArr);
  preg_match_all('/MSN：(.*)/Usi', $re, $msnArr);
  preg_match_all('/Email：(.*)/Usi', $re, $emailArr);
  preg_match_all('/网址：(.*)/Usi', $re, $websiteArr);
  preg_match_all('/注册日期：(.*)/Usi', $re, $regDateArr);
  $infoArr = array(
   'company'=>$this->formatString($companyArr[1][0]),
   'address'=>$this->formatString($addressArr[1][0]),
   'phone'=>$this->formatString($phoneArr[1][0]),
   'mobile'=>$this->formatString($mobileArr[1][0]),
   'fax'=>$this->formatString($faxArr[1][0]),
   'zip'=>$this->formatString($zipArr[1][0]),
   'person'=>$this->formatString($personArr[1][0]),
   'qq'=>$this->formatString($qqArr[1][0]),
   'msn'=>$this->formatString($msnArr[1][0]),
   'email'=>$this->formatString($emailArr[1][0]),
   'website'=>$this->stripATags($this->formatString($websiteArr[1][0])),
   'regDate'=>$this->formatString($regDateArr[1][0]),
   'shopUrl'=>$url
  );
  return $infoArr;
}

/**
* Get the supplier url array based on the page
* @param string $re
* @return ArrayObject
*/
private function shopUrlMatchReArr($re) {
  preg_match_all('/.+/Usi', $re, $arr);
  $arr = $this->formatUrlArr(array_unique($arr[1]));
  return $arr;
}

/**
* Format array
* @param Array $arr
* @return ArrayObject
*/
private function formatUrlArr($arr) {
  $newArr = array();
  foreach ($arr as $key=>$value) {
   if ($this->isExistsHttp($value)) {
    $newArr[$key] = $value;
   }
  }
  return $newArr;
}

/**
* Format QQ
* @param string $str
* @return string
*/
private function formatQqMsn($str, $e='QQ') {
  if (empty($str)) return '';
  preg_match_all('/alt="'.$e.':(.+)"/Usi', $str, $arr);
  if (count($arr[1])==1) return $arr[1][0];
  $newStr = null;
  foreach ($arr[1] as $value) {
   $newStr .= $value . ' ';
  }
  return rtrim($newStr, ' ');
}

/**
* Remove the A tag from the URL
* @param string $site
* @return string
*/
private function stripATags($site) {
$site = preg_replace('/(.+)/', '1', $site);
return $site;
}

/**
* Check if the url has http
* @param string $url
* @return bool
*/
private function isExistsHttp($url) {
  if (stristr($url, 'http://')) {
   return true;
  } else {
   return false;
  }
}

/**
* Get page content
* @param Number $page
* @return string
*/
private function getContent($page=1) {
  $re = file_get_contents($this->getUrl($this->key, $page));
  return $re;
}

/**
* Get page number
* @return Number
*/
private function getPageNum() {
  $i = null;
  $re = $this->getContent();
  preg_match_all('/共(.+)页/Usi', $re, $arr);
  $i = $arr[1][0];
  return $i;
}

/**
* Get URL link
* @param string $str
* @param int $page page number
* @return string
*/
private function getUrl($str, $page=1) {
return "http://www.ic.net.cn/partsearch/searchinstock.asp?newtype=1&area=&Page={$page}&partnumber={$str}&mfg=&DateCode=&QTY=&PRICE=&Exact=&orderby=inputdate&qty_filter=50&usertype2=1&pack=";
}

/**
* Get page content
* @param string $url
* @return string
*/
private function getUrlInfo($url) {
$re = file_get_contents($url);
return $re;
}
}

/*
程序运行思路：根据“IC 交易网”的IC搜索功能，输入型号进行搜索，然后抓取供应商信息

数据库结构
CREATE TABLE `ic` (
`id` mediumint(8) unsigned NOT NULL auto_increment,
`company` varchar(500) NOT NULL,
`address` varchar(500) default NULL,
`phone` varchar(500) default NULL,
`mobile` varchar(500) default NULL,
`fax` varchar(300) default NULL,
`zip` varchar(300) default NULL,
`person` varchar(500) default NULL,
`qq` varchar(300) default NULL,
`msn` varchar(300) default NULL,
`email` varchar(500) default NULL,
`website` varchar(300) default NULL,
`regDate` varchar(500) default NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=gb2312
*/

$i = new ic();
$arr = array_unique(array('MAX3232', 'AML8613', 'MT6225A', 'OM8373PS/N3/A', 'PT7313', 'MAX8212ESA', 'TL431', 'S3C2440', 'TMS320F2812PGFA', 'PCM1704', 'AN6717', 'CA3162E', 'CA3161E', 'LM393N', 'DS18B20', 'SHT10', 'AML8613', 'AN6717', 'LM393N', 'CA3161E', 'CA3162E', 'PCM1704', 'STK392-040', 'K1667', 'MAX232', 'STM32F103', 'LM358'));
foreach ($arr as $v) {
$i->go($v);
}
?>

摘自 Lee.的专栏

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Assassin's Creed Shadows: Seashell Riddle Solution

3 weeks ago By DDD

What's New in Windows 11 KB5054979 & How to Fix Update Issues

2 weeks ago By DDD

Where to find the Crane Control Keycard in Atomfall

3 weeks ago By DDD

Assassin's Creed Shadows - How To Find The Blacksmith And Unlock Weapon And Armour Customisation

1 months ago By DDD

Roblox: Dead Rails - How To Complete Every Challenge

3 weeks ago By DDD

Hot Tools

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Where is the login entrance for gmail email?

7610

CakePHP Tutorial

1387

What is the format of the account name of steam

win11 activation key permanent

nyt connections hints and answers

136

Related knowledge

PHP 8.4 Installation and Upgrade guide for Ubuntu and Debian Dec 24, 2024 pm 04:42 PM

PHP 8.4 brings several new features, security improvements, and performance improvements with healthy amounts of feature deprecations and removals. This guide explains how to install PHP 8.4 or upgrade to PHP 8.4 on Ubuntu, Debian, or their derivati

7 PHP Functions I Regret I Didn't Know Before Nov 13, 2024 am 09:42 AM

If you are an experienced PHP developer, you might have the feeling that you’ve been there and done that already.You have developed a significant number of applications, debugged millions of lines of code, and tweaked a bunch of scripts to achieve op

How To Set Up Visual Studio Code (VS Code) for PHP Development Dec 20, 2024 am 11:31 AM

Visual Studio Code, also known as VS Code, is a free source code editor — or integrated development environment (IDE) — available for all major operating systems. With a large collection of extensions for many programming languages, VS Code can be c

Explain JSON Web Tokens (JWT) and their use case in PHP APIs. Apr 05, 2025 am 12:04 AM

JWT is an open standard based on JSON, used to securely transmit information between parties, mainly for identity authentication and information exchange. 1. JWT consists of three parts: Header, Payload and Signature. 2. The working principle of JWT includes three steps: generating JWT, verifying JWT and parsing Payload. 3. When using JWT for authentication in PHP, JWT can be generated and verified, and user role and permission information can be included in advanced usage. 4. Common errors include signature verification failure, token expiration, and payload oversized. Debugging skills include using debugging tools and logging. 5. Performance optimization and best practices include using appropriate signature algorithms, setting validity periods reasonably,

How do you parse and process HTML/XML in PHP? Feb 07, 2025 am 11:57 AM

This tutorial demonstrates how to efficiently process XML documents using PHP. XML (eXtensible Markup Language) is a versatile text-based markup language designed for both human readability and machine parsing. It's commonly used for data storage an

PHP Program to Count Vowels in a String Feb 07, 2025 pm 12:12 PM

A string is a sequence of characters, including letters, numbers, and symbols. This tutorial will learn how to calculate the number of vowels in a given string in PHP using different methods. The vowels in English are a, e, i, o, u, and they can be uppercase or lowercase. What is a vowel? Vowels are alphabetic characters that represent a specific pronunciation. There are five vowels in English, including uppercase and lowercase: a, e, i, o, u Example 1 Input: String = "Tutorialspoint" Output: 6 explain The vowels in the string "Tutorialspoint" are u, o, i, a, o, i. There are 6 yuan in total

Explain late static binding in PHP (static::). Apr 03, 2025 am 12:04 AM

Static binding (static::) implements late static binding (LSB) in PHP, allowing calling classes to be referenced in static contexts rather than defining classes. 1) The parsing process is performed at runtime, 2) Look up the call class in the inheritance relationship, 3) It may bring performance overhead.

What are PHP magic methods (__construct, __destruct, __call, __get, __set, etc.) and provide use cases? Apr 03, 2025 am 12:03 AM

What are the magic methods of PHP? PHP's magic methods include: 1.\_\_construct, used to initialize objects; 2.\_\_destruct, used to clean up resources; 3.\_\_call, handle non-existent method calls; 4.\_\_get, implement dynamic attribute access; 5.\_\_set, implement dynamic attribute settings. These methods are automatically called in certain situations, improving code flexibility and efficiency.

See all articles