HTTP proxy server-Python Tutorial-php.cn

HTTP proxy server

高洛峰

Release： 2016-10-19 15:24:20

Original

4360 people have browsed it

Recently, I plan to study python socket programming in depth, so I plan to learn it and imitate it. I find that it is not easy to write well. Many problems arise in the middle. It is easy to see but difficult to do.

import socket
import thread
import urlparse
import select
   
BUFLEN=8192
   
   
class Proxy(object):
    def __init__(self,conn,addr):
        self.source=conn
        self.request=""
        self.headers={}
        self.destnation=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
        self.run()
   
    def get_headers(self):
        header=&#39;&#39;
        while True:
            header+=self.source.recv(BUFLEN)
            index=header.find(&#39;\n&#39;)
            if index >0:
                break
        #firstLine,self.request=header.split(&#39;\r\n&#39;,1) 
        firstLine=header[:index]
        self.request=header[index+1:]
        self.headers[&#39;method&#39;],self.headers[&#39;path&#39;],self.headers[&#39;protocol&#39;]=firstLine.split()
   
    def conn_destnation(self):
        url=urlparse.urlparse(self.headers[&#39;path&#39;])
        hostname=url[1]
        port="80"
        if hostname.find(&#39;:&#39;) >0:
            addr,port=hostname.split(&#39;:&#39;)
        else:
            addr=hostname
        port=int(port)
        ip=socket.gethostbyname(addr)
        print ip,port
        self.destnation.connect((ip,port))
        data="%s %s %s\r\n" %(self.headers[&#39;method&#39;],self.headers[&#39;path&#39;],self.headers[&#39;protocol&#39;])
        self.destnation.send(data+self.request)
        print data+self.request
   
   
    def renderto(self):
        readsocket=[self.destnation]
        while True:
            data=&#39;&#39;
            (rlist,wlist,elist)=select.select(readsocket,[],[],3)
            if rlist:
                data=rlist[0].recv(BUFLEN)
                if len(data)>0:
                    self.source.send(data)
                else:
                    break
    def run(self):
        self.get_headers()
        self.conn_destnation()
        self.renderto()
   
   
   
class Server(object):
   
    def __init__(self,host,port,handler=Proxy):
        self.host=host
        self.port=port
        self.server=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
        self.server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.server.bind((host,port))
        self.server.listen(5)
        self.handler=handler
   
    def start(self):
        while True:
            try:
                conn,addr=self.server.accept()
                thread.start_new_thread(self.handler,(conn,addr))
            except:
                pass
   
   
if __name__==&#39;__main__&#39;:
    s=Server(&#39;127.0.0.1&#39;,8080)
    s.start()
import socket
import thread
import urlparse
import select
BUFLEN=8192
class Proxy(object):
    def __init__(self,conn,addr):
        self.source=conn
        self.request=""
        self.headers={}
        self.destnation=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
        self.run()
    def get_headers(self):
        header=&#39;&#39;
        while True:
            header+=self.source.recv(BUFLEN)
            index=header.find(&#39;\n&#39;)
            if index >0:
                break
        #firstLine,self.request=header.split(&#39;\r\n&#39;,1)
        firstLine=header[:index]
        self.request=header[index+1:]
        self.headers[&#39;method&#39;],self.headers[&#39;path&#39;],self.headers[&#39;protocol&#39;]=firstLine.split()
    def conn_destnation(self):
        url=urlparse.urlparse(self.headers[&#39;path&#39;])
        hostname=url[1]
        port="80"
        if hostname.find(&#39;:&#39;) >0:
            addr,port=hostname.split(&#39;:&#39;)
        else:
            addr=hostname
        port=int(port)
        ip=socket.gethostbyname(addr)
        print ip,port
        self.destnation.connect((ip,port))
        data="%s %s %s\r\n" %(self.headers[&#39;method&#39;],self.headers[&#39;path&#39;],self.headers[&#39;protocol&#39;])
        self.destnation.send(data+self.request)
        print data+self.request
    def renderto(self):
        readsocket=[self.destnation]
        while True:
            data=&#39;&#39;
            (rlist,wlist,elist)=select.select(readsocket,[],[],3)
            if rlist:
                data=rlist[0].recv(BUFLEN)
                if len(data)>0:
                    self.source.send(data)
                else:
                    break
    def run(self):
        self.get_headers()
        self.conn_destnation()
        self.renderto()
   
class Server(object):
    def __init__(self,host,port,handler=Proxy):
        self.host=host
        self.port=port
        self.server=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
        self.server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.server.bind((host,port))
        self.server.listen(5)
        self.handler=handler
    def start(self):
        while True:
            try:
                conn,addr=self.server.accept()
                thread.start_new_thread(self.handler,(conn,addr))
            except:
                pass
if __name__==&#39;__main__&#39;:
    s=Server(&#39;127.0.0.1&#39;,8080)
    s.start()

Copy after login

In fact, Http The proxy server itself is not difficult, but it is quite troublesome to write. I will not go into the source code in detail here, it is very simple. Mainly talk about the problems I encountered.

1: I originally only knew that the first parameter of thread.start_new_thread is a function object, but when I saw the above blog post, I was stunned. This is okay, so I quickly tested it:

import thread
   
class Hello:
    def __init__(self,content):
        print content
   
def cs():
        thread.start_new_thread(Hello, ("Hello World",))
   
if __name__==&#39;__main__&#39;:
    cs()
import thread
class Hello:
    def __init__(self,content):
        print content
def cs():
        thread.start_new_thread(Hello, ("Hello World",))
if __name__==&#39;__main__&#39;:
    cs()

Copy after login

Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:

How can a parameter be an object? I laughed and despised the author a little. So, I went to bed. The next day, I still didn’t give up, so I downloaded the code and tried it locally. It was ok. I immediately realized that it was me, so I immediately went to Baidu.

It turns out that in the thread module, if the main thread ends before the child thread, this exception will be thrown, so we must let the child thread end first. The simplest way is to let the main thread sleep for a long enough time. As for how long , seems not to know, so how to solve it?

A better solution is that the main thread adds a lock to each sub-thread, the sub-thread releases the lock before ending, and the main thread keeps checking the status of the lock in a loop. The code is as follows:

import thread
   
class Hello:
    def __init__(self,content,lock):
        print content
        """
        do something
        ....
        At the end,release the lock
        """
        lock.release()
   
def cs():
        lock=thread.allocate_lock()
        lock.acquire()
        thread.start_new_thread(Hello, ("Hello World",lock))
        while True:
            if not lock.locked():
                break
        print "lock release"
   
if __name__==&#39;__main__&#39;:
    cs()
import thread
class Hello:
    def __init__(self,content,lock):
        print content
        """
        do something
        ....
        At the end,release the lock
        """
        lock.release()
def cs():
        lock=thread.allocate_lock()
        lock.acquire()
        thread.start_new_thread(Hello, ("Hello World",lock))
        while True:
            if not lock.locked():
                break
        print "lock release"
if __name__==&#39;__main__&#39;:
    cs()

Copy after login

2. The second error is comparing 2

self.source.send[data]

peError: 'builtin_function_or_method' object is unsubscriptable

self.source.send[data]

TypeError : 'builtin_function_or_method' object is unsubscriptable

The main meaning is that built-in functions or methods cannot have subscripts, you know