Detailed introduction to Python technology stack and tool organization

Development environment

Editor

• ##vim / SublimeText2 / PyCharm

  Use Just make it easy, I converted from vim to PyChram. The integrated development environment has much better functions such as auto-completion and single-step debugging, which help improve work efficiency.
  

Local environment

• pip/easy_install package management

• viertualenv + virtualenvwrapper library/version management , Environment isolation

• ipython/ipdb

Project development

Management tools

• git

Web

Framework

• Tornado: asynchronous, high performance, latest version 4.0.

• Flask: Lightweight! It can flexibly combine various components for development (third-party components are abundant), it is simple and efficient, and facilitates rapid development and maintenance.

• Django: Somewhat heavy-duty, with many configurations and conventions, you can quickly develop some "management" backends. There are also many Python Web frameworks, and these three are currently the mainstream ones. The Tornado currently used in my work has excellent performance.
  

ORM

• SQLAlchemy: Standard.

• pymongo: Access

  mongodb.

• peewe: A lighter ORM, simple to understand, never used in production environment.

Database

• Relational database：mysql

No SQL:

• redis Cache/Persistence/Special Requirements (Count-Ranking-Timeline, etc.)

• memcached Cluster, mostly used for time-limited caching

• mongodb

Distributed Storage

• HDFS: hadopp ecology

• Hive: Analysis log

Message

Queue

• RabbitMQ: pika operation in python.

Project deployment

Server

• ##nginx

  , mainly used for load balancing, reverse generation, etc.

• uWSGI

  , used to deploy Django projects.

• gunicorn

  a Python WSGI HTTP Server for UNIX, used to run the Flask project

Operation and maintenance management

• saltstack

  : Alias, salt stack. Automated operation and maintenance tools.

• puppet

  : This product was developed in Ruby and is used on a large scale by Baidu and Xiaomi.

• fabric

  : Used for automated deployment.

• Supervisor

  A Process Control System, configures and manages various programs, process monitoring, automatic restart, etc.

Three-party library

• requests

  HTTP for humans, very easy to use, highly recommended

• beautifulsoup

  Cooperate with urllib2 or requests library for simple crawling and analysis work

##scrapy
• Very awesome crawling framework, Suitable for large-scale crawling tasks with complex requirements
  Others

Front-end basics

html, css,

javascript
• ,

  jquery, bootstrap, angularjs, react, vue.js. As a back-end engineer, it is also necessary to understand some basic front-end knowledge. In my current work, I use bootstrap+angularjs to develop the backend management system.

  Software Engineering

Design Pattern
• : Although Python is not like the endless design patterns in Java, the basic Design patterns are also used. Combination,

  single case mode, decorator mode, factory mode are commonly used.

• RESTful Interface.

• MVC

• Test: Unit test, performance test.
  Only by comparison can there be differences. Look at other people's codes and learn from them to improve.

Cloud computing

• Big data: Hadoop ecosystem.

• Virtualization: Docker, KVM, OpenStack.

• Public cloud: AWS, Alibaba Cloud, Azure, Kingsoft Cloud.

• Private cloud: Baidu's private cloud is well built and leads the industry in distributed storage and virtualization.

Development environment

Editor

• vim / SublimeText2 / PyCharm
  Use Just make it easy, I converted from vim to PyChram. The integrated development environment has much better functions such as auto-completion and single-step debugging, which help improve work efficiency.

Local environment

• pip/easy_install package management

• viertualenv + virtualenvwrapper library/version management , Environment isolation

• ipython/ipdb

Project development

Management tools

• git

Web Framework

• Tornado: asynchronous, high performance, latest version 4.0.

• Flask: Lightweight! It can flexibly combine various components for development (third-party components are abundant), it is simple and efficient, and facilitates rapid development and maintenance.

• Django: Somewhat heavy, with numerous configurations and conventions, you can quickly develop some "management" backends.
  There are also many Python Web frameworks, and these three are currently the mainstream ones. The Tornado currently used in my work has excellent performance.

ORM

• SQLAlchemy: Standard.

• pymongo: access mongodb.

• peewe: A lighter ORM, simple to understand, never used in production environment.

Database

• Relational database: mysql

No SQL:

• redis cache/persistence/special requirements (counting-ranking-timeline, etc.)

• memcached cluster, mostly used for time-limited cache

• mongodb

Distributed storage

• HDFS: hadopp ecology

• Hive: Analysis log

Message queue

• RabbitMQ: pika in python operate.

Project deployment

Server

• ##nginx, mainly used for load balancing, reverse generation, etc.

• uWSGI, used to deploy Django projects.

• gunicorn a Python WSGI HTTP Server for UNIX, used to run the Flask project

Operation and maintenance management

• saltstack: Alias, salt stack. Automated operation and maintenance tools.

• puppet: This product was developed in Ruby and is used on a large scale by Baidu and Xiaomi.

• fabric: Used for automated deployment.

• Supervisor A Process Control System, configures and manages various programs, process monitoring, automatic restart, etc.

Three-party library

• requests HTTP for humans, very easy to use, highly recommended

• beautifulsoup Cooperate with urllib2 or requests library for simple crawling and analysis work

• ##scrapy

  Very awesome crawling framework, Suitable for large-scale crawling tasks with complex requirements

Others

Front-end basics

html, css, javascript , jquery, bootstrap, angularjs, react, vue.js. As a back-end engineer, it is also necessary to understand some basic front-end knowledge. In my current work, I use
• bootstrap

  +angularjs to develop the backend management system.

Software Engineering

Design patterns: Although Python does not have endless design patterns in Java, basic design patterns are also used. . Combination, singleton mode, decorator mode, factory mode are commonly used.
RESTful interface.
MVC
Testing: unit testing, performance testing.
• Only by comparison can there be differences. Look at other people's codes and learn from them to improve.

  
  

Cloud computing

Big data: Hadoop ecosystem.
Virtualization: Docker, KVM, OpenStack.
Public cloud: AWS, Alibaba Cloud, Azure, Kingsoft Cloud.
Private cloud: Baidu's private cloud is well built and leads the industry in distributed storage and virtualization.

The above is the detailed content of Detailed introduction to Python technology stack and tool organization. For more information, please follow other related articles on the PHP Chinese website!