High concurrency and fast response correspond to the two cores of performance optimization Metrics:
Throughput and LatencyPicture from: www.ctq6.cn
##Application loadAngle: Directly affects the user experience of the product terminal
System resourcesPerspective: resource usage, saturation, etc.
The essence of performance problemsThat is, the system resources have reached the bottleneck, but the request processing is not fast enough to support more requests. Performance analysis is actually to find the bottlenecks of the application or system and try to avoid or alleviate them.
Select metrics to evaluate application and system performance
Set performance goals for applications and systems
Proceed Performance benchmark test
Performance analysis to locate bottlenecks
Performance monitoring and alarm
For different Performance issues require the selection of different performance analysis tools. The following are commonly used Linux Performance Tools and the corresponding types of performance problems analyzed.
Picture from: www.ctq6.cn
##How should we understand "average load"
Average load: The average number of processes in the system's runnable and uninterruptible states per unit time, that is, the average number of active processes. It is not directly related to CPU usage as we traditionally understand it.
The uninterruptible process is a process that is in a critical process in the kernel state (such as a common I/O response waiting for a device).
The uninterruptible state is actually a protection mechanism of the system for processes and hardware devices.
What is the reasonable average load?
In the actual production environment, monitor the average load of the system and determine the load change trend based on historical data. When there is an obvious upward trend in load, conduct timely analysis and investigation. Of course, you can also set a threshold (such as when the average load is higher than 70% of the number of CPUs)
In real work, we often confuse the concepts of average load and CPU usage. In fact, the two are not completely equivalent. :
CPU-intensive process, a large amount of CPU usage will cause the average load to increase, at this time the two are consistent
I/O-intensive Processes, waiting for I/O will also cause the average load to increase. At this time, the CPU usage is not necessarily high.
A large number of process scheduling waiting for the CPU will cause the average load to increase. At this time, The CPU usage will also be relatively high
#When the average load is high, it may be caused by CPU-intensive processes or busy I/O. During specific analysis, you can combine the mpstat/pidstat tool to assist in analyzing the load source
2CPU
##CPU context switching (Part 1)
CPU context switching is to save the CPU context (CPU registers and PC) of the previous task, then load the context of the new task into these registers and program counter, and finally Then jump to the location pointed by the program counter and run the new task. Among them, the saved context will be stored in the system kernel and loaded again when the task is rescheduled for execution to ensure that the original task status is not affected. Follow the Linux Chinese community
According to task type, CPU context switching is divided into:
Process context switching
Thread context switching
Interrupt context switching
Process context switching
Linux process divides the running space of the process into kernel space and user space according to the level of permissions. The transition from user mode to kernel mode needs to be completed through system calls.
A system call process actually performs two CPU context switches:
The user mode instruction position in the CPU register is saved first, the CPU register is updated to the kernel mode instruction position, and jumps to the kernel mode to run the kernel task;
After the system call is completed, the CPU register restores the originally saved user state data, and then switches to user space to continue running.
The system call process does not involve process user-mode resources such as virtual memory, nor does it switch processes. It is different from process context switching in the traditional sense. Therefore system calls are often called privileged mode switches.
The process is managed and scheduled by the kernel, and process context switching can only occur in the kernel state. Therefore, compared with system calls, before saving the kernel state and CPU registers of the current process, the virtual memory and stack of the process need to be saved first. After loading the kernel state of the new process, the virtual memory and user stack of the process must be refreshed.
The process only needs to switch context when it is scheduled to run on the CPU. There are the following scenarios: CPU time slices are allocated in turns, insufficient system resources cause the process to hang, the process actively hangs through the sleep function, high priority The first-level process seizes the time slice. When the hardware interrupts, the process on the CPU is suspended and instead executes the interrupt service in the kernel.
Thread context switching
Thread context switching is divided into two types:
The front and rear threads belong to the same process. The virtual memory resources remain unchanged when switching. You only need to switch the thread's private data, registers, etc.;
The front and rear threads belong to the same process. Different processes, the same as process context switching.
Thread switching in the same process consumes less resources, which is also the advantage of multi-threading.
Interrupt context switching
Interrupt context switching does not involve the user mode of the process, so the interrupt context only includes the state necessary for the execution of the kernel mode interrupt service program (CPU registers, kernel stack, hardware interrupt parameters, etc.).
Interrupt processing priority is higher than that of the process, so interrupt context switching and process context switching will not occur at the same time
CPU context switching (Part 2)
You can view the overall context switching situation of the system through vmstat
cs (context switch) Number of context switches per second
in (interrupt) Number of interrupts per second
r (running or runnable) ready queue The length of For detailed information about each process, you need to use pidstat to view the context switching of each process
The CPU usage of an application reaches 100%, what should I do?
Linux, as a multi-tasking operating system, divides the CPU time into short time slices and allocates them to each task in turn through the scheduler. In order to maintain CPU time, Linux triggers time interrupts through pre-defined beat rates and uses global jiffies to record the number of beats since booting. Time interrupt occurs once. This value is 1.
CPU usage, the percentage of total CPU time other than idle time. CPU usage can be calculated from the data in /proc/stat. Because the cumulative value of the number of beats since booting in /proc/stat is calculated as the average CPU usage since booting, which is generally of little significance. You can calculate the average CPU usage during that period by taking the difference between two values taken at intervals of a period of time. The performance analysis tool gives the average CPU usage over a period of time. Pay attention to the setting of the interval.
CPU usage can be viewed through top or ps. You can analyze the CPU problems of the process through perf, which is based on performance event sampling. It can not only analyze various events of the system and kernel performance, but also can be used to analyze the performance problems of specified applications.
perf top / perf record / perf report (-g turns on the sampling of calling relationships)
sudo docker run --name nginx -p 10000:80 -itd feisky/nginx
sudo docker run --name phpfpm -itd --network container:nginx feisky/php-fpm
ab -c 10 -n 100 http://XXX.XXX.XXX.XXX:10000/ #测试Nginx服务性能
This is still just a guess. The next step is to continue analyzing it through the perf tool. The performance report shows that stress actually takes up a lot of CPU. It can be optimized and solved by fixing the permission problem.
What should I do if there are a large number of uninterruptible processes and zombie processes in the system?
Process status
R Running/Runnable, indicating that the process is in the ready queue of the CPU and is running or waiting to be run;
D Disk Sleep, uninterruptible state sleep, generally indicates that the process is interacting with the hardware, and is not allowed to be interrupted by other processes during the interaction;
Z Zombie, a zombie process, means that the process has actually ended, but the parent process has not reclaimed its resources;
S Interruptible Sleep, which can interrupt the sleep state, means that the process is waiting for something Suspended by the system for an event, when the waiting event occurs, it will be awakened and enter the R state;
I Idle, idle state, used on kernel threads that cannot interrupt sleep. This state will not cause the average load to increase;
T Stop/Traced, indicating that the process is suspended or traced (SIGSTOP/SIGCONT, GDB debugging);
X Dead, the process has died and will not be seen in top/ps.
After layer-by-layer analysis, the root cause is direct disk I/O inside the app. Then locate the specific code location for optimization.
Zombie Process
After the above optimization, iowait has dropped significantly, but the number of zombie processes is still increasing. First, locate the parent process of the zombie process. Use pstree -aps XXX to print out the call tree of the zombie process and find that the parent process is the app process.
Check the app code to see if the child process end is handled correctly (whether wait()/waitpid() is called, whether there is a SIGCHILD signal processing function registered, etc.).
When encountering an increase in iowait, first use tools such as dstat pidstat to confirm whether there is a disk I/O problem, and then find out which processes are causing the I/O. If you cannot use strace to directly analyze process calls, you can Analyzed by perf tool.
For the zombie problem, use pstree to find the parent process, and then look at the source code to check the processing logic for the end of the child process.
CPU performance indicators
CPU usage
User CPU usage, including user mode (user) and low-priority user mode (nice). If this indicator is too high, it indicates that the application is busy.
System CPU usage Rate, the percentage of time the CPU is running in kernel mode (excluding interrupts). A high indicator indicates that the kernel is relatively busy.
CPU usage waiting for I/O, iowait, a high indicator It shows that the I/O interaction time between the system and the hardware device is relatively long.
Soft/hard interrupt CPU usage. A high indicator indicates that a large number of interrupts occur in the system.
steal CPU / guest CPU, indicates the percentage of CPU occupied by the virtual machine.
##Average loadIdeally average load Equal to the number of logical CPUs, it means that each CPU is fully utilized. If it is greater than the number, it means that the system load is heavy.
Process context switching
Includes voluntary switching when resources cannot be obtained and involuntary switching when the system forces scheduling. Context switching itself is a core function to ensure the normal operation of Linux. Excessive switching will consume the CPU time of the original running process in the register , kernel occupancy and virtual memory and other data saving and recovery
CPU cache hit rate
CPU cache reuse situation, the higher the hit rate The higher the performance, the better. L1/L2 is commonly used in single-core, and L3 is used in multi-core
Performance Tools
Load Average Case
First use uptime to check the average load of the system
After judging that the load has increased, use mpstat and pidstat to check the CPU usage of each CPU and each process respectively. Find There are processes that lead to higher load average. In addition, search the public account Linux and reply "git books" in the background to get a surprise gift package.
Context switching case
First use vmstat to check the system context switching and interrupt times
Use pidstat to observe the voluntary and involuntary context switching of the process
Finally, use pidstat to observe the context switching of the thread
Case of high CPU usage of process
First use top to check the CPU usage of the system and process, and locate the process
Use perf top to observe the process call chain and locate the specific function
Case of high system CPU usage
# #First use top to check the CPU usage of the system and process. Neither top/pidstat can find the process with high CPU usage
Re-examine the top output
Start with processes that have low CPU usage but are in the Running state
perf record/report found that short-term processes caused (execsnoop tool)
Case of uninterruptible and zombie processes
First use top to observe the increase of iowait and find a large number of uninterruptible and zombie processes
strace cannot trace process system calls
perf analysis call chain found that the root cause comes from disk direct I/O
Soft interrupt case
topObserve the high CPU usage of system soft interrupts
Check /proc/softirqs to find several soft interrupts with fast changing rates
sar The command found that it was a network packet problem
tcpdump found out the type and source of the network frame, and determined that the SYN FLOOD attack caused
According to different Use the performance indicators to find the appropriate tool:
Picture from: www.ctq6.cn
In a production environment, developers often do not have permission to install new tools. Toolkits can only make the most of the tools already installed in the system. Therefore, it is necessary to understand what indicator analysis some mainstream tools can provide.
Pictures from: www.ctq6.cn
First run a few tools that support more indicators, such as top/vmstat/pidstat. Based on their output, you can get Find out what type of performance problem it is. After locating the process, use strace/perf to analyze the calling situation for further analysis. If it is caused by soft interrupt, use /proc/softirqs
Picture from: www .ctq6.cn
CPU Optimization
##Application Optimization
Compiler optimization: Enable optimization options during the compilation phase, such as gcc -O2
Algorithm optimization
Asynchronous processing: Avoid the program from being blocked waiting for a certain resource, and improve the concurrent processing capability of the program. (Replace polling with event notification)
Make good use of cache: Speed up program processing
System optimization
CPU binding: Bind the process to one/multiple CPUs to improve the CPU cache hit rate and reduce context switching caused by CPU scheduling
CPU exclusive: CPU affinity mechanism to allocate processes
Priority adjustment: Use nice to appropriately reduce the priority of non-core applications
Set resource display for the process: cgroups sets the usage limit to prevent an application from exhausting system resources
NUMA optimization: CPU accesses local memory as much as possible
Interrupt load balancing: irpbalance, automatically load balance the interrupt processing process to each CPU
##TPS, QPS , The difference and understanding of system throughput
##QPS(TPS)
Number of concurrencies
Response time
QPS(TPS)=Number of concurrencies /Average response time
User request server
Server internal processing
The server returns to the client
QPS is similar to TPS, but a visit to a page forms a TPS, but a page request may include multiple requests to the server , may be counted multiple times QPS
QPS (Queries Per Second) query rate per second, the number of queries a server can respond to per second.
TPS (Transactions Per Second) number of transactions per second, software test results.
System Throughput, including several important parameters:
3Memory
How Linux memory works
Memory mapping
The main memory used by most computers is dynamic random access memory (DRAM) ), only the kernel can directly access physical memory. The Linux kernel provides an independent virtual address space for each process, and this address space is continuous. In this way, the process can easily access memory (virtual memory).
The interior of the virtual address space is divided into two parts: kernel space and user space. The range of the address space of processors with different word lengths is different. The 32-bit system kernel space occupies 1G and the user space occupies 3G. The kernel space and user space of 64-bit systems are both 128T, occupying the highest and lowest parts of the memory space respectively, and the middle part is undefined.
Not all virtual memory will allocate physical memory, only the actual use will. The allocated physical memory is managed through memory mapping. In order to complete memory mapping, the kernel maintains a page table for each process to record the mapping relationship between virtual addresses and physical addresses. The page table is actually stored in the CPU's memory management unit MMU, and the processor can directly find out the memory to be accessed through hardware.
When the virtual address accessed by the process cannot be found in the page table, the system will generate a page fault exception, enter the kernel space to allocate physical memory, update the process page table, and then return to user space to resume the operation of the process.
MMU manages memory in units of pages, with a page size of 4KB. In order to solve the problem of too many page table entries, Linux provides the Multi-level page table and HugePage mechanisms.
Virtual memory space distribution
User space memory has five different memory segments from low to high:
Read-only segment Code and constants, etc.
##Data segment Global variables, etc.
Heap Dynamically allocated memory starts from low address and grows upward
File mapping Dynamic library, shared memory Wait, starting from the high address and growing downward
Stack Including local variables and the context of function calls, etc., the size of the stack is fixed. Generally 8MB
Memory allocation and recycling
Allocation
malloc corresponds to the system call Two implementation methods:
brk() For small blocks of memory (<128K), allocate by moving the top position of the heap. The memory is not returned immediately after it is released, but is cached.
**mmap()**For large blocks of memory (>128K), allocate directly using memory mapping, that is, find a free memory allocation in the file mapping segment.
The former cache can reduce the occurrence of page fault exceptions and improve memory access efficiency. However, because the memory is not returned to the system, frequent memory allocation/release will cause memory fragmentation when the memory is busy.
The latter is directly returned to the system when released, so a page fault exception will occur every time mmap occurs. When memory work is busy, frequent memory allocation will cause a large number of page fault exceptions, increasing the kernel management burden.
The above two calls do not actually allocate memory. These memories only enter the kernel through page fault exceptions when they are accessed for the first time, and are allocated by the kernel.
Recycling
When memory is tight, the system reclaims memory in the following ways:
RES The size of the resident memory, that is, the actual use of the process The physical memory size, excluding swap and shared memory
SHR Shared memory size, memory shared with other processes, loaded dynamic link libraries and program code segments
%MEM The percentage of physical memory used by the process to the total system memory
How to understand Buffer and Cache in memory?
buffer is a cache of disk data, cache is a cache of file data, they will be used in both read requests and write requests
How to use the system cache to optimize the running efficiency of the program
Cache hit rate
Cache hit Rate refers to the number of requests to obtain data directly through the cache, accounting for the percentage of all requests. The higher the hit rate, the higher the benefits brought by the cache and the better the performance of the application.
After installing the bcc package, you can monitor cache read and write hits through cachestat and cachetop.
After installing pcstat, you can view the cache size and cache ratio of files in memory
#首先安装Go
export GOPATH=~/go
export PATH=~/go/bin:$PATH
go get golang.org/x/sys/unix
go ge github.com/tobert/pcstat/pcstat
The allocated memory is not properly reclaimed, resulting in leakage
The address outside the allocated memory boundary is accessed, causing the program to exit abnormally
Memory allocation and recycling
Virtual memory distribution from low to high is read-only segment, data segment, heap, memory mapping segment, StackFive parts. Among them, the ones that can cause memory leaks are:
Heap: allocated and managed by the application itself. Unless the program exits, these heap memories will not be automatically released by the system.
Memory mapping segment: including dynamic link libraries and shared memory, where shared memory is automatically allocated and managed by the program
Memory leak The harm is greater. Not only can the application itself not access the memory that is forgotten to be released, but the system cannot allocate it to other applications again. Memory leaks accumulate and may even exhaust system memory.
Cache/buffer is a recyclable resource, usually called a file page in file management
In the application, fsync is used to remove dirty files The pages are synchronized to the disk
and handed over to the system. The kernel thread pdflush is responsible for refreshing these dirty pages.
has been modified by the application and has not been written yet. Disk data (dirty pages) must be written to the disk first before the memory can be released.
The file mapping page obtained by memory mapping can also be released the next time it is accessed. Re-read from the file
For the heap memory automatically allocated by the program, which is our anonymous page in memory management, although this memory cannot be released directly, Linux provides the Swap mechanism to The memory that is not frequently accessed is written to the disk to free up the memory, and the memory can be read from the disk when accessed again.
Swap principle
The essence of Swap is to use a piece of disk space or a local file as memory, including two processes of swapping in and swapping out:
Swapping out: Store the memory data temporarily unused by the process to disk and release the memory
Swapping in: When the process accesses the memory again, They are read from disk into memory
How does Linux measure whether memory resources are tight?
Direct Memory Recycling New large block memory allocation request, but insufficient remaining memory. At this time, the system will reclaim a part of the memory;
kswapd0 The kernel thread periodically reclaims the memory. In order to measure memory usage, three thresholds of pages_min, pages_low, and pages_high are defined, and memory recycling operations are performed based on them.
Remaining memory
pages_min < remaining memory < pages_low, the memory pressure is large, kswapd0 performs memory recycling until the remaining memory > pages_high
The above is the detailed content of Summary of Linux performance optimization knowledge points · Practice + Collection Edition. For more information, please follow other related articles on the PHP Chinese website!
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
