High concurrency and fast response correspond to the two cores of performance optimization Metrics: Throughput and Latency
Picture from: www.ctq6.cn
Application loadAngle : Directly affects the user experience of the product terminal
System resourcesPerspective: resource usage, saturation, etc.
The essence of the performance problem is that the system resources have reached the bottleneck, but the request processing is not fast enough to support more requests. Performance analysis is actually to find the bottlenecks of the application or system and try to avoid or alleviate them.
Select metrics to evaluate application and system performance
Set performance goals for applications and systems
Perform performance benchmark testing
Performance analysis to locate bottlenecks
Performance monitoring and alarm
For different performance problems, different performance analysis tools should be selected. The following are commonly used Linux Performance Tools and the corresponding types of performance problems analyzed.
Picture from: www.ctq6.cn
How should we understand "average load"
Average load: The average number of processes in the system's runnable and uninterruptible states per unit time, that is, the average number of active processes. It is not directly related to CPU usage as we traditionally understand it.
The uninterruptible process is a process that is in a critical process in the kernel state (such as a common I/O response waiting for a device). The uninterruptible state is actually a protection mechanism of the system for processes and hardware devices.
What is the reasonable average load?
In the actual production environment, the average load of the system is monitored, and the load change trend is judged based on historical data. When there is an obvious upward trend in load, conduct timely analysis and investigation. Of course, you can also set a threshold (such as when the average load is higher than 70% of the number of CPUs)
In real work, we often confuse the concepts of average load and CPU usage. In fact, the two They are not completely equivalent:
CPU-intensive processes, a large amount of CPU usage will cause the average load to increase, and the two are consistent at this time
For I/O-intensive processes, waiting for I/O will also cause the average load to increase. At this time, the CPU usage is not necessarily high.
A lot of waiting for the CPU Process scheduling will cause the average load to increase, and the CPU usage will also be relatively high at this time
When the average load is high, it may be caused by CPU-intensive processes , it may also be caused by busy I/O. During specific analysis, you can combine the mpstat/pidstat tool to assist in analyzing the load source
2CPU
CPU context switching (Part 1)
CPU context switching is to save the CPU context (CPU registers and PC) of the previous task, then load the context of the new task into these registers and program counter, and finally jump to the program counter pointed to. location to run a new task. Among them, the saved context will be stored in the system kernel and loaded again when the task is rescheduled for execution to ensure that the original task status is not affected.
According to task type, CPU context switching is divided into:
Process context switching
Thread context switching
Interrupt context switching
Process context switching
Linux process assigns the running space of the process according to the level of permissions Divided into kernel space and user space. The transition from user mode to kernel mode needs to be completed through system calls.
A system call process actually performs two CPU context switches:
The user mode instruction position in the CPU register is saved first, the CPU register is updated to the kernel mode instruction position, and jumps to the kernel mode to run the kernel task;
After the system call is completed, the CPU registers restore the originally saved user state data, and then switch to user space to continue running.
# During the system call process, process user-mode resources such as virtual memory will not be involved, nor will processes be switched. It is different from process context switching in the traditional sense. Therefore system calls are often called privileged mode switches.
The process is managed and scheduled by the kernel, and process context switching can only occur in the kernel state. Therefore, compared with system calls, before saving the kernel state and CPU registers of the current process, the virtual memory and stack of the process need to be saved first. After loading the kernel state of the new process, the virtual memory and user stack of the process must be refreshed.
The process only needs to switch context when it is scheduled to run on the CPU. There are the following scenarios: CPU time slices are allocated in turn, insufficient system resources cause the process to hang, and the process passes the sleep function In active suspension, high-priority processes seize the time slice. When a hardware interrupt occurs, the process on the CPU is suspended and instead executes the interrupt service in the kernel.
Thread context switching
Thread context switching is divided into two types:
The front and rear threads are the same Belongs to one process, the virtual memory resources remain unchanged when switching, only the private data, registers, etc. of the thread need to be switched;
The front and rear threads belong to different processes and are different from the process context Switching is the same.
# Thread switching in the same process consumes less resources, which is also the advantage of multi-threading.
Interrupt context switching
Interrupt context switching does not involve the user mode of the process, so the interrupt context only includes the state necessary for the execution of the kernel mode interrupt service program (CPU registers, kernel stack, hardware interrupt parameters, etc.).
#Interrupt processing priority is higher than that of the process, so interrupt context switching and process context switching will not occur at the same time
Docker K8s Jenkins mainstream technology comprehensive video information
CPU context switching (Part 2)
You can view the overall context switching situation of the system through vmstat
What should I do if there are a large number of uninterruptible processes and zombie processes in the system?
Process status
R Running/Runnable, indicating that the process is in the CPU's ready queue, running or waiting Run;
#D Disk Sleep, uninterruptible state sleep, generally means that the process is interacting with the hardware, and is not allowed to be interrupted by other processes during the interaction;
Z Zombie, a zombie process, means that the process has actually ended, but the parent process has not reclaimed its resources;
S Interruptible Sleep, which can interrupt the sleep state, means that the process is suspended by the system because it is waiting for an event. When the waiting event occurs, it will be awakened and enter the R state;
I Idle, idle state, used on kernel threads that cannot interrupt sleep.该状态不会导致平均负载升高；
T Stop/Traced，表示进程处于暂停或跟踪状态（SIGSTOP/SIGCONT， GDB调试）；
After the above optimization, iowait dropped significantly, but the number of zombie processes is still increasing. First, locate the parent process of the zombie process. Use pstree -aps XXX to print out the call tree of the zombie process and find that the parent process is the app process.
Check the app code to see if the end of the child process is handled correctly (whether wait()/waitpid() is called, whether there is a SIGCHILD signal processing function registered, etc.).
When encountering an increase in iowait, first use tools such as dstat pidstat to confirm whether there is a disk I/O problem, and then find out which processes are causing the I/O. Strace cannot be used When directly analyzing process calls, you can use the perf tool.
#For the zombie problem, use pstree to find the parent process, and then look at the source code to check the processing logic for the end of the child process.
CPU Performance Indicators
CPU Usage
User CPU usage, including user mode (user) and low-priority user mode (nice). If this indicator is too high, it indicates that the application is busy.
System CPU usage, the percentage of time the CPU is running in kernel mode (excluding interrupts). A high indicator indicates that the kernel is busy.
CPU usage waiting for I/O, iowait. A high indicator indicates that the I/O interaction time between the system and the hardware device is relatively long.
Soft/hard interrupt CPU usage, a high indicator indicates that a large number of interrupts occur in the system.
steal CPU / guest CPU, means The percentage of CPU occupied by the virtual machine.
Average load
Ideal load average Equal to the number of logical CPUs, it means that each CPU is fully utilized. If it is greater, it means that the system load is heavier.
Process Context Switch
Includes voluntary switching when resources cannot be obtained and involuntary switching when the system forces scheduling. Context switching itself is a core function to ensure the normal operation of Linux. Excessive switching will reduce the CPU time of the originally running process. It is consumed in saving and restoring data such as registers, kernel occupancy and virtual memory. In addition, search the public account programmer Xiaole's backstage to reply to "interview questions" and get a surprise gift package.
CPU cache hit rate
CPU cache reuse situation, the higher the hit rate The higher the performance, the better. L1/L2 is commonly used in single-core, and L3 is used in multi-core
Performance Tools
Load Average Case
First use uptime to check the average load of the system
After judging that the load has increased, use mpstat and pidstat to check each CPU separately. and per-process CPU usage. Find out which process is causing the higher load average.
Context switch case
First use vmstat to check the number of system context switches and interruptions
Then use pidstat to observe the voluntary and non-voluntary processes of the process Voluntary context switch situation
Finally observe the context switching of the thread through pidstat
Case of high process CPU usage
First use top to check the CPU usage of the system and process, locate the process
and then Use perf top to observe the process call chain and locate the specific function
Case of high system CPU usage
First use top to check the CPU usage of the system and process. Neither top/pidstat can find the process with high CPU usage
Re-examine the top output
Start with processes that have low CPU usage but are in the Running state
perf record/report found short-lived processes (execsnoop tool)
Uninterruptible and zombie process cases
First use top to observe the rise of iowait and find a large number of uninterruptible and zombie processes
strace cannot trace process system calls
perf analysis of the call chain found that the root cause comes from direct disk I/O
Soft interrupt case
topObserve the high system soft interrupt CPU usage
Check /proc/softirqs to find several soft interrupts with fast changing rates
The sar command found that they are network packets question
tcpdump finds out the type and source of network frames and determines the SYN FLOOD attack caused
According to different performance indicators Find the right tool:
Picture from: www.ctq6.cn
In the production environment Developers often do not have permission to install new tool packages and can only maximize the use of tools already installed in the system. Therefore, it is necessary to understand what indicator analysis some mainstream tools can provide.
Picture from: www.ctq6.cn
First run a few tools that support more indicators, such as top/vmstat /pidstat, based on their output, you can determine what type of performance problem it is. After locating the process, use strace/perf to analyze the calling situation for further analysis. If it is caused by soft interrupt, use /proc/softirqs
Picture from: www.ctq6.cn
##CPU optimization
Application Optimization
Compiler optimization: Enable optimization options during the compilation phase, such as gcc -O2
Algorithm optimization
#Asynchronous processing: Avoid the program from being blocked waiting for a certain resource, and improve the concurrent processing capabilities of the program. (Replace polling with event notification)
Multiple threads instead of multiple processes: Reduce context switching costs
Make good use of cache: Speed up program processing
System optimization
CPU binding: Bind the process to one/multiple CPUs to improve the CPU cache hit rate and reduce context switching caused by CPU scheduling
CPU exclusive: CPU affinity mechanism to allocate processes
Priority adjustment: Use nice to appropriately lower the priority of non-core applications
#Set resource display for the process: cgroups sets the usage limit to prevent an application from depleting system resources due to its own problems
NUMA optimization: CPU accesses local memory as much as possible
Interrupt load balancing: irpbalance , Automatically load balance the interrupt processing process to each CPU
##The difference and understanding of TPS, QPS, and system throughput
QPS(TPS)
Concurrency
##Response time
QPS(TPS)=number of concurrencies/average response time
##User request server
Internal processing of the server
The server returns to Client
##QPS is similar to TPS, but a visit to a page forms a TPS, but a page request may include multiple requests to the server and may be counted into multiple QPS
QPS (Queries Per Second) query rate per second, the number of queries a server can respond to per second.
TPS (Transactions Per Second) number of transactions per second, software test results.
System throughput, including several important parameters:
3Memory
How Linux memory works
Memory mapping
The main memory used in most computers is dynamic random access memory (DRAM). Only the kernel can directly access the physical memory. Memory. The Linux kernel provides an independent virtual address space for each process, and this address space is continuous. In this way, the process can easily access memory (virtual memory).
The interior of the virtual address space is divided into two parts: kernel space and user space. The range of the address space of processors with different word lengths is different. The 32-bit system kernel space occupies 1G and the user space occupies 3G. The kernel space and user space of 64-bit systems are both 128T, occupying the highest and lowest parts of the memory space respectively, and the middle part is undefined.
Not all virtual memory will allocate physical memory, only the actual use will. The allocated physical memory is managed through memory mapping. In order to complete memory mapping, the kernel maintains a page table for each process to record the mapping relationship between virtual addresses and physical addresses. The page table is actually stored in the CPU's memory management unit MMU, and the processor can directly find out the memory to be accessed through hardware.
When the virtual address accessed by the process cannot be found in the page table, the system will generate a page fault exception, enter the kernel space to allocate physical memory, update the process page table, and then return to the user The space recovery process runs.
MMU manages memory in units of pages, with a page size of 4KB. In order to solve the problem of too many page table entries, Linux provides the Multi-level page table and HugePage mechanisms.
Virtual memory space distribution
User space memory has five different memory segments from low to high:
Read-only segment Code and constants, etc.
Data segment Global variables, etc.
Heap Dynamically allocated memory starts from the low address and grows upward
File mapping Dynamic libraries, shared memory, etc., start from high addresses and grow downward
Stack Including local variables and function call context, etc. The size of the stack is fixed.Generally 8MB
Memory allocation and recycling
Allocation
##malloc There are two implementation methods corresponding to system calls:
brk() For small blocks of memory (<128K), allocate by moving the top position of the heap. The memory is not returned immediately after it is released, but is cached.
**mmap()**For large blocks of memory (>128K), allocate directly using memory mapping, that is, find a free area in the file mapping segment Memory allocation.
#The former cache can reduce the occurrence of page fault exceptions and improve memory access efficiency. However, because the memory is not returned to the system, frequent memory allocation/release will cause memory fragmentation when the memory is busy.
The latter is directly returned to the system when released, so a page fault exception will occur every time mmap occurs. When memory work is busy, frequent memory allocation will cause a large number of page fault exceptions, increasing the kernel management burden.
The above two calls do not actually allocate memory. These memories only enter the kernel through page fault exceptions when they are accessed for the first time, and are allocated by the kernel
Recycling
When memory is tight, the system reclaims memory in the following ways:
Recycle cache: LRU algorithm recycles the least recently used memory pages;
Recycle infrequently accessed memory: recycle infrequently used memory Write to the disk through the swap partition
Kill the process: OOM kernel protection mechanism (the greater the memory consumed by the process, the greater the oom_score, and the CPU occupied The more oom_score, the smaller it is. You can manually adjust oom_adj through /proc)
echo -16 > /proc/$(pidof XXX)/oom_adj
Copy after login
如何查看内存使用情况
free来查看整个系统的内存使用情况
top/ps来查看某个进程的内存使用情况
VIRT The virtual memory size of the process
RES Normal The size of the resident memory, that is, the actual physical memory size used by the process, excluding swap and shared memory
SHR Shared memory size, and Memory shared by other processes, loaded dynamic link libraries and program code segments
##%MEM The physical memory used by the process accounts for the total system memory Percent
#How to understand Buffer and Cache in memory?
buffer is a cache of disk data, cache is a cache of file data, they will be used in both read requests and write requests
How to use the system cache to optimize the running efficiency of the program
Cache hit rate
Cache hit Rate refers to the number of requests to obtain data directly through the cache, accounting for the percentage of all requests. The higher the hit rate, the higher the benefits brought by the cache and the better the performance of the application.
After installing the bcc package, you can monitor cache read and write hits through cachestat and cachetop.
安装pcstat后可以查看文件在内存中的缓存大小以及缓存比例
#首先安装Go
export GOPATH=~/go
export PATH=~/go/bin:$PATH
go get golang.org/x/sys/unix
go ge github.com/tobert/pcstat/pcstat
For applications, dynamic memory allocation and recycling is a core and complex logical function module. Various "accidents" will occur in the process of managing memory:
The allocated memory is not properly reclaimed, resulting in a leak
The address outside the allocated memory boundary is accessed , causing the program to exit abnormally
Allocation and recycling of memory
Virtual memory distribution from low to high isRead-only segment, data segment, heap, memory mapping segment, stackFive parts. Among them, the ones that will cause memory leaks are:
Heap: Allocated and managed by the application itself. Unless the program exits, these heap memories will not be automatically released by the system.
Memory mapping segment: including dynamic link library and shared memory, where shared memory is automatically allocated and managed by the program
Memory leaks are more harmful. Not only can the application itself not access the memory that is forgotten to be released, but the system cannot allocate it to other applications again. Memory leaks accumulate and even exhaust system memory.
Cache/buffer is a recyclable resource, usually called file page in file management
# #Synchronize dirty pages to disk through fsync in the application
Leave it to the system, and the kernel thread pdflush is responsible for refreshing these dirty pages
Data (dirty pages) that have been modified by the application and have not yet been written to the disk must be written to the disk first and then the memory can be released
The file mapping page obtained by memory mapping can also be released and re-read from the file the next time it is accessed
For the heap memory automatically allocated by the program, which is our anonymous page in memory management, although this memory cannot be released directly, Linux provides a Swap mechanism to write the infrequently accessed memory to the disk to release the memory. Just read the disk into the memory.
Swap principle
The essence of Swap is to use a piece of disk space or a local file as memory, including two processes of swapping in and swapping out:
Swap out: store the memory data temporarily unused by the process to disk and release the memory
Swap in: When the process accesses the memory again, read them from the disk into the memory
How does Linux measure whether memory resources are tight?
Direct Memory Recycling New large block memory allocation request, but insufficient remaining memory. At this time, the system will reclaim a portion of the memory;
kswapd0 The kernel thread regularly reclaims memory. In order to measure memory usage, three thresholds of pages_min, pages_low, and pages_high are defined, and memory recycling operations are performed based on them.
##Remaining memory
##pages_min < remaining memory < pages_low, the memory pressure is large, kswapd0 performs memory recycling until the remaining memory > pages_high
pages_low < remaining memory < pages_high, there is a certain pressure on the memory, but it can satisfy new memory requests
When a Node is out of memory, the system can find free resources from other Nodes or reclaim memory from local memory. Adjust via /proc/sys/vm/zone_raclaim_mode.
0 means that you can either find free resources from other Nodes or reclaim memory locally
1, 2, 4 means that only local memory is recycled, 2 means that dirty data can be returned to memory, and 4 means that memory can be recycled using Swap.
swappiness
During the actual recycling process, Linux adjusts the usage according to the /proc/sys/vm/swapiness option The positivity of Swap ranges from 0 to 100. The larger the value, the more actively using Swap, that is, the more inclined to recycle anonymous pages; the smaller the value, the more passive the use of Swap, that is, the more inclined to recycle file pages.
How to find problems with system memory quickly and accurately
Memory performance indicators
System memory index
Used memory/remaining memory
Shared memory (tmpfs implementation)
Available memory: including remaining memory and recyclable memory
Cache: page cache of disk read files, slab allocation The recyclable part in the server
Buffer: Temporary storage of original disk blocks, caching the data to be written to disk
Process memory indicators
Virtual memory: 5 most
Resident memory: the physical memory actually used by the process, excluding Swap And shared memory
Shared memory: memory shared with other processes, as well as code segments of dynamic link libraries and programs
Swap memory: Swap out the memory to disk through Swap. Follow the Linux Chinese Community
Page Missing Exception
Can be allocated directly from physical memory, secondary page fault exception
Requires disk IO intervention (such as Swap) , the main page is missing abnormally. At this time, memory access will be much slower
Memory performance tool
Find the appropriate tool based on different performance indicators :
Picture from: www.ctq6.cn
Performance indicators included in the memory analysis tool:
Picture from: www.ctq6.cn
How to quickly analyze memory performance bottlenecks
Usually run several performance tools with relatively large coverage first, such as free, top, vmstat, pidstat, etc.
First use free and top to check the overall memory usage of the system
Then use vmstat and pidstat to check for a period of time trend to determine the type of memory problem
Finally, conduct detailed analysis, such as memory allocation analysis, cache/buffer analysis, memory usage analysis of specific processes, etc.
Common optimization ideas:
It is best to disable Swap. If it must be enabled, try to reduce the value of swappiness
Reduce the dynamic allocation of memory. For example, you can use memory pools, HugePage, etc.
Try to use caches and buffers to access data. For example, use the stack to explicitly declare the memory space to store the data that needs to be cached, or use the Redis external cache component to optimize data access
cgroups and other methods to limit the process. Memory usage to ensure that system memory is not exhausted by abnormal processes
/proc/pid/oom_adj adjusts the oom_score of core applications to ensure that even if memory is tight, core applications Will not be killed by OOM
Detailed explanation of vmstat usage
The vmstat command is the most common Linux/Unix monitoring tool , can display the status value of the server at a given time interval, including the server's CPU usage, memory usage, virtual memory swap status, and IO read and write status.可以看到整个机器的CPU,内存,IO的使用情况，而不是单单看到各个进程的CPU使用率和内存使用率(使用场景不一样)。
The above is the detailed content of Summary of Linux performance all-round tuning experience. For more information, please follow other related articles on the PHP Chinese website!
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
