This article rewrites the core algorithm through the C extension interface to optimize the C/Python code statistics tool (CPLineCounter) implemented in the previous series of articles, and compares it with common statistical tools on the Internet. Actual measurements show that CPLineCounter is superior to other similar statistical tools in terms of statistical accuracy and performance. Taking tens of millions of lines of code as an example to evaluate performance, when CPLineCounter is run in Cpython and Pypy environments, it is 14.5 times and 29 times faster than the foreign statistical tool cloc1.64, respectively, and 1.8 times and 3.6 times faster than the domestic SourceCounter3.4.
Run test environment This article is based on the Windows system platform and runs and tests the code examples involved. The platform information is as follows:
>>> import sys, platform
>>> print '%s %s, Python %s' %(platform.system(), platform.release(), platform.python_version())
Windows XP, Python 2.7.11
>>> sys.version
'2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:32:19) [MSC v.1500 32 bit (Intel)]'
Copy after login
Note that there are syntax differences between different versions of Python, so some code examples in the article need to be slightly modified in order to run in a lower version of Python environment. 1. Code implementation and optimization To avoid fragmentation, this section will give the complete implementation code. Note that some variable or function definitions in this section are slightly different from the implementations in previous series of articles, please pay attention to the screening. 1.1 Code Implementation First, define two lists to store statistical results:
Among them, rawCountInfo stores the rough total number of file lines. The list elements are the total number of file lines, code lines, comment lines and blank lines, as well as the number of files. detailCountInfo stores detailed statistical information, including the line count information and file name of a single file, and the sum of the line counts of all files.
The specific implementation code will be given below. To avoid pasting large sections of code, describe functions briefly.
def CalcLinesCh(line, isBlockComment):
lineType, lineLen = 0, len(line)
if not lineLen:
return lineType
line = line + '\n' #添加一个字符防止iChar+1时越界
iChar, isLineComment = 0, False
while iChar < lineLen:
if line[iChar] == ' ' or line[iChar] == '\t': #空白字符
iChar += 1; continue
elif line[iChar] == '/' and line[iChar+1] == '/': #行注释
isLineComment = True
lineType |= 2; iChar += 1 #跳过'/'
elif line[iChar] == '/' and line[iChar+1] == '*': #块注释开始符
isBlockComment[0] = True
lineType |= 2; iChar += 1
elif line[iChar] == '*' and line[iChar+1] == '/': #块注释结束符
isBlockComment[0] = False
lineType |= 2; iChar += 1
else:
if isLineComment or isBlockComment[0]:
lineType |= 2
else:
lineType |= 1
iChar += 1
return lineType #Bitmap：0空行，1代码，2注释，3代码和注释
def CalcLinesPy(line, isBlockComment):
#isBlockComment[single quotes, double quotes]
lineType, lineLen = 0, len(line)
if not lineLen:
return lineType
line = line + '\n\n' #添加两个字符防止iChar+2时越界
iChar, isLineComment = 0, False
while iChar < lineLen:
if line[iChar] == ' ' or line[iChar] == '\t': #空白字符
iChar += 1; continue
elif line[iChar] == '#': #行注释
isLineComment = True
lineType |= 2
elif line[iChar:iChar+3] == "'''": #单引号块注释
if isBlockComment[0] or isBlockComment[1]:
isBlockComment[0] = False
else:
isBlockComment[0] = True
lineType |= 2; iChar += 2
elif line[iChar:iChar+3] == '"""': #双引号块注释
if isBlockComment[0] or isBlockComment[1]:
isBlockComment[1] = False
else:
isBlockComment[1] = True
lineType |= 2; iChar += 2
else:
if isLineComment or isBlockComment[0] or isBlockComment[1]:
lineType |= 2
else:
lineType |= 1
iChar += 1
return lineType #Bitmap：0空行，1代码，2注释，3代码和注释
Copy after login
The CalcLinesCh() and CalcLinesPy() functions determine file line attributes based on C and Python syntax respectively, and count by code, comment or blank line respectively.
from ctypes import c_uint, c_ubyte, CDLL
CFuncObj = None
def LoadCExtLib():
try:
global CFuncObj
CFuncObj = CDLL('CalcLines.dll')
except Exception: #不捕获系统退出(SystemExit)和键盘中断(KeyboardInterrupt)异常
pass
def CalcLines(fileType, line, isBlockComment):
try:
#不可将CDLL('CalcLines.dll')放于本函数内，否则可能严重拖慢执行速度
bCmmtArr = (c_ubyte * len(isBlockComment))(*isBlockComment)
CFuncObj.CalcLinesCh.restype = c_uint
if fileType is 'ch': #is(同一性运算符)判断对象标识(id)是否相同，较==更快
lineType = CFuncObj.CalcLinesCh(line, bCmmtArr)
else:
lineType = CFuncObj.CalcLinesPy(line, bCmmtArr)
isBlockComment[0] = True if bCmmtArr[0] else False
isBlockComment[1] = True if bCmmtArr[1] else False
#不能采用以下写法，否则本函数返回后isBlockComment列表内容仍为原值
#isBlockComment = [True if i else False for i in bCmmtArr]
except Exception, e:
#print e
if fileType is 'ch':
lineType = CalcLinesCh(line, isBlockComment)
else:
lineType = CalcLinesPy(line, isBlockComment)
return lineType
Copy after login
In order to improve the running speed, the author rewrote the CalcLinesCh() and CalcLinesPy() functions in C language and compiled them into a dynamic link library. For details on the implementation and use of the C language versions of these two functions, see Section 1.2. The LoadCExtLib() and CalcLines() functions are designed to load the dynamic link library and execute the corresponding C version statistical function. If the loading fails, the slower Python version statistical function will be executed.
The above code runs in the CPython environment, and the C dynamic library is loaded and executed through the built-in ctypes module of Python2.5 and subsequent versions. As an external function library for Python, this module provides data types compatible with the C language and allows calling functions in DLLs or shared libraries. Therefore, ctypes is often used to wrap external dynamic libraries in pure Python code.
If the code runs in the Pypy environment, you need to use the cffi interface to call the C program:
from cffi import FFI
CFuncObj, ffiBuilder = None, FFI()
def LoadCExtLib():
try:
global CFuncObj
ffiBuilder.cdef('''
unsigned int CalcLinesCh(char *line, unsigned char isBlockComment[2]);
unsigned int CalcLinesPy(char *line, unsigned char isBlockComment[2]);
''')
CFuncObj = ffiBuilder.dlopen('CalcLines.dll')
except Exception: #不捕获系统退出(SystemExit)和键盘中断(KeyboardInterrupt)异常
pass
def CalcLines(fileType, line, isBlockComment):
try:
bCmmtArr = ffiBuilder.new('unsigned char[2]', isBlockComment)
if fileType is 'ch': #is(同一性运算符)判断对象标识(id)是否相同，较==更快
lineType = CFuncObj.CalcLinesCh(line, bCmmtArr)
else:
lineType = CFuncObj.CalcLinesPy(line, bCmmtArr)
isBlockComment[0] = True if bCmmtArr[0] else False
isBlockComment[1] = True if bCmmtArr[1] else False
#不能采用以下写法，否则本函数返回后isBlockComment列表内容仍为原值
#isBlockComment = [True if i else False for i in bCmmtArr]
except Exception, e:
#print e
if fileType is 'ch':
lineType = CalcLinesCh(line, isBlockComment)
else:
lineType = CalcLinesPy(line, isBlockComment)
return lineType
Copy after login
cffi usage is similar to ctypes, but allows direct loading of C files to call functions inside (automatically compiled during interpretation). For the sake of unification, the method of loading dynamic libraries is still used here.
Pay attention to the "%d files processed..." progress prompt. Because it is impossible to know whether the output is redirected to the file through the command line (sys.stdout remains unchanged, sys.argv does not contain ">out"), the progress prompt writes a newline to the output file. Assuming that the number of code files is N, the output file will contain N lines of progress information. Currently, you can only use the feature that redirection only affects standard output by default to output progress information from standard error to the console; at the same time, add the -o option to explicitly distinguish between standard output and file writing, reducing the user's redirection risk. possibility.
In addition, when calling the CalcLines() function, the strip() method is used to remove blank characters at the beginning and end of the file line. Therefore, there is no need for line terminator judgment branches in CalcLinesCh() and CalcLinesPy().
SORT_ORDER = (lambda x:x[0], False)
def SetSortArg(sortArg=None):
global SORT_ORDER
if not sortArg:
return
if any(s in sortArg for s in ('file', '0')): #条件宽松些
#if sortArg in ('rfile', 'file', 'r0', '0'):
keyFunc = lambda x:x[1][0]
elif any(s in sortArg for s in ('code', '1')):
keyFunc = lambda x:x[1][1]
elif any(s in sortArg for s in ('cmmt', '2')):
keyFunc = lambda x:x[1][2]
elif any(s in sortArg for s in ('blan', '3')):
keyFunc = lambda x:x[1][3]
elif any(s in sortArg for s in ('ctpr', '4')):
keyFunc = lambda x:x[1][4]
elif any(s in sortArg for s in ('name', '5')):
keyFunc = lambda x:x[0]
else: #因argparse内已限制排序参数范围，此处也可用assert
print >>sys.stderr, 'Unsupported sort order(%s)!' %sortArg
return
isReverse = sortArg[0]=='r' #False:升序(ascending); True:降序(decending)
SORT_ORDER = (keyFunc, isReverse)
def ReportCounterInfo(isRawReport=True, stream=sys.stdout):
#代码注释率 = 注释行 / (注释行+有效代码行)
print >>stream, 'FileLines CodeLines CommentLines BlankLines CommentPercent %s'\
%(not isRawReport and 'FileName' or '')
if isRawReport:
print >>stream, '%-11d%-11d%-14d%-12d%-16.2f<Total:%d Code Files>' %(rawCountInfo[0],\
rawCountInfo[1], rawCountInfo[2], rawCountInfo[3], \
SafeDiv(rawCountInfo[2], rawCountInfo[2]+rawCountInfo[1]), rawCountInfo[4])
return
total = [0, 0, 0, 0]
#对detailCountInfo排序。缺省按第一列元素(文件名)升序排序，以提高输出可读性。
detailCountInfo.sort(key=SORT_ORDER[0], reverse=SORT_ORDER[1])
for item in detailCountInfo:
print >>stream, '%-11d%-11d%-14d%-12d%-16.2f%s' %(item[1][0], item[1][1], item[1][2], \
item[1][3], item[1][4], item[0])
total[0] += item[1][0]; total[1] += item[1][1]
total[2] += item[1][2]; total[3] += item[1][3]
print >>stream, '-' * 90 #输出90个负号(minus)或连字号(hyphen)
print >>stream, '%-11d%-11d%-14d%-12d%-16.2f<Total:%d Code Files>' \
%(total[0], total[1], total[2], total[3], \
SafeDiv(total[2], total[2]+total[1]), len(detailCountInfo))
Copy after login
ReportCounterInfo() outputs statistical reports. Note that before the detailed report is output, the output content will be sorted according to the specified sorting rules. Additionally, the terminology for blank lines has been changed from EmptyLines to BlankLines. The former means that the line does not contain any other characters except the line terminator, and the latter means that the line only contains whitespace characters (spaces, tabs, line terminators, etc.).
To support counting multiple directories and/or files at the same time, use ParseTargetList() to parse the directory-file mixed list, and store its elements in the directory and file lists respectively:
def ParseTargetList(targetList):
fileList, dirList = [], []
if targetList == []:
targetList.append(os.getcwd())
for item in targetList:
if os.path.isfile(item):
fileList.append(os.path.abspath(item))
elif os.path.isdir(item):
dirList.append(os.path.abspath(item))
else:
print >>sys.stderr, "'%s' is neither a file nor a directory!" %item
return [fileList, dirList]
Copy after login
The LineCounter() function performs statistics based on directory and file lists:
def CountDir(dirList, isKeep=False, isRawReport=True, isShortName=False):
for dir in dirList:
if isKeep:
for file in os.listdir(dir):
CountFileLines(os.path.join(dir, file), isRawReport, isShortName)
else:
for root, dirs, files in os.walk(dir):
for file in files:
CountFileLines(os.path.join(root, file), isRawReport, isShortName)
def CountFile(fileList, isRawReport=True, isShortName=False):
for file in fileList:
CountFileLines(file, isRawReport, isShortName)
def LineCounter(isKeep=False, isRawReport=True, isShortName=False, targetList=[]):
fileList, dirList = ParseTargetList(targetList)
if fileList != []:
CountFile(fileList, isRawReport, isShortName)
if dirList != []:
CountDir(dirList, isKeep, isRawReport, isShortName)
Copy after login
Then, add command line parsing processing:
import argparse
def ParseCmdArgs(argv=sys.argv):
parser = argparse.ArgumentParser(usage='%(prog)s [options] target',
description='Count lines in code files.')
parser.add_argument('target', nargs='*',
help='space-separated list of directories AND/OR files')
parser.add_argument('-k', '--keep', action='store_true',
help='do not walk down subdirectories')
parser.add_argument('-d', '--detail', action='store_true',
help='report counting result in detail')
parser.add_argument('-b', '--basename', action='store_true',
help='do not show file\'s full path')
## sortWords = ['0', '1', '2', '3', '4', '5', 'file', 'code', 'cmmt', 'blan', 'ctpr', 'name']
## parser.add_argument('-s', '--sort',
## choices=[x+y for x in ['','r'] for y in sortWords],
## help='sort order: {0,1,2,3,4,5} or {file,code,cmmt,blan,ctpr,name},' \
## "prefix 'r' means sorting in reverse order")
parser.add_argument('-s', '--sort',
help='sort order: {0,1,2,3,4,5} or {file,code,cmmt,blan,ctpr,name}, ' \
"prefix 'r' means sorting in reverse order")
parser.add_argument('-o', '--out',
help='save counting result in OUT')
parser.add_argument('-c', '--cache', action='store_true',
help='use cache to count faster(unreliable when files are modified)')
parser.add_argument('-v', '--version', action='version',
version='%(prog)s 3.0 by xywang')
args = parser.parse_args()
return (args.keep, args.detail, args.basename, args.sort, args.out, args.cache, args.target)
Copy after login
Note the -s option added to the ParseCmdArgs() function. This option specifies how the output is sorted, and the r prefix specifies ascending or descending order. For example, -s 0 or -s file indicates that the output is sorted in ascending order by the number of file lines, and -s r0 or -s rfile indicates that the output is sorted in descending order by the number of file lines.
The -c cache option is most useful when changing the output collation. To support this option, use the Json module to persist statistical reports:
CACHE_FILE = 'Counter.dump'
CACHE_DUMPER, CACHE_GEN = None, None
from json import dump, JSONDecoder
def CounterDump(data):
global CACHE_DUMPER
if CACHE_DUMPER == None:
CACHE_DUMPER = open(CACHE_FILE, 'w')
dump(data, CACHE_DUMPER)
def ParseJson(jsonData):
endPos = 0
while True:
jsonData = jsonData[endPos:].lstrip()
try:
pyObj, endPos = JSONDecoder().raw_decode(jsonData)
yield pyObj
except ValueError:
break
def CounterLoad():
global CACHE_GEN
if CACHE_GEN == None:
CACHE_GEN = ParseJson(open(CACHE_FILE, 'r').read())
try:
return next(CACHE_GEN)
except StopIteration, e:
return []
def shouldUseCache(keep, detail, basename, cache, target):
if not cache: #未指定启用缓存
return False
try:
(_keep, _detail, _basename, _target) = CounterLoad()
except (IOError, EOFError, ValueError): #缓存文件不存在或内容为空或不合法
return False
if keep == _keep and detail == _detail and basename == _basename \
and sorted(target) == sorted(_target):
return True
else:
return False
其间，作者还尝试其他C扩展工具，如PyInline。在http://pyinline.sourceforge.net/下载压缩包，解压后拷贝目录PyInline-0.03至Lib\site-packages下。在命令提示符窗口中进入该目录，执行python setup.py install安装PyInline
执行示例时提示BuildError: error: Unable to find vcvarsall.bat。查阅网络资料，作者下载Microsoft Visual C++ Compiler for Python 2.7并安装。然而，实践后发现PyInline非常难用，于是作罢。
由于对MinGW编译效果存疑，作者最终决定安装VS2008 Express Edition。之所以选择2008版本，是考虑到CPython2.7的Windows版本基于VS2008的运行时(runtime)库。安装后，在C:\Program Files\Microsoft Visual Studio 9.0\VC\bin目录可找到cl.exe(编译器)和link.exe(链接器)。按照网络教程设置环境变量后，即可在Visual Studio 2008 Command Prompt命令提示符中编译和链接程序。输入cl /help或cl -help可查看编译器选项说明。
将CalcLines.c编译为动态链接库前，还需要对函数头添加_declspec(dllexport)，以指明这是从dll导出的函数： _declspec(dllexport) unsigned int CalcLinesCh(char *line, unsigned char isBlockComment[2]) {...
_declspec(dllexport) unsigned int CalcLinesPy(char *line, unsigned char isBlockComment[2]) {... 否则Python程序加载动态库后，会提示找不到相应的C函数。
Finally, if you only count the number of lines of code, you can use the following shell command on Linux or Mac: find ./codeDir -name "*.c" -or -name "*.h" | xargs wc -l #Total number of lines except blank lines
find ./codeDir -name "*.c" -or -name "*.h" | xargs wc -l #The number of lines and sum of each file
The above is the entire content of this article. I hope it will be helpful to everyone’s study. I also hope that everyone will support Script Home.
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
经
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
