How to generate caffe train_list.txt using python script

The following is a python script to share with you how to generate caffe train_list.txt. It has a good reference value and I hope it will be helpful to everyone. Let’s take a look together

First give the code:

import os

path = "/home/data//"
path_exp = os.path.expanduser(path)
classes = [int(p) for p in os.listdir(path_exp)]
classes.sort()
# nrof_classes一个数据集下有多少个文件夹,就是说有多少个人,多少个类别
nrof_classes = len(classes)
count=0
files = open("train_list.txt",'w')
filets = open("test_list.txt",'w')
count_u=0
for i in range(nrof_classes):
  class_name = str(classes[i])
  count=count+1
  count_u=count_u+1
  facedir = os.path.join(path_exp, class_name)
  prefix1 = path+class_name+"/"

  if os.path.isdir(facedir):
    images = os.listdir(facedir)
    #print(images[0])
    image_paths = [(prefix1+img+" "+class_name+"\n") for img in images]
    #print(image_paths[0])
    if count < 0.8*nrof_classes:
      if len(image_paths)>4:
        test_path=[]
        for x in range(2):
          test_path.append(image_paths[0])
          del image_paths[0]
        filets.writelines(test_path)
    files.writelines(image_paths)
    #if count==2:
    #  break
    #imgae_pathses = []
    #防止图像大小为0
    #for x in image_paths:
    #  if os.path.getsize(x)>0:
    #    imgae_pathses.append(x)
    #if len(imgae_pathses)==0:
    #  os.rmdir(facedir)
files.close()
filets.close()

Some useful usage of the os module under python:

0 Rename: Files and folders are both one command:

os.rename(original_dir,new_dir)

1 File operations:

os.mknod("test.txt") Create an empty file
fp = open("test.txt",w) Directly open a file, if the file does not exist Create a file

About open mode:

w Open for writing,
a Open for append mode (start from EOF, create a new file if necessary)
r Open for reading and writing Mode open
w Open in read-write mode (see w)
a Open in read-write mode (see a)
rb Open in binary read mode
wb Open in binary write mode (see w)
ab Open in binary append mode (see a)
rb Open in binary read-write mode (see r)
wb Open in binary read-write mode (see w)
ab Open in binary read-write mode Open (see a)

fp.read([size])                                                                                                                                                                                                                                                                                       If size is defined, it is possible to return only part of a line

fp.readlines([size])                                                                                         #                                                                                                                                                       but only a part of a line may be returned

fp.readlines([size]) In fact, it is implemented internally by calling readline() in a loop. If the size parameter is provided, size represents the total length of the read content, which means that only a part of the file may be read.

FP.Write (STR)#Write STR into the file, write () will not add a change line after STR

FP.Writelines (SEQ)# All contents are written to the file (multiple lines are written at once). This function also just writes faithfully, without adding anything after each line.

fp.close()                       #Close the file. Python will automatically close a file after it is no longer used. However, this function is not guaranteed. It is best to develop the habit of closing it yourself. If a file is operated on after it is closed, a ValueError will be generated.

fp.flush() Returns a long Type of "file tag"

fp.isatty()                                                                                                               #                  Returns the current position of the file operation marker , taking the beginning of the file as the origin

fp.next()                                                                                                                                                                                                  because When a file is used in a statement such as for...in file, the next() function is called to implement traversal.

fp.seek(offset[,whence]) #将文件打操作标记移到offset的位置。这个offset一般是相对于文件的开头来计算的，一般为正数。但如果提供了whence参数就不一定了，whence可以为0表示从头开始计算，1表示以当前位置为原点计算。2表示以文件末尾为原点进行计算。需要注意，如果文件以a或a+的模式打开，每次进行写操作时，文件操作标记会自动返回到文件末尾。

fp.truncate([size]) #把文件裁成规定的大小，默认的是裁到当前文件操作标记的位置。如果size比文件的大小还要大，依据系统的不同可能是不改变文件，也可能是用0把文件补到相应的大小，也可能是以一些随机的内容加上去。

也可以直接打开式的新建文件：

  with open('path0.txt','ab') as f:
    for d in arr:
      np.savetxt(f,d,fmt='%5f')

path0.txt将直接在python脚本所在目录新建

arr是shape是height x width x channel的numpy数组，注意，这里这么写是因为numpy一次只能够保存一个height x width维度的数据

2 新建、删除文件夹

新建：os.makedirs()

比如windows下新建E:\\dir\\subdir\\

os.makedirs('E:\\dir\\subdir\\') 或者 os.makedirs('E:/dir/subdir/')

ubuntu下新建就是os.makedirs('/home/dir/subdir/')

复制文件：

shutil.copyfile("oldfile","newfile")   oldfile和newfile都只能是文件
shutil.copy("oldfile","newfile")oldfile只能是文件夹，newfile可以是文件，也可以是目标目录

复制文件夹：

shutil.copytree("olddir","newdir")olddir和newdir都只能是目录，且newdir必须不存在

重命名文件（目录）

os.rename("oldname","newname")   文件或目录都是使用这条命令

移动文件（目录）

shutil.move("oldpos","newpos")  

删除文件

os.remove("file")

删除目录

os.rmdir("dir")只能删除空目录
shutil.rmtree("dir")空目录、有内容的目录都可以删

转换目录

os.chdir("path")   换路径

获取文件大小：os.path.getsize（filename）

3   需要注意的是，在ubuntu下有可能路径包括~，因此需要展开，展开使用：

path_exp = os.path.expanduser(nam)

4  还有一种情况，路径相连，需要将一级一级的目录连接起来，os会自动处理层级目录之间的/,比如，将目录/home 和 数据 1之间连接起来：

os.path.join('/home', str(1))，

后面也可以为文件名，是一样的用法

5  路径存在:用于判断目录是否存在

a = os.path.exists(dir)

存在返回True，否则False

6   获取目录列表，就是给出的目录下的文件或目录列表：

classes = os.listdir(path_exp)

获得的是path_exp下的所有文件的列表，包含目录和文件，

os.path.dirname(path) #返回文件路径，或者是给定路径去掉最后一个目录后的路径，

还有一种使用方式是用在文件中：

os.path.dirname(__file__)用于获取所在文件的路径

7  运行shell命令: os.system()：

ubuntu下：os.system(‘pwd')

Python脚本工作的目录路径: os.getcwd()

读取和设置环境变量:os.getenv() 与os.putenv()

给出当前平台使用的行终止符:os.linesep Windows使用'\r\n'，Linux使用'\n'而Mac使用'\r'

指示你正在使用的平台：os.name 对于Windows，它是'nt'，而对于Linux/Unix用户，它是'posix'

创建多级目录：os.makedirs（r“c：\python\test”）

创建单个目录：os.mkdir（“test”）

获取文件属性：os.stat（file）

修改文件权限与时间戳：os.chmod（file）

终止当前进程：os.exit（）

更全面的os.path用法：

os.path.abspath(path) #返回绝对路径
os.path.basename(path) #返回文件名

os.path.basename('E:a\b\c.jpg')

'c.jpg'

os.path.commonprefix(list) #Return the longest path common to all paths in the list (multiple paths).
os.path.dirname(path) #Return the file path
os.path.exists(path) #Return True if the path exists, False if the path is damaged
os.path.lexists #Return if the path exists True, True will also be returned if the path is damaged
os.path.expanduser(path) #Convert the "~" and "~user" contained in the path into the user directory
os.path.expandvars(path) #According to the environment The value of the variable replaces the "$name" and "${name}" contained in the path
os.path.getatime(path) #Return the time when this path was last entered.
os.path.getmtime(path) #Return the time of the last modification under this path.
os.path.getctime(path) #Return the size of the path
os.path.getsize(path) #Return the file size, if the file does not exist, return an error
os.path.isabs(path) #Judge whether it is an absolute path
os.path.isfile(path) #Judge whether the path is a file
os.path.isdir(path) #Judge whether the path is a directory
os .path.islink(path) #Determine whether the path is a link
os.path.ismount(path) #Determine whether the path is a mount point ()
os.path.join(path1[, path2 [, ...]]) #Combine directory and file names into one path
os.path.normcase(path) #Convert the case and slashes of path
os.path.normpath(path ) #Standard path string form
os.path.realpath(path) #Return the real path of path
os.path.relpath(path[, start]) #Calculate the relative path from start
os .path.samefile(path1, path2) #Determine whether the directories or files are the same
os.path.sameopenfile(fp1, fp2) #Determine whether fp1 and fp2 point to the same file
os.path.samestat(stat1, stat2 ) #Determine whether stat tuple stat1 and stat2 point to the same file

os.path.split(path) #Split the path into dirname and basename, and return a tuple:
os.path.split('E:a\b\c.jpg')

'E:\\a\\b','c.jpg'
os .path.splitdrive(path) #Generally used under windows, returns a tuple of drive name and path
os.path.splitext(path) #Splits the path, returns a tuple of path name and file extension

os.path.split('E:a\b\c.jpg')

'E:\\a\\b\\c ','.jpg'

os.path.splitunc(path) #Split the path into loading points and files
os.path.walk(path, visit, arg) #Traverse the path , call the visit function when entering each directory. The visit function must have

3 parameters (arg, dirname, names). dirname represents the directory name of the current directory, and names represents all # in the current directory. ##File name, args is the third parameter of walk

os.path.supports_unicode_filenames #Set whether to support unicode path names

The above is the detailed content of How to generate caffe train_list.txt using python script. For more information, please follow other related articles on the PHP Chinese website!