


FastDFS plus Redis implement custom file names to store massive files
FastDFS is very suitable for storing a large number of small files. Unfortunately, it does not support custom file names. The file name is a file_id generated based on the storage location after successful storage. Many application scenarios have to use custom file names. Without modifying the source code, you can add a database to the storage client fdfs_client to store the mapping relationship between the custom file name and the file_id of fastdfs to indirectly implement the custom file. For name access and access, here we choose reids. By the way, Taobao also has a file storage system TFS similar to FastDFS. For custom file names, it uses mysql to store mapping relationships. I think mysql itself is a bottleneck under high concurrent access, so it is used in this solution. redis.
Preparation work:
fastdfs environment installation... slightly... (official: https://code.google.com/p/fastdfs/)
redis environment installation... slightly... (official: http://redis.io/)
is implemented in python, so you need to install the python client of fastdfs (download: https://fastdfs.googlecode.com/files/fdfs_client-py-1.2.6.tar.gz)
Python’s redis client, go to https://pypi.python.org/pypi/redis to download
# -*- coding: utf-8 -*- import setting from fdfs_client.client import * from fdfs_client.exceptions import * from fdfs_client.connection import * import redis import time import logging import random logging.basicConfig(format='[%(levelname)s]: %(message)s', level=logging.DEBUG) logger = logging.getLogger(__name__) logger.setLevel(logging.DEBUG) class RedisError(Exception): def __init__(self, value): self.value = value def __str__(self): return repr(self.value) class fastdfsClient(Fdfs_client): def __init__(self): self.tracker_pool = ConnectionPool(**setting.fdfs_tracker) self.timeout = setting.fdfs_tracker['timeout'] return None def __del__(self): try: self.pool.destroy() self.pool = None except: pass class fastdfs(object): def __init__(self): ''' conf_file:配置文件 ''' self.fdfs_client = fastdfsClient() self.fdfs_redis = [] for i in setting.fdfs_redis_dbs: self.fdfs_redis.append(redis.Redis(host=i[0], port=i[1], db=i[2])) def store_by_buffer(self,buf,filename=None,file_ext_name = None): ''' buffer存储文件 参数: filename:自定义文件名,如果不指定,将远程file_id作为文件名 file_ext_name:文件扩展名(可选),如果不指定,将根据自定义文件名智能判断 返回值: { 'group':组名, 'file_id':不含组名的文件ID, 'size':文件尺寸, 'upload_time':上传时间 } ''' if filename and random.choice(self.fdfs_redis).exists(filename): logger.info('File(%s) exists.'%filename) return random.choice(self.fdfs_redis).hgetall(filename) t1 = time.time() # try: ret_dict = self.fdfs_client.upload_by_buffer(buf,file_ext_name) # except Exception,e: # logger.error('Error occurred while uploading: %s'%e.message) # return None t2 = time.time() logger.info('Upload file(%s) by buffer, time consume: %fs' % (filename,(t2 - t1))) for key in ret_dict: logger.debug('[+] %s : %s' % (key, ret_dict[key])) stored_filename = ret_dict['Remote file_id'] stored_filename_without_group = stored_filename[stored_filename.index('/')+1:] if not filename: filename =stored_filename_without_group vmp = {'group':ret_dict['Group name'],'file_id':stored_filename_without_group,'size':ret_dict['Uploaded size'],'upload_time':int(time.time()*1000)} try: for i in self.fdfs_redis: if not i.hmset(filename,vmp): raise RedisError('Save Failure') logger.info('Store file(%s) by buffer successful' % filename) except Exception,e: logger.error('Save info to Redis failure. rollback...') try: ret_dict = self.fdfs_client.delete_file(stored_filename) except Exception,e: logger.error('Error occurred while deleting: %s'%e.message) return None return vmp def remove(self,filename): ''' 删除文件, filename是用户自定义文件名 return True|False ''' fileinfo = random.choice(self.fdfs_redis).hgetall(filename) stored_filename = '%s/%s'%(fileinfo['group'],fileinfo['file_id']) try: ret_dict = self.fdfs_client.delete_file(stored_filename) logger.info('Remove stored file successful') except Exception,e: logger.error('Error occurred while deleting: %s'%e.message) return False for i in self.fdfs_redis: if not i.delete(filename): logger.error('Remove fileinfo in redis failure') logger.info('%s removed.'%filename) return True def download(self,filename): ''' 下载文件 返回二进制 ''' finfo = self.getInfo(filename) if finfo: ret = self.fdfs_client.download_to_buffer('%s/%s'%(finfo['group'],finfo['file_id'])) return ret['Content'] else: logger.debug('%s is not exists'%filename) return None def list(self,pattern='*'): ''' 列出文件列表 ''' return random.choice(self.fdfs_redis).keys(pattern) def getInfo(self,filename): ''' 获得文件信息 return:{ 'group':组名, 'file_id':不含组名的文件ID, 'size':文件尺寸, 'upload_time':上传时间 } ''' return random.choice(self.fdfs_redis).hgetall(filename)
Configuration:
# -*- coding: utf-8 -*- #fastdfs tracker, multiple tracker supported fdfs_tracker = { 'host_tuple':('192.168.2.233','192.168.2.234'), 'port':22122, 'timeout':30, 'name':'Tracker Pool' } #fastdfs meta db, multiple redisdb supported fdfs_redis_dbs = ( ('192.168.2.233',6379,0), ('192.168.2.233',6379,1) )

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

This tutorial demonstrates how to use Python to process the statistical concept of Zipf's law and demonstrates the efficiency of Python's reading and sorting large text files when processing the law. You may be wondering what the term Zipf distribution means. To understand this term, we first need to define Zipf's law. Don't worry, I'll try to simplify the instructions. Zipf's Law Zipf's law simply means: in a large natural language corpus, the most frequently occurring words appear about twice as frequently as the second frequent words, three times as the third frequent words, four times as the fourth frequent words, and so on. Let's look at an example. If you look at the Brown corpus in American English, you will notice that the most frequent word is "th

This article explains how to use Beautiful Soup, a Python library, to parse HTML. It details common methods like find(), find_all(), select(), and get_text() for data extraction, handling of diverse HTML structures and errors, and alternatives (Sel

Dealing with noisy images is a common problem, especially with mobile phone or low-resolution camera photos. This tutorial explores image filtering techniques in Python using OpenCV to tackle this issue. Image Filtering: A Powerful Tool Image filter

This article compares TensorFlow and PyTorch for deep learning. It details the steps involved: data preparation, model building, training, evaluation, and deployment. Key differences between the frameworks, particularly regarding computational grap

Python, a favorite for data science and processing, offers a rich ecosystem for high-performance computing. However, parallel programming in Python presents unique challenges. This tutorial explores these challenges, focusing on the Global Interprete

This tutorial demonstrates creating a custom pipeline data structure in Python 3, leveraging classes and operator overloading for enhanced functionality. The pipeline's flexibility lies in its ability to apply a series of functions to a data set, ge

Serialization and deserialization of Python objects are key aspects of any non-trivial program. If you save something to a Python file, you do object serialization and deserialization if you read the configuration file, or if you respond to an HTTP request. In a sense, serialization and deserialization are the most boring things in the world. Who cares about all these formats and protocols? You want to persist or stream some Python objects and retrieve them in full at a later time. This is a great way to see the world on a conceptual level. However, on a practical level, the serialization scheme, format or protocol you choose may determine the speed, security, freedom of maintenance status, and other aspects of the program

Python's statistics module provides powerful data statistical analysis capabilities to help us quickly understand the overall characteristics of data, such as biostatistics and business analysis. Instead of looking at data points one by one, just look at statistics such as mean or variance to discover trends and features in the original data that may be ignored, and compare large datasets more easily and effectively. This tutorial will explain how to calculate the mean and measure the degree of dispersion of the dataset. Unless otherwise stated, all functions in this module support the calculation of the mean() function instead of simply summing the average. Floating point numbers can also be used. import random import statistics from fracti
