How to Use Python to Find the Zipf Distribution of a Text File

This tutorial demonstrates how to use Python to process the statistical concept of Zipf's law and demonstrates the efficiency of Python's reading and sorting large text files when processing the law.

You may be wondering what the term Zipf distribution means. To understand this term, we first need to define the Zipf law . Don't worry, I'll try to simplify the instructions.

Zipf's Law

Zipf's law simply means: in a large natural language corpus, the most frequently occurring words appear about twice as frequently as the second frequent words, three times as the third frequent words, four times as the fourth frequent words, and so on.

Let's look at an example. If you look at the Brown corpus in American English, you will notice that the word that appears most frequently is "the" (appears 69,971 times). The second frequently appeared word "of" appeared 36,411 times.

"the" accounts for about 7% of the Brown corpus vocabulary (69,971 out of more than 1 million words). And "of" accounts for about 3.6% of the corpus (about half of "the"). Therefore, we can see that Zipf's law applies to this case.

Therefore, Zipf's law tries to tell us that a small number of items usually occupy most of the activity we observe. For example, a few diseases (cancer, cardiovascular disease) account for the majority of deaths. This also applies to words that occupy most of the frequency of words in literary works, as well as many other examples in our lives.

Data preparation

Before continuing, let me introduce you to the experimental data we will use in this tutorial. Our data comes from the Dracula text version available on Project Gutenberg's website.

Program Construction

After downloading the data from the previous section, let's start building a Python script that will look for the Zipf distribution of the data in

dracula.txt.

The first step is to use the

function to read the file. read()

Since we are looking for patterns (i.e. words), regular expressions come in handy. We will use Python's

to remove any words that are not words in the traditional sense. For example, it won't match robotics_89, 40_pie_40, and BIGmango. "BIGmango" does not match because it starts with multiple capital letters. b[A-Za-z][a-z]{2,9}b

This regular expression basically tells us to find all words that start with letters (caps or lowercase) followed by at least 2 characters and no more than 9 characters. In other words, the word sizes contained in the output range from 3 to 10 characters.

In Python, this can be expressed as:

words = re.findall(r'(\b[A-Za-z][a-z]{2,9}\b)', file_to_string)

Now, we can run a loop to calculate the frequency of each word occurrence:

for word in words:
    count = frequency.get(word,0)
    frequency[word] = count + 1

Here, if the word is not found in the word list, we use the

function to traverse the values ​​so that we can also track the index positions of different words instead of throwing a for loop error. enumerate()

The frequency of the most frequent words is then divided by the frequency of the other words to calculate their ratio. This allows us to see how well Zipf's law is followed.

Integrate all content

After understanding the different building blocks of a program, let's see how they are put together:

words = re.findall(r'(\b[A-Za-z][a-z]{2,9}\b)', file_to_string)

Here I will display the first ten words returned by the program and their frequency:

for word in words:
    count = frequency.get(word,0)
    frequency[word] = count + 1

From this Zipf distribution, we can verify Zipf's law, that is, some words (high frequency words) represent most words, such as "the", "and", "that", "was" and "for".

Conclusion

In this tutorial, we see how Python simplifies the processing of statistical concepts such as Zipf's law. Especially when dealing with large text files, Python is very convenient, and if we manually look up Zipf distributions, it takes a lot of time and effort. As we can see, we are able to quickly load, parse and find Zipf distributions of files of size 28 MB. And because of Python's dictionary, sorting output is also simple.

This article has been updated and contains contributions from Monty Shokeen. Monty is a full stack developer who also loves writing tutorials and learning new JavaScript libraries.

The above is the detailed content of How to Use Python to Find the Zipf Distribution of a Text File. For more information, please follow other related articles on the PHP Chinese website!