Chinese character frequency list - News articles

I think a lot of those studying Mandarin Chinese have sooner or later started to wonder how many characters one really needs in order to normally function in a Chinese-only world or what for instance the most frequent 500 Chinese characters are. I personally have heard a lot of numbers and saw several Chinese character frequency lists, but often didn't understand why this or that character made it to the top 500 or why the list said I needed this or that number of characters to read something when I had the feeling the number was either overstated or understated so I decided to try to do a little study on my own. 

I tried to analyze how many characters and words are approximately necessary to read news in Mandarin. I chose four sections of Taiwanese news - politics, international, society and finance, all written in traditional Chinese characters during a 6 week sample period.

If I'm correct, the field of computational linguistics deals with projects of this kind and I'm sure that there are several teams of experts at linguistic departments worldwide that must have done similar researches using much more sophisticated methods than I have and after the amount of effort it took me to analyze these few articles, I have a lot of respect for what they do. 


I just tried to do a little research on my own and one point where I think I might have achieved a slightly better research result compared to research done with more sophisticated computer programs is in the the analysis of Mandarin word frequency because I did the analysis manually. This however leaves my study with a higher error margin.


In this post I will present the results of the Mandarin character frequency analysis, in the posts to follow I will talk about the Mandarin word frequency analysis. 

Research method

I analyzed 80 articles in 4 sections (politics, international, society and finance), 20 articles per section which is a very very small amount of data any way you look at it, but some interesting things came out of the analysis anyway. One of the obvious problems with sampling such a small amount of data over a short period of time is, that there are a lot of 'seasonal' characters and expressions appearing in it. In the case of this small study these were expressions based around the elections in Taiwan, Greece, the Chinese New Year, European debt crisis and the Makiyo case. 

In my opinion, sampling characters over a very long period of time also has its flaws, because if for instance someone would like to know the approximate amount of characters necessary to read the news in Mandarin in Taiwan and in order to do that he would sample a very large amount of news articles over the course of one year instead of 6 weeks as was my case, then of course the number of unique characters occurring in those articles would be much higher than in the case of my study, simply because of the fact, that the probability of rare characters appearing over a longer  period of time is much higher. That in my opinion doesn't mean that the study is more accurate in telling you how many characters you need to approximately know, but it does make it more complete. In the case of my little study I simply tried to map out the number of unique characters, words and their frequency in the news in Taiwan in 4 sections over the course of 6 weeks. 

To analyze character frequency was relatively easy, to analyze word frequency was much more difficult. To analyze character frequency and the total amount of unique characters present in the 80 articles, I asked my good friend (many thanks to Martin again) to write a computer program which basically did the work for me. All I had to do was input any amount of text in any format into a text file, run the program  which produced a list of unique characters from the most frequent to the least frequent one with a count of unique characters at the end of the file.

To analyze word frequency took a lot more effort and time. As most of those learning Mandarin know, words are not separated by spaces in Mandarin so I had to manually do that. In addition, some constructions like 在..內 which have other characters in between them had to be manually rewritten as 在內 in order to be counted as separate words. 

I think this is the second most questionable part of my little research (first being the small amount of research data), because I decided by feel and my Taiwanese friends' suggestions what should and what should not be considered as a separate word.

Also, while analyzing character frequency, there was practically no margin for error as the computer program did everything for me. While manually separating the words however, I had to add a 5% error margin simply because as I was gradually working myself through the articles I changed my approach several times in what I did and what I did not consider as a separate word and even though I edited input data over and over again to even out these changes, I might have overlooked something.

As for the technical part, what I did was, that I put all 80 articles into MS Word, manually move each word or construction onto a new line and then ran my friends program on the file. 

After having 8 frequency files (4 character frequency files + 4 word frequency files) I transferred them to OpenOffice Calc and started doing some statistics. 

Data analysis

This is the data table containing the most important information about the total number of sampled characters and the number of unique characters per section in the 80 analyzed articles as a whole. To my surprise there were only 2105 unique characters found in the 80 articles I analyzed. 

There are several reasons to this: 

1. The amount of the data I sampled was very small. 
2. Direct speech is rarely used in news articles, it appears only in the form of citations so as I will later show, characters typical for dialogues 我, 你, 去 or 回 for instance are nowhere close the top 100 (我 - 183rd, 你 - 982nd, 去 - 204th, 回 - 238th) and a lot of them didn't even make it to the entire list. 
3. In my opinion, Mandarin used in the news is really very different from Mandarin used in books  for instance so a lot of these characters didn't make it into the list either. Books again contain completely different expressions and sentence patterns, a lot of direct speech and therefore would have a different character count with a lot of characters that would not overlap with characters used in news. 

As can be seen from the table a higher total number of characters per section does not necessarily mean a higher amount of unique characters in that section. The total amount of characters in the  financial section 財經 is by 20% higher than the total amount of characters in the international 國際 section, but the number of unique characters in 財經 is 4% lower than it is in the 國際 section. The reason for this is that in the financial section a lot of the same expressions were used in a lot of articles over and over again (loan, debt, stock, growth ect.) whereas the international section, most of the articles were different, talking about different countries and people, with a lot of unique personal and country names appearing in them. 

This chart is a visual representation of the ratio between the number of unique characters you know and the percentage of characters you should recognize in the 80 articles. As can be seen from this chart, by knowing the first 50% of the unique characters, you should be able to recognize about 93% of all the characters in the 80 articles I analyzed. 

There are several reasons why this is not enough to flawlessly understand news articles, but an explanation would be very lengthy, as these reasons are related to how written Chinese and Chinese in general works, but one of the reasons (and this will be even more obvious in the Word frequency analysis section) can be better understood by looking at the following table:

In this table I stated the number of characters appearing only once, twice or three times (characters that are rare) in the texts I sampled and the percentage that they constitute out of all the unique characters. 

This table and the previous graph show the obvious, which is that even if a reader can recognize 93% of all the characters in these 80 articles, if he were to come across a character from the remaining 7% of the chart (which he will), it is highly probable that he will not recognize it even if he knows some random characters from the 'rare' 7% section of the graph.

Another obvious conclusion is that to learn how to understand the remaining 7% of the characters found in the texts will take the reader the same amount of effort as learning the first 93%. Or maybe another way to look at it is that by learning the remaining 50% of all the unique characters found in the texts, one will only increase his understanding abilities by 7%. 

For reference, I also stated those characters that that have a very high frequency of appearance compared to other characters. As you can see, there are only 15 characters that appeared more than 200 times and only 68 characters that appeared more than 100 times in the texts I sampled.

Character frequency table: 

Again for reference, I'm posting the list of the first 80 characters appearing in the articles I sampled, starting with the most frequent one. As I mentioned earlier, my small research was influenced by a lot of seasonal characters, but this table shows pretty nice results. As I mentioned earlier, typical direct speech characters like 我, 你, 去, 回, 來 ect. are nowhere to be seen. 

Also, another thing that I will try to talk about a little more in the Word frequency analysis section is, that knowing for instance the first 90% of all the unique characters really is not the same thing as being able to understand 90% of the information in the text. Chinese characters are representing morphemes rather than words and even though there are a lot of Chinese words that consist only of one character (or one morpheme only if you like), the majority of them consist of 2 characters. Doing a character frequency analysis is therefore the same thing as doing a morpheme frequency analysis. 

A way to look at it is that if a person would do the same frequency study as I did by sampling 80 English news articles, there might be an entry saying that the English morpheme tele appeared 220 times in the 80 sampled English articles. It does say a lot about the frequency of appearance of this morpheme, but nothing about the words it has appeared in (i.e: telephone, telegraph, television, teleport ect.).

The same thing  can be applied to the character frequency analysis. You can basically take any character from the above list and not know what it meant in the articles I sampled. For instance:

年 - year

今年 this year 

去年 last year 

龍年 the year of the dragon 

青少年 teenagers/teenage years

新 - new

新年 - new year

新聞 - news

重新 - repeat, again

Or a more extreme example:

馬 - horse

馬上 - immediately

馬英九 - Ma YingJiu (president of Taiwan)

歐巴馬 - Barack Obama

馬路 - road

馬政府 - abbreviation for 馬英九政府 - The government of Ma Yingjiu 

ect.

You can download the full character frequency list in the download section of this blog.

End of Part I. To be continued in Part II - Word frequency analysis.