一、文本預處理階段###
1.1 設定訓練集和測試集
訓練集每一類的數量為500個文檔，測試集每一類的數量也為500個文檔。
1.2 計算每個文本的DF
為每一個文本計算TF，return格式為：'word', 'file_name', term-frequency
先算出每個文檔中的'word', term-frequency, 在結束改文本的循環后將該文本中出現的詞以 'word', 'file_name', term-frequency的形式加入 word_docid_tf
def compute_tf_by_file(self):
word_docid_tf = []
for name in self.filenames:
with open(join(name), 'r') as f:
tf_dict = dict()
for line in f:
line = self.process_line(line)
words = jieba.cut(line.strip(), cut_all=False)
for word in words:
tf_dict[word] = tf_dict.get(word, 0) + 1
tf_list = tf_dict.items()
word_docid_tf += [[item[0], name, item[1]] for item in tf_list]
return word_docid_tf
1.3 計算每個詞項的TF、DF
為每一個詞項計算TF，return的term_freq格式為：'word', dict ( 'file_name ', tf )
為每一個詞項計算DF，return的doc_freq格式為：'word', df
def compute_tfidf(self):
word_docid_tf = self.compute_tf_by_file()
word_docid_tf.sort()
doc_freq = dict()
term_freq = dict()
for current_word, group in groupby(word_docid_tf, itemgetter(0)):
doclist = []
df = 0
for current_word, file_name, tf in group:
doclist.append((file_name, tf))
df += 1
term_freq[current_word] = dict(doclist)
doc_freq[current_word] = df
return term_freq, doc_freq
1.4 精簡term_freq, doc_freq
除去只出現在一個或0個文檔中的詞項
除去數字詞項
def reduce_tfidf(self, term_freq, doc_freq):
remove_list = []
for key in term_freq.keys():
if len(key) < 2:#該詞只出現在一個或0個文檔中
remove_list.append(key)
else:
try:
float(key)#該詞是數字
remove_list.append(key)
except ValueError:
continue
for key in remove_list:
term_freq.pop(key)
doc_freq.pop(key)
return term_freq, doc_freq
1.5 為每個文本構建特征向量train_feature, train_target
為term_freq, doc_freq中的key，也就是詞項標明index
用jieba分詞，將分好的詞放入一個臨時的數組中。
遍歷數組，由doc_freq[word]取得DF并計算iDF，由term_freq[word][name]
取得該詞項在該文檔中的TF，并計算每個詞項的tf-idf值，并作為向量中詞項對應index那一維的值。
train_feature, train_target = train_tfidf.tfidf_feature(os.path.join(input_path, 'train'),train_tf, train_df, N)
def tfidf_feature(self, dir, term_freq, doc_freq, N):
filenames = []
for (dirname, dirs, files) in os.walk(dir):
for file in files:
filenames.append(os.path.join(dirname, file))
word_list = dict()
for idx, word in enumerate(doc_freq.keys()):
word_list[word] = idx
features = []
target = []
for name in filenames:
feature = np.zeros(len(doc_freq.keys()))
words_in_this_file = set()
tags = re.split('[/\\\\]', name)
tag = tags[-2]
with open(name, 'rb') as f:
for line in f:
line = self.process_line(line)
words = jieba.cut(line.strip(), cut_all=False)
for word in words:
words_in_this_file.add(word)
for word in words_in_this_file:
try:
idf = np.log(float(N) / doc_freq[word])
tf = term_freq[word][name]
feature[word_list[word]] = tf*idf
except KeyError:
continue
features.append(feature)
target.append(tag)
return sparse.csr_matrix(np.asarray(features)), np.asarray(target)
1.6 存儲&加載
為了節約之后運行的時間，可以通過如下方式把測試集tf和df的值直接存儲：
Pickle.dump(train_tf, open(os.path.join(input_path, 'train_tf.pkl'), 'wb'))
print "saved train_tf.pkl"
Pickle.dump(train_df, open(os.path.join(input_path, 'train_df.pkl'), 'wb'))
print "saved train_df.pkl"
之后運行時，可以通過如下方式把測試集tf和df的值直接加載到內存，省去了重新計算的時間：
train_tf = Pickle.load(open(os.path.join(input_path, 'train_tf.pkl'), 'rb'))
print "loaded train_tf.pkl"
train_df = Pickle.load(open(os.path.join(input_path, 'train_df.pkl'), 'rb'))
train_tfidf.doc_freq=train_df
print "loaded train_df.pkl"
