[ ML 文章收集 ] Topic Modeling in Python: Latent Dirichlet Allocation (LDA)

Source From Here
Introduction
Topic Models, in a nutshell, are a type of statistical language models used for uncovering hidden structure in a collection of texts. In a practical and more intuitively, you can think of it as a task of:

Dimensionality Reduction, where rather than representing a text T in its feature space as {Word_i: count(Word_i, T) for Word_i in Vocabulary}, you can represent it in a topic space as {Topic_i: Weight(Topic_i, T) for Topic_i in Topics}

Unsupervised Learning, where it can be compared to clustering, as in the case of clustering, the number of topics, like the number of clusters, is an output parameter. By doing topic modeling, we build clusters of words rather than clusters of texts. A text is thus a mixture of all the topics, each having a specific weight.

Tagging, abstract “topics” that occur in a collection of documents that best represents the information in them.

There are several existing algorithms you can use to perform the topic modeling. The most common of it are, Latent Semantic Analysis (LSA/LSI), Probabilistic Latent Semantic Analysis (pLSA), and Latent Dirichlet Allocation (LDA)

In this article, we’ll take a closer look at LDA, and implement our first topic model using the sklearn implementation in python 3.x

Theoretical Overview
LDA is a generative probabilistic model that assumes each topic is a mixture over an underlying set of words, and each document is a mixture of over a set of topic probabilities.

http://chdoig.github.io/pytexas2015-topic-modeling/#/3/4

We can describe the generative process of LDA as, given the M number of documents, N number of words, and prior K number of topics, the model trains to output:

* psi, the distribution of words for each topic K
* phi, the distribution of topics for each document i

Parameters of LDA

Alpha parameter is Dirichlet prior concentration parameter that represents document-topic density — with a higher alpha, documents are assumed to be made up of more topics and result in more specific topic distribution per document.

Beta parameter is the same prior concentration parameter that represents topic-word density — with high beta, topics are assumed to made of up most of the words and result in a more specific word distribution per topic.

LDA Implementation
The complete code is available as a Jupyter Notebook on GitHub (my version):

1. Loading data
2. Data cleaning
3. Exploratory analysis
4. Preparing data for LDA analysis
5. LDA model training
6. Analyzing LDA model results

Loading data
For this tutorial, we’ll use the dataset of papers published in NIPS conference. The NIPS conference (Neural Information Processing Systems) is one of the most prestigious yearly events in the machine learning community. The CSV data file contains information on the different NIPS papers that were published from 1987 until 2016 (29 years!). These papers discuss a wide variety of topics in machine learning, from neural networks to optimization methods, and many more.

Let’s start by looking at the content of the file (Unzip this file to get papers.csv):

view plaincopy to clipboardprint?

1. # Importing modules  
2. import pandas as pd  
3. import os  
4.   
5. # Read data into papers  
6. papers = pd.read_csv('data/papers.csv')  
7. # Print head  
8. papers.head()  

Data Cleaning
Since the goal of this analysis is to perform topic modeling, we will solely focus on the text data from each paper, and drop other metadata columns:

view plaincopy to clipboardprint?

1. # Remove the columns  
2. papers = papers.drop(columns=['id', 'event_type', 'pdf_name'], axis=1)  
3. # Print out the first rows of papers  
4. papers.head()  

Remove punctuation/lower casing
Next, let’s perform a simple preprocessing on the content of paper_text column to make them more amenable for analysis, and reliable results. To do that, we’ll use a regular expression to remove any punctuation, and then lowercase the text:

view plaincopy to clipboardprint?

1. %%time  
2. # Load the regular expression library  
3. import re  
4. from nltk.stem import PorterStemmer  
5.   
6. # Remove punctuation  
7. papers['paper_text_processed'] = papers['paper_text'].map(lambda x: re.sub('[,\.!?]', '', x))  
8. # Remove none alphabic character(s)  
9. papers['paper_text_processed'] = papers['paper_text_processed'].map(lambda x: re.sub('[~\'(){}:;+-=*"&]', ' ', x))  
10. # Remove number  
11. papers['paper_text_processed'] = papers['paper_text_processed'].map(lambda x: re.sub('[0-9]+', '', x))  
12. # Convert the titles to lowercase  
13. papers['paper_text_processed'] = papers['paper_text_processed'].map(lambda x: x.lower().strip())  
14. # Stemming  
15. porter = PorterStemmer()  
16. def stemming(text):      
17.     return ' '.join(list(map(lambda w: porter.stem(w),filter(lambda w: len(w)>3, text.split()))))  
18.   
19. papers['paper_text_processed'] = papers['paper_text_processed'].map(lambda x: stemming(x))  
20. # Print out the first rows of papers  
21. papers['paper_text_processed'].head()  

Exploratory Analysis
To verify whether the preprocessing happened correctly, we’ll make a word cloud using the wordcloud package to get a visual representation of most common words. It is key to understanding the data and ensuring we are on the right track, and if any more preprocessing is necessary before training the model:

view plaincopy to clipboardprint?

1. # Import the wordcloud library  
2. from wordcloud import WordCloud  
3.   
4. # Join the different processed titles together.  
5. long_string = ','.join(list(papers['paper_text_processed'].values))  
6. # Create a WordCloud object  
7. wordcloud = WordCloud(background_color="white", max_words=5000, contour_width=3, contour_color='steelblue')  
8. # Generate a word cloud  
9. wordcloud.generate(long_string)  
10. # Visualize the word cloud  
11. wordcloud.to_image()  

Prepare text for LDA Analysis
Next, let’s work to transform the textual data in a format that will serve as an input for training LDA model. We start by converting the documents into a simple vector representation (Bag of Words BOW). Next, we will convert a list of titles into lists of vectors, all with length equal to the vocabulary.

We’ll then plot the ten most frequent words based on the outcome of this operation (the list of document vectors). As a check, these words should also occur in the word cloud.

view plaincopy to clipboardprint?

1. # Load the library with the CountVectorizer method  
2. from sklearn.feature_extraction.text import CountVectorizer  
3. import numpy as np  
4. import matplotlib.pyplot as plt  
5. import seaborn as sns  
6. sns.set_style('whitegrid')  
7. %matplotlib inline  
8.   
9. # Helper function  
10. def plot_10_most_common_words(count_data, count_vectorizer):  
11.     import matplotlib.pyplot as plt  
12.     words = count_vectorizer.get_feature_names()  
13.     total_counts = np.zeros(len(words))  
14.     for t in count_data:  
15.         total_counts+=t.toarray()[0]  
16.       
17.     count_dict = (zip(words, total_counts))  
18.     count_dict = sorted(count_dict, key=lambda x:x[1], reverse=True)[0:10]  
19.     words = [w[0] for w in count_dict]  
20.     counts = [w[1] for w in count_dict]  
21.     x_pos = np.arange(len(words))   
22.       
23.     plt.figure(2, figsize=(12, 12/1.6180))  
24.     plt.subplot(title='10 most common words')  
25.     sns.set_context("notebook", font_scale=1.25, rc={"lines.linewidth": 2.5})  
26.     sns.barplot(x_pos, counts, palette='husl')  
27.     plt.xticks(x_pos, words, rotation=90)   
28.     plt.xlabel('words')  
29.     plt.ylabel('counts')  
30.     plt.show()  
31.       
32. # Initialise the count vectorizer with the English stop words  
33. count_vectorizer = CountVectorizer(stop_words='english')  
34.   
35. # Fit and transform the processed titles  
36. count_data = count_vectorizer.fit_transform(papers['paper_text_processed'])  
37. print(count_data.shape)  
38.   
39. # Visualise the 10 most common words  
40. plot_10_most_common_words(count_data, count_vectorizer)  

LDA model training and results visualization
To keep things simple, we will only tweak the number of topic parameters:

view plaincopy to clipboardprint?

1. %%timeit  
2. import warnings  
3. warnings.simplefilter("ignore", DeprecationWarning)  
4. # Load the LDA model from sk-learn  
5. from sklearn.decomposition import LatentDirichletAllocation as LDA  
6.   
7. # Helper function  
8. def print_topics(model, count_vectorizer, n_top_words):  
9.     words = count_vectorizer.get_feature_names()  
10.     for topic_idx, topic in enumerate(model.components_):  
11.         print("\nTopic #%d:" % topic_idx)  
12.         print("/".join([words[i]  
13.                         for i in topic.argsort()[:-n_top_words - 1:-1]]))  
14.           
15. # Tweak the two parameters below  
16. number_topics = 5  
17. number_words = 20  
18. # Create and fit the LDA model  
19. lda = LDA(n_components=number_topics, n_jobs=-1)  
20. lda.fit(count_data)  
21. # Print the topics found by the LDA model  
22. print("Topics found via LDA:")  
23. print_topics(lda, count_vectorizer, number_words)  

Analyzing LDA model results
Now that we have a trained model let’s visualize the topics for interpretability. To do so, we’ll use a popular visualization package, pyLDAvis which is designed to help interactively with:

1. Better understanding and interpreting individual topics, and
2. Better understanding the relationships between the topics.

For (1), you can manually select each topic to view its top most frequent and/or “relevant” terms, using different values of the λ parameter. This can help when you’re trying to assign a human interpretable name or “meaning” to each topic. For (2), exploring the Intertopic Distance Plot can help you learn about how topics relate to each other, including potential higher-level structure between groups of topics.

view plaincopy to clipboardprint?

1. from pyLDAvis import sklearn as sklearn_lda  
2. import pickle   
3. import pyLDAvis  
4.   
5. LDAvis_data_filepath = os.path.join('./ldavis_prepared_'+str(number_topics))  
6. # # this is a bit time consuming - make the if statement True  
7. # # if you want to execute visualization prep yourself  
8. if not os.path.isfile(LDAvis_data_filepath):  
9.     LDAvis_prepared = sklearn_lda.prepare(lda, count_data, count_vectorizer)  
10.     with open(LDAvis_data_filepath, 'wb') as f:  
11.         pickle.dump(LDAvis_prepared, f)  
12.           
13. # load the pre-prepared pyLDAvis data from disk  
14. with open(LDAvis_data_filepath, 'rb') as f:  
15.     LDAvis_prepared = pickle.load(f)  
16.       
17. pyLDAvis.save_html(LDAvis_prepared, './ldavis_prepared_'+ str(number_topics) +'.html')  

Closing Notes
Machine learning has become increasingly popular over the past decade, and recent advances in computational availability have led to exponential growth to people looking for ways how new methods can be incorporated to advance the field of Natural Language Processing.

Often, we treat topic models as black-box algorithms, but hopefully, this post addressed to shed light on the underlying math, and intuitions behind it, and high-level code to get you started with any textual data. In the next article, we’ll go one step deeper into understanding how you can evaluate the performance of topic models, tune its hyper-parameters to get more intuitive and reliable results.

Source
* [1] Topic model — Wikipedia. https://en.wikipedia.org/wiki/Topic_model
* [2] Distributed Strategies for Topic Modeling. https://www.ideals.illinois.edu/bitstream/handle/2...els.pdf?sequence=2&isAllowed=y
* [3] Topic Mapping — Software — Resources — Amaral Lab. https://amaral.northwestern.edu/resources/software/topic-mapping
* [4] A Survey of Topic Modeling in Text Mining. https://thesai.org/Downloads/Volume6No1/Paper_21-A...ic_Modeling_in_Text_Mining.pdf

Supplement
* Stemming and Lemmatization in Python

This tutorial covers the introduction to Stemming & Lemmatization used in Text and Natural Language Processing.