内容简介:大数据文摘出品编译:Hope、臻臻、CoolBoy
大数据文摘出品
编译:Hope、臻臻、CoolBoy
最近,租房这事儿成了北漂族的一大bug,要想租到称心如意的房子,不仅要眼明手快,还得看清各类“前辈”的评价避开大坑。一位 程序员 在出行选酒店的时候就借用了程序工具:先用 python 爬下了海外点评网站TripAdvisor的数千评论,并且用R进行了文本分析和情感分析,科学选房,高效便捷,极具参考价值。
以下,这份超详实的教程拿好不谢。
TripAdvisor提供的信息对于旅行者的出行决策非常重要。但是,要去了解TripAdvisor的泡沫评分和数千个评论文本之间的细微差别是极具挑战性的。
为了更加全面地了解酒店旅客的评论是否会对之后酒店的服务产生影响,我爬取了TripAdvisor中一个名为Hilton Hawaiian Village酒店的所有英文评论。这里我不会对爬虫的细节进行展开。
Python源码:
https://github.com/susanli2016/NLP-with-Python/blob/master/Web%20scraping%20Hilton%20Hawaiian%20Village%20TripAdvisor%20Reviews.py
加载扩展包
- library(dplyr)
- library(readr)
- library(lubridate)
- library(ggplot2)
- library(tidytext)
- library(tidyverse)
- library(stringr)
- library(tidyr)
- library(scales)
- library(broom)
- library(purrr)
- library(widyr)
- library(igraph)
- library(ggraph)
- library(SnowballC)
- library(wordcloud)
- library(reshape2)
- theme_set(theme_minimal())
数据集
df <- read_csv("Hilton_Hawaiian_Village_Waikiki_Beach_Resort-Honolulu_Oahu_Hawaii__en.csv") df <- df[complete.cases(df), ] df$review_date <- as.Date(df$review_date, format = "%d-%B-%y") dim(df); min(df$review_date); max(df$review_date)
Figure 2
我们在TripAdvisor上一共获得了13,701条关于Hilton Hawaiian Village酒店的英文评论,这些评论的时间范围是从2002–03–21 到2018–08–02。
df %>% count(Week = round_date(review_date, "week")) %>% ggplot(aes(Week, n)) + geom_line() + ggtitle('The Number of Reviews Per Week')
Figure 2
在2014年末,周评论数量达到最高峰。那一个星期里酒店被评论了70次。
对评论文本进行文本挖掘
df <- tibble::rowid_to_column(df, "ID") df <- df %>% mutate(review_date = as.POSIXct(review_date, origin = "1970-01-01"),month = round_date(review_date, "month")) review_words <- df %>% distinct(review_body, .keep_all = TRUE) %>% unnest_tokens(word, review_body, drop = FALSE) %>% distinct(ID, word, .keep_all = TRUE) %>% anti_join(stop_words, by = "word") %>% filter(str_detect(word, "[^\\d]")) %>% group_by(word) %>% mutate(word_total = n()) %>% ungroup() word_counts <- review_words %>% count(word, sort = TRUE) word_counts %>% head(25) %>% mutate(word = reorder(word, n)) %>% ggplot(aes(word, n)) + geom_col(fill = "lightblue") + scale_y_continuous(labels = comma_format()) + coord_flip() + labs(title = "Most common words in review text 2002 to date", subtitle = "Among 13,701 reviews; stop words removed", y = "# of uses")
Figure 3
我们还可以更进一步的把“stay”和“stayed”,“pool”和“pools”这些意思相近的词合并起来。这个步骤被称为词干提取,也就是将变形(或是衍生)词语缩减为词干,基词或根词的过程。
word_counts %>% head(25) %>% mutate(word = wordStem(word)) %>% mutate(word = reorder(word, n)) %>% ggplot(aes(word, n)) + geom_col(fill = "lightblue") + scale_y_continuous(labels = comma_format()) + coord_flip() + labs(title = "Most common words in review text 2002 to date", subtitle = "Among 13,701 reviews; stop words removed and stemmed", y = "# of uses")
Figure 4
二元词组
通常我们希望了解评论中单词的相互关系。哪些词组在评论文本中比较常用呢?如果给出一列单词,那么后面会随之出现什么单词呢?哪些词之间的关联性最强?许多有意思的文本挖掘都是基于这些关系的。在研究两个连续单词的时候,我们称这些单词对为“二元词组”。
所以,在Hilton Hawaiian Village的评论中,哪些是最常见的二元词组呢?
review_bigrams <- df %>% unnest_tokens(bigram, review_body, token = "ngrams", n = 2) bigrams_separated <- review_bigrams %>% separate(bigram, c("word1", "word2"), sep = " ") bigrams_filtered <- bigrams_separated %>% filter(!word1 %in% stop_words$word) %>% filter(!word2 %in% stop_words$word) bigram_counts <- bigrams_filtered %>% count(word1, word2, sort = TRUE) bigrams_united <- bigrams_filtered %>% unite(bigram, word1, word2, sep = " ") bigrams_united %>% count(bigram, sort = TRUE)
Figure 5
最常见的二元词组是“rainbow tower”(彩虹塔),其次是“hawaiian village”(夏威夷村)。
我们可以利用网络可视化来展示这些二元词组:
review_subject <- df %>% unnest_tokens(word, review_body) %>% anti_join(stop_words) my_stopwords <- data_frame(word = c(as.character(1:10))) review_subject <- review_subject %>% anti_join(my_stopwords) title_word_pairs <- review_subject %>% pairwise_count(word, ID, sort = TRUE, upper = FALSE) set.seed(1234) title_word_pairs %>% filter(n >= 1000) %>% graph_from_data_frame() %>% ggraph(layout = "fr") + geom_edge_link(aes(edge_alpha = n, edge_width = n), edge_colour = "cyan4") + geom_node_point(size = 5) + geom_node_text(aes(label = name), repel = TRUE, point.padding = unit(0.2, "lines")) + ggtitle('Word network in TripAdvisor reviews') theme_void()
Figure 6
上图展示了TripAdvisor评论中较为常见的二元词组。这些词至少出现了1000次,而且其中不包含停用词。
在网络图中我们发现出现频率最高的几个词存在很强的相关性(“hawaiian”, “village”, “ocean” 和“view”),不过我们没有发现明显的聚集现象。
三元词组
二元词组有时候还不足以说明情况,让我们来看看TripAdvisor中关于Hilton Hawaiian Village酒店最常见的三元词组有哪些。
review_trigrams <- df %>% unnest_tokens(trigram, review_body, token = "ngrams", n = 3) trigrams_separated <- review_trigrams %>% separate(trigram, c("word1", "word2", "word3"), sep = " ") trigrams_filtered <- trigrams_separated %>% filter(!word1 %in% stop_words$word) %>% filter(!word2 %in% stop_words$word) %>% filter(!word3 %in% stop_words$word) trigram_counts <- trigrams_filtered %>% count(word1, word2, word3, sort = TRUE) trigrams_united <- trigrams_filtered %>% unite(trigram, word1, word2, word3, sep = " ") trigrams_united %>% count(trigram, sort = TRUE)
Figure 7
最常见的三元词组是“hilton hawaiian village”,其次是“diamond head tower”,等等。
评论中关键单词的趋势
随着时间的推移,哪些单词或话题变得更加常见,或者更加罕见了呢?从这些信息我们可以探知酒店做出的调整,比如在服务上,翻新上,解决问题上。我们还可以预测哪些主题会更多地被提及。
我们想要解决类似这样的问题:随着时间的推移,在TripAdvisor的评论区中哪些词出现的频率越来越高了?
reviews_per_month <- df %>% group_by(month) %>% summarize(month_total = n()) word_month_counts <- review_words %>% filter(word_total >= 1000) %>% count(word, month) %>% complete(word, month, fill = list(n = 0)) %>% inner_join(reviews_per_month, by = "month") %>% mutate(percent = n / month_total) %>% mutate(yearyear = year(month) + yday(month) / 365) mod <- ~ glm(cbind(n, month_total - n) ~ year, ., family = "binomial") slopes <- word_month_counts %>% nest(-word) %>% mutate(model = map(data, mod)) %>% unnest(map(model, tidy)) %>% filter(term == "year") %>% arrange(desc(estimate)) slopes %>% head(9) %>% inner_join(word_month_counts, by = "word") %>% mutate(word = reorder(word, -estimate)) %>% ggplot(aes(month, n / month_total, color = word)) + geom_line(show.legend = FALSE) + scale_y_continuous(labels = percent_format()) + facet_wrap(~ word, scales = "free_y") + expand_limits(y = 0) + labs(x = "Year", y = "Percentage of reviews containing this word", title = "9 fastest growing words in TripAdvisor reviews", subtitle = "Judged by growth rate over 15 years")
Figure 8
在2010年以前我们可以看到大家讨论的焦点是“friday fireworks”(周五的烟花)和“lagoon”(环礁湖)。而在2005年以前“resort fee”(度假费)和“busy”(繁忙)这些词的词频增长最快。
评论区中哪些词的词频在下降呢?
slopes %>% tail(9) %>% inner_join(word_month_counts, by = "word") %>% mutate(word = reorder(word, estimate)) %>% ggplot(aes(month, n / month_total, color = word)) + geom_line(show.legend = FALSE) + scale_y_continuous(labels = percent_format()) + facet_wrap(~ word, scales = "free_y") + expand_limits(y = 0) + labs(x = "Year", y = "Percentage of reviews containing this term", title = "9 fastest shrinking words in TripAdvisor reviews", subtitle = "Judged by growth rate over 4 years")
Figure 9
这张图展示了自2010年以来逐渐变少的主题。这些词包括“hhv” (我认为这是 hilton hawaiian village的简称), “breakfast”(早餐), “upgraded”(升级), “prices”(价格) and “free”(免费)。
让我们对一些单词进行比较。
word_month_counts %>% filter(word %in% c("service", "food")) %>% ggplot(aes(month, n / month_total, color = word)) + geom_line(size = 1, alpha = .8) + scale_y_continuous(labels = percent_format()) + expand_limits(y = 0) + labs(x = "Year", y = "Percentage of reviews containing this term", title = "service vs food in terms of reviewers interest")
Figure 10
在2010年之前,服务(service)和食物(food)都是热点主题。关于服务和食物的讨论在2003年到达顶峰,自2005年之后就一直在下降,只是偶尔会反弹。
情感分析
情感分析被广泛应用于对评论、调查、网络和社交媒体文本的分析,以反映客户的感受,涉及范围包括市场营销、客户服务和临床医学等。
在本案例中,我们的目标是对评论者(也就是酒店旅客)在住店之后对酒店的态度进行分析。这个态度可能是一个判断或是评价。
下面来看评论中出现得最频繁的积极词汇和消极词汇。
reviews <- df %>% filter(!is.na(review_body)) %>% select(ID, review_body) %>% group_by(row_number()) %>% ungroup() tidy_reviews <- reviews %>% unnest_tokens(word, review_body) tidy_reviews <- tidy_reviews %>% anti_join(stop_words) bing_word_counts <- tidy_reviews %>% inner_join(get_sentiments("bing")) %>% count(word, sentiment, sort = TRUE) %>% ungroup() bing_word_counts %>% group_by(sentiment) %>% top_n(10) %>% ungroup() %>% mutate(word = reorder(word, n)) %>% ggplot(aes(word, n, fill = sentiment)) + geom_col(show.legend = FALSE) + facet_wrap(~sentiment, scales = "free") + labs(y = "Contribution to sentiment", x = NULL) + coord_flip() + ggtitle('Words that contribute to positive and negative sentiment in the reviews')
Figure 11
让我们换一个情感文本库,看看结果是否一样。
contributions <- tidy_reviews %>% inner_join(get_sentiments("afinn"), by = "word") %>% group_by(word) %>% summarize(occurences = n(), contribution = sum(score)) contributions %>% top_n(25, abs(contribution)) %>% mutate(word = reorder(word, contribution)) %>% ggplot(aes(word, contribution, fill = contribution > 0)) + ggtitle('Words with the greatest contributions to positive/negative sentiment in reviews') + geom_col(show.legend = FALSE) + coord_flip()
Figure 12
有意思的是,“diamond”(出自“diamond head-钻石头”)被归类为积极情绪。
这里其实有一个潜在问题,比如“clean”(干净)是什么词性取决于语境。如果前面有个“not”(不),这就是一个消极情感了。事实上一元词在否定词(如not)存在的时候经常碰到这种问题,这就引出了我们下一个话题:
在情感分析中使用二元词组来辨明语境
我们想知道哪些词经常前面跟着“not”(不)
bigrams_separated %>% filter(word1 == "not") %>% count(word1, word2, sort = TRUE)
Figure 13
“a”前面跟着“not”的情况出现了850次,而“the”前面跟着“not”出现了698次。不过,这种结果不是特别有实际意义。
AFINN <- get_sentiments("afinn") not_words <- bigrams_separated %>% filter(word1 == "not") %>% inner_join(AFINN, by = c(word2 = "word")) %>% count(word2, score, sort = TRUE) %>% ungroup() not_words
Figure 14
上面的分析告诉我们,在“not”后面最常见的情感词汇是“worth”,其次是“recommend”,这些词都被认为是积极词汇,而且积极程度得分为2。
所以在我们的数据中,哪些单词最容易被误解为相反的情感?
not_words %>% mutate(contribution = n * score) %>% arrange(desc(abs(contribution))) %>% head(20) %>% mutate(word2 = reorder(word2, contribution)) %>% ggplot(aes(word2, n * score, fill = n * score > 0)) + geom_col(show.legend = FALSE) + xlab("Words preceded by \"not\"") + ylab("Sentiment score * number of occurrences") + ggtitle('The 20 words preceded by "not" that had the greatest contribution to sentiment scores, positive or negative direction') + coord_flip()
Figure 15
二元词组“not worth”, “not great”, “not good”, “not recommend”和“not like”是导致错误判断的最大根源,使得评论看起来比原来积极的多。
除了“not”以外,还有其他的否定词会对后面的内容进行情绪的扭转,比如“no”, “never” 和“without”。让我们来看一下具体情况。
negation_words <- c("not", "no", "never", "without") negated_words <- bigrams_separated %>% filter(word1 %in% negation_words) %>% inner_join(AFINN, by = c(word2 = "word")) %>% count(word1, word2, score, sort = TRUE) %>% ungroup() negated_words %>% mutate(contribution = n * score, word2 = reorder(paste(word2, word1, sep = "__"), contribution)) %>% group_by(word1) %>% top_n(12, abs(contribution)) %>% ggplot(aes(word2, contribution, fill = n * score > 0)) + geom_col(show.legend = FALSE) + facet_wrap(~ word1, scales = "free") + scale_x_discrete(labels = function(x) gsub("__.+$", "", x)) + xlab("Words preceded by negation term") + ylab("Sentiment score * # of occurrences") + ggtitle('The most common positive or negative words to follow negations such as "no", "not", "never" and "without"') + coord_flip()
Figure 16
看来导致错判为积极词汇的最大根源来自于“not worth/great/good/recommend”,而另一方面错判为消极词汇的最大根源是“not bad” 和“no problem”。
最后,让我们来观察一下最积极和最消极的评论。
sentiment_messages <- tidy_reviews %>% inner_join(get_sentiments("afinn"), by = "word") %>% group_by(ID) %>% summarize(sentiment = mean(score), words = n()) %>% ungroup() %>% filter(words >= 5) sentiment_messages %>% arrange(desc(sentiment))
Figure 17
最积极的评论来自于ID为2363的记录:“哇哇哇,这地方太好了!从房间我们可以看到很漂亮的景色,我们住得很开心。Hilton酒店就是很棒!无论是小孩还是大人,这家酒店有着所有你想要的东西。”
df[ which(df$ID==2363), ]$review_body[1]
Figure 18
sentiment_messages %>% arrange(sentiment)
Figure 19
最消极的评论来自于ID为3748的记录:“(我)住了5晚(16年5月12日-5月17日)。第一晚,我们发现地砖坏了,小孩子在玩手指。第二晚,我们看到小蟑螂在儿童食物上爬。前台给我们换了房间,但他们让我们一小时之内搬好房间,否则就不能换房。。。已经晚上11点,我们都很累了,孩子们也睡了。我们拒绝了这个建议。退房的时候,前台小姐跟我讲,蟑螂在他们的旅馆里很常见。她还反问我在加州见不到蟑螂吗?我没想到能在Hilton遇到这样的事情。”
df[ which(df$ID==3748), ]$review_body[1]
Figure 20
Github源码:
https://github.com/susanli2016/Data-Analysis-with-R/blob/master/Text%20Mining%20Hilton%20Hawaiian%20Village%20TripAdvisor%20Reviews.Rmd
相关报道:
https://towardsdatascience.com/scraping-tripadvisor-text-mining-and-sentiment-analysis-for-hotel-reviews-cc4e20aef333
【本文是51CTO专栏机构大数据文摘的原创文章,微信公众号“大数据文摘( id: BigDataDigest)”】
以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网
猜你喜欢:- 一款租房 APP,基于 Node.js 爬虫
- 爬取两万多租房数据,告诉你广州房租现状
- 做一个跑通前后端的`豆瓣租房`移动端webApp
- 到今年6月 北京市所有公租房都将安装人脸识别系统
- 北京五环实录:合租房里互不相识,凌晨四点互联网大厂上班
- 如何成为数据科学家?数据科学业界大牛们倾囊相授
本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。
Web信息架构(第3版)
Peter Morville、Louis Rosenfeld / 陈建勋 / 电子工业出版社 / 2008年8月 / 85.00
本书涵盖了信息架构基本原理和实践应用的方方面面。全书共7个部分,包括信息架构概述、信息架构的基本原理、信息架构的开发流程和方法论、信息架构实践、信息架构与组织、两个案例研究,以及参考资料清单。 本书兼具较高的理论价值和实用价值,曾被Web设计领域多本书籍重点推荐,是信息架构领域公认的经典书,不论新手还是专家都能各取所需。本书可供Web设计与开发者、Web架构师、网站管理者及信息管理相关人员参......一起来看看 《Web信息架构(第3版)》 这本书的介绍吧!