使用深度学习建立电影推荐系统，不到10分钟即可进行DIY！-FlyAI

使用深度学习建立电影推荐系统，不到10分钟即可进行DIY！

深度学习

最后更新 2020-10-30 14:26 阅读 7857

最后更新 2020-10-30 14:26

阅读 7857

深度学习

##FlyAI文献翻译
**英文原文：**[：Build Floating Movie Recommendations using Deep Learning — DIY in <10 Mins](https://medium.com/towards-artificial-intelligence/build-floating-movie-recommendations-using-deep-learning-diy-in-10-mins-26f585821697)
> **标签** ：深度学习

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603335794346277.png)

A deep-learning-based floating movie recommender

Have your ever watched inception and wondered about a dream within a dream and the possibility of real-world items floating in space? Well, we can’t float real items in space (at least so far) but can definitely project the python vectors in Multi-dimensional space.

Follow along and by end of this post, we will have a floating recommender system for movies to your name without any fuss around renting servers, writing flask codes, data scraping, or portal maintenance. Additionally, why just limit ourselves to movies and not extend it to our social network, stocks, sportsperson, TV shows, books, etc. Guess, we are on the same page or at least similar pages when projected in multi-dimensional vector space.

For those who would want to have a preview of the web-app, [Link here](https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/ravi07bec/4b033bc6299b655a2abda8b441e82364/raw/715f5656ae5dc2753f62ce4e9f89e7cb88f4e76e/test2.json "Link here")
Major Pre-requisites: Curiosity to learn new concepts and eagerness to deploy their own web-apps.

Minor Pre-Requisites: Understanding of word2vec embedding, Neural Nets, Tensorflow, Github, and recommender systems.

>一个基于深度学习的电影推荐系统

>你看过《盗梦空间》吗？是否被影片的梦中梦烧脑，或是想象现实世界中物品能否像电影中那样漂浮在空中？嗯，至少目前我们没有办法使物体悬浮在空中，但是，数据科学家可以将python向量映射到多维空间。

>本文将为介绍一个电影推荐系统，在推荐系统中，有许多电影向量悬浮于多维空间。同时，本文介绍的推荐系统无需租用服务器、编写代码、抓取数据或网页维护等繁杂的工作，非常容易上手。此外，你还可以将其扩展到社交网络、股票、运动员、电视节目、书籍等领域。想象一下，众多实例都会被映射到多维矢量空间，一览无余，那时多么直观啊。

>你可以点击该链接预览我们的电影推荐系统。

>学习本文需要满足以下条件：对新概念的好奇心，渴望部署自己的Web应用程序。

>同时，你最好对这些知识有所了解：word2vec嵌入，神经网络，Tensorflow，Github和推荐系统。

**Toolkit/Concepts:**

1. Embeddings- They are low dimensional space which can project the high dimensional vector. Since an image/word can have millions of parameters as characterized by pixels or tokens, we need a constrained uniform structure which can be an input to our ML models. When high dimensions or a large number of features are modeled to derive a pattern in limited data size, we generally encounter the curse of dimensionality problem i.e the model is unable to extract relevant patterns from input data. We may use techniques like Principle Component Analysis (PCA), TSNE, or L1 regularization to bring down the dimensions. However, for extremely high dimensions and sparse datasets, e.g Millions of tokens in Wikipedia corpus, neural net-based embeddings are more suitable to extract the relevant attributes.

Embeddings capture the semantic similarity across the tokens or pixels and project them in user-defined vector space. E.g- “Federer” and “Tennis” are mentioned together and thus on a 100-D embedding, Federer would be closer to tennis than Trump.

Sample diagram below:

>**工具/概念：**

>1.Embedding：将高维向量投射到低维稠密向量空间。由于表示图像/单词的像素或标记特征可能需要数百万个参数，因此我们需要定义一种受限统一结构，以此作为ML模型的输入。在对高维或大量特征进行建模以导出受限数据大小模式时，通常会遇到维数灾难，即模型无法从输入数据中提取相关模式。通常我们会使用主成分分析（PCA），TSNE或L1正则化等方法进行降维，但是对于极高维度和稀疏数据集（如Wikipedia语料库中的数百万个标记），基于神经网络的嵌入可以很好的提取相关属性。

>嵌入能够捕获标记或像素之间的语义相似性，并将其投影到向量空间中。例如，我们知道“费德勒”和“网球”之间存在密切关联，那么在100维嵌入空间中，“费德勒”比“特朗普”更接近“网球”。

>图示如下：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603335798846565.png)

Word2Vec vector similarity

2. Tensorboard Projector- A web application open-sourced by Tensorflow that interactively visualizes embeddings by reading them from our model and rendering them in lower dimensions. Sample projector for Wikipedia and MNIST corpus us available on tensorboard portal. Please spend a good amount of time understanding the intuitive projector, filters, embedding space, and various dimension reduction algorithms.

>word2vec向量相似性

>2. Tensorboard Projector：Tensorflow开源的Web应用，可以接收模型嵌入并在低维空间进行渲染，实现可视化交互。在tensorboard主页上可以查看Wikipedia和MNIST语料库的示例投影。建议读者花时间具体了解下映射、滤波、嵌入空间以及各种降维算法。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603335798673376.png)

</centerTensorboard>Tensorboard Projector trained on Wikipedia corpus</center>

>Wikipedia的Tensorboard项目示例

3. Dimensionality reduction algorithms- Though we provide a 100 Dimensional embedding to the projector, the human eye can only view items till 3 dimensions and we need to bring down our vector into 2-D or 3-D.

a. Principal Component Analysis- It is a technique for reducing the dimensionality of big datasets which improves model interpretability while minimizing information loss. The new principal components are created which are uncorrelated with each other with the aim to maximize variance i.e maximum information of large feature data into lower dimensional principal components. Based on the incremental variance explained by principal components e.g 90% of Variance, we can decide on the cutoff.

b. T-Stochastic Neighbor embedding- It is a non-linear technique for dimensionality reduction and generally used for visualization of high dimensional datasets. Its implementation is based on minimizing the divergence between two distributions: a distribution that measures pairwise similarities of the input objects and a distribution that measures pairwise similarities of the corresponding low-dimensional points in the embedding.

Major differences between PCA and TSNE are the linearity vs Non-Linearity dimensionality reduction and the objective function i.e PCA trying to preserve the global data structure while TSNE algorithm preserving local structure.

[Geeks For Geeks](https://www.geeksforgeeks.org/difference-between-pca-vs-t-sne/ "Geeks For Geeks") provide an 8 pointer difference between TSNE and PCA and act as a good reference point for our future experimentation.

>3.降维算法：尽管模型嵌入维度为100，但人眼只能看到3维物品，为此，我们需要将矢量压缩为2维或3维形式。

>方法a：主成分分析PCA。一种常用的降维算法，可提高模型的可解释性，同时最大程度地减少信息丢失。其主要思想是，构建新的互不相关的主成分，最大程度地提高方差，即将包含最大信息的特征转化为低维主成分。根据主成分增量方差（如90％的方差）确定截止值。

>方法b：t分布随机邻域嵌入TSNE。一种用于探索高维数据的非线性降维算法，将多维数据映射到适合于人类观察的两个或多个维度。该算法的关键在于最小化两个分布之间的差异，即测量输入对象的成对相似性的分布以及相应低维嵌入点的成对相似性的分布。

>PCA和TSNE之间的主要区别为：一是PCA为线性降维，而TSNE是非线性度降维；二是目标函数不同，PCA试图保留全局数据结构，而TSNE保留局部结构。

>Geeks For Geeks更详细的指出了TSNE和PCA之间的八项区别，为人们提供了很好的参考。

Movie lens data -Considered as the Gold standard dataset for movie recommender systems, Movie lens data provides [userid-movieid] level data for as big as 26Mn Ratings and 58k movies. Since 1997, Movie lens data has played a prominent role in the evolution of recommender systems ranging from Netflix to Amazon to individual projects. The landslide influence of the Movie lens can be gauged from the fact that “a search for “movielens” produced 2,750 results in Google Books and 7,580 in Google Scholar”.These datasets can be used to build recommender systems using item-based, user-based, SVD matrix, Neural nets, etc with the dependent variable as rating prediction for a new user based on their past preferences.

Using the concepts above, let us strive to create a floating recommendation web app with a Movies universe in multiples of Netflix e.g 3.7k Netflix movies vs 58k movies in our dataset, Source of data [here](https://www.foxbusiness.com/technology/how-many-movies-on-netflix "here")

Similar to how a word2vec embedding identifies closest words in vector space, we are creating closest movies in 100-dimensional vector space i.e reducing dimensions from 58k to 100.

The input dataset is sourced from MovieLens which is a transaction data of users giving 0–5 rating to movies. To prevent noise MovieLens has removed ratings from users who have fewer than 20 ratings and inexact demographics. There are multiple variants of data i.e ranging from [1k Users * 1.7k Movies] to [280k Users * 58k Movies], feel free to check out their [data repository](https://grouplens.org/datasets/movielens/ "data repository"). For our user-based Collaborative filtering, we can ignore the quality of ratings and consider the user-movie combination as a binary relationship i.e whether the user has rated or not rated a movie.

>4.MovieLens数据集：作为电影推荐系统的标准数据集，该数据集提供[userid-movieid]数据，包含58,000电影和2,600万评分。自1997年以来，该数据对于推动推荐系统的发展起着重要作用。在Google books搜索“ movielens”得到2,750个结果，在Google Scholar搜索得到7,580个结果，MovieLens数据集的影响可见一斑。应用该数据集可基于SVD矩阵或神经网络构建面向电影条目或是用户的推荐系统，并根据过去的偏好预测新用户评分。

>根据上述这些概念，我们可以创建一个电影推荐的Web应用，使用的电影数据包括3700Netflix电影和上述数据集中的58k电影。（数据源见此）

>就像word2vec嵌入将相近单词映射到向量空间后距离较小，我们在100维向量空间中对电影进行嵌入，这样就将数据维度从58k（类似one-hot表示）减少到100k（嵌入表示）。

>输入MovieLens数据集，该数据集包含用户对电影的评分，分值从0到5。为避免噪声，MovieLens剔除了评分次数低于20且统计不准确的用户。同时，该数据集有诸多版本，范围从[1k用户* 1.7k电影]到[280k用户* 58k电影]，使用时请注意查看。对基于用户的协同过滤，可以忽略分级质量，以0/1表示用户是否对电影进行评分。

Recollecting the concepts, a Movie recommendation can be of 3 types:

1. Popularity Based- Identifies the movies watched with the highest watch time in trailing X time period and recommends these movies to all the users.

2. Item Based- Based on metatags of the movies i.e actor, director, Language, release year, etc, However, the attributes of the items don’t change with time and the recommendations are not refreshed in real time user behavior is not taken into consideration.

3. User-based- Tags users to a particular cohort/subgroup based on their watching patterns and recommendations are other movies in the same cohort, e.g If I have started with Inception and Dark Knight and those who are on the platform before I have watched Prestige after watching these 2 movies, I will be recommended Prestige during my next visit.

I have written a detailed blog post a couple of years back on creating recommender systems using item-based collaborative filtering. Please feel free to check it out and provide feedback/comparisons of item-based vs user-based recommendation results.
Blog link: https://towardsdatascience.com/build-your-own-recommendation-engine-netflix-demystified-demo-code-550401d4885e

>根据之前的介绍，电影推荐可分为以下三种类型：

>1.基于流行度：标识过去X时段内最受欢迎（观看最多）的电影，将这些电影推荐给所有用户。

>2.基于电影属性：根据电影的元标记（如演员，导演，语言，发行年份等）进行推荐。但这样的缺点是，电影的属性不会随时间改变，无法考虑用户行为进行实时推荐。

>3.基于用户：根据用户的观看模式和喜好，对用户进行分组，为用户推荐同组内其他用户观看的电影。例如，如果我观看了Inception和Dark Knight，其他看过这两部电影的人也喜欢看Prestige，那么推荐给我Prestige是较好的选择。

>关于基于电影的协同过滤创建推荐系统，我之前写过一篇详细的博客。感兴趣的读者可以点此查看，并将推荐结果与基于用户的推荐结果进行比较。

For emerging OTT platforms, user-based CF is a quick-start way as they may not have resources to generate meta tags and label movies and instead would prefer the implicit user behavior who interact with their platform. For bigger players like Netflix, Youtube, Prime Video hybrid recommendations are the de-facto approach. For the purpose of this post, we are constraining ourselves to user-based filtering i.e defining a Movie by its user vectors. The data structure would look as below:

>对于新兴的OTT平台，由于没有资源生成元标签和标注电影，同时希望用户与平台进行隐式交互，因此基于用户的协同过滤（CF）方法是最适合的。至于像Netflix，Youtube这样的大公司，实际上使用Prime-Video混合推荐方法。在本文中，我们仅讨论基于用户的协同过滤，即通过用户矢量表征电影。数据集结构如下：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603335794748216.png)

Input Data from MovieLens

Similar to word2vec embedding, we train this shallow neural net based on User IDs, Once the model is trained we can find the ‘Tennis-Federer’ equivalent of user ids. Using the work so far, we can find look-alike users and probably divide users into clusters for targeted communication via Push Notifications/emailers. However more than user targeting, we are interested in movie recommendations which would require an additional step of comparing movies.

>MovieLens输入数据格式

>与word2vec嵌入类似，我们通过用户ID训练浅层神经网络。经过训练，我们在用户ID嵌入中实现类似“网球-费德勒”的关系。至此，我们可以找到相似的用户，并可以将用户分为几类，通过推送或电子邮件发送有针对性的讯息。除了基于相似用户的推荐方法，我们还可以根据电影进行推荐，这就需要对电影信息进行类似操作了。

In comes, sentence embedding, where the users would be treated as a sentence, and the semantic similarity of sentences would be captured. There has been a lot of action going in sentence embedding space with the entry of BERT, Elmo, Doc2Vec, and Universal Sentence Encoder. However, speed is of the essence here, and based on my individual benchmarking, I have considered [Fast_Sentence_Embeddings](https://github.com/oborchers/Fast_Sentence_Embeddings "Fast_Sentence_Embeddings")(FSE) as a go-to way.FSE creates a composite vector based on individual word2vec vector projections and we are not going to get any value additions by using pre-trained models on Wikipedia or Google News as the user ids are illegible for their SOTA systems. There are no existing semantic similarities to be captured by these models and thus we can safely consider the speed of our algorithm as a predominant deciding factor(FSE claims ~500K sentence processing per second while my personal experience with Universal sentence encoder has seen sub 100/sec speed).

Once we have sentence vectors, we can project them in multi-dimensional space,e.g “Obama speaks to the media in Illinois” would be similar to “The President greets the press in Chicago”

>接下来是句子嵌入，即将用户视为句子，捕获句子的语义相似性。通过BERT，Elmo，Doc2Vec和Universal Sentence Encoder等方式均可方便的实现句子嵌入，具体选择哪种方式主要取决于模型的速度，根据我的测试，Fast_Sentence_Embeddings（FSE）是我比较推荐的方法，该方法直接根据各词的word2vec生成复合向量。对于BERT这样的SOTA系统而言，用户ID很难表现出语义相似性，因此，即使使用经过Wikipedia或GoogleNews预训练的模型，也不会有太大差异，这样，模型的速度就成为唯一的衡量标准。FSE每秒可处理约500K句子，而预训练编码模型低于每秒100句。

>之后，将句子向量投影到多维空间中。例如，"Obama speaks to the media in Illinois" 和"The President greets the press in Chicago"表达的意思相近，在多维空间中距离就比较近。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603335794227892.png)

Context-based sentence similarity

Code for creating movie embeddings using Movielens data:

>基于上下文的句子相似性

>Movielens数据的电影嵌入代码：

```
from datetime import datetime
from gensim.models import Word2Vec
from fse import IndexedList
#Data Inputs
movies=pd.read_csv('ml-latest/movies.csv')
links=pd.read_csv('ml-latest/ratings.csv')
links=links.sample(frac=0.05)
links['movieId']=links['movieId'].astype('str')
links['userId']=links['userId'].astype('str')
def combine(kw):        
a=1        
kw=kw.tolist()        
size=len(kw)        
base=[]        
for i in range(0,size):            
#print(kw[0][i].split(" "))            
string=kw[i].split(",")            
#print(string)            
base=base+string                
if(len(base)>2):            
base = ','.join(base)        
return base
s_test=links
s_test=s_test[['userId','movieId']].drop_duplicates()
s_test['keywords_comb']=s_test.groupby(['movieId'])['userId'].transform(combine)
print("Start of Word2Vec",datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
data=s_test[['movieId','keywords_comb']].drop_duplicates()
data['word_count'] = data.keywords_comb.str.count(',')+1
data=data[data['word_count']>5]
data['tokenise'] = data['keywords_comb'].astype('str').apply(lambda x: [x for x in x.split(',')])
a1=data['tokenise'].tolist()
model_items = Word2Vec(a1, size=100)
#data=data.dropna()
print("Start of Sentence To vec",datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
sent=data['keywords_comb'].astype('str').apply(lambda x: [x for x in x.split(',')]).tolist()
s = IndexedList(sent)
from fse.models import uSIF
model = uSIF(model_items, lang_freq="en")
model.train(s)
```

Once we have the movie embeddings, we will be using a combination of tensorboard projector and Github to traverse the last mile of deployment, Special thanks to Google for open sourcing their tensorboard projector.

Step by step process to creating an amazing portal using embeddings:

1. Upload the embeddings data to a GitHub gist and update it in the config JSON. For starters, you can freely use my embeddings hosted here.

2. Upload the metadata on a Github gist so that the bubbles have a label

3. Create a JSON file and provide the URL to Tensorflow, Sample JSON config

>完成影片嵌入后，我们将结合tensorboard和Github完成电影推荐系统的部署。特别感谢Google对tensorboard开源的工作。

>使用嵌入结构构建web步骤如下：

>1.将嵌入数据上传到GitHub gist，并在config JSON中进行更新。对于初学者，可以直接使用我在此托管的嵌入数据。

>2.将元数据上传到Github gist上，以便图示节点带有标签。

>3.创建一个JSON文件并将URI提供给TensorflowL，见JSON配置示例。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603335797236277.png)

Input Config file location from Github

Deploy the web app and voila!

Webapps deployed using the above steps:

a.[ Floating recommender without images](https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/ravi07bec/4b033bc6299b655a2abda8b441e82364/raw/715f5656ae5dc2753f62ce4e9f89e7cb88f4e76e/test2.json " Floating recommender without images") and b. [Floating recommender with images](https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/ravi07bec/4b033bc6299b655a2abda8b441e82364/raw/915e0dbc1c69b6e9d31100c4967249a7a2548e85/test2.json "Floating recommender with images")

>Github输入配置文件

>4.部署Web应用程序！

>web应用包括以下两种：a. Floating recommender without images and b. Floating recommender with images

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603335798916079.png)

Webapp and Reccos for Ratatouille

>Ratatouille示例

Play around with projector, click on a movie, find closest movies, Project movie vectors using TSNE, or PCA.

Summary: We have learned about processing a cross-grid of Movie lens data, concepts behind vector embeddings(User IDs in this case), sentence similarity matrix using FSE, creating a sprite Image, and deploying a DIY projector using Google’s TensorFlow. Snapshot from final portal below

>5.使用tensorboard可视化交互，单击某一电影即可查找类似的电影，还提供TSNE或PCA降维操作。

>总结：本文介绍了如何处理Movielens数据集，矢量嵌入（文中指用户ID），基于FSE的句子相似度矩阵以及使用Google的TensorFlow部署web电影推荐应用。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603335807921635.png)

Recommender portal with Movie Posters

I would be happy to see your DIY web apps on movies or related datasets, feel free to hit me up here or Linkedin, Happy to chat if you face any issues/improvements.

Related datasets which can be used for recommendations:

1. Books- Kaggle https://www.kaggle.com/zygmunt/goodbooks-10k

2. E-Commerce product purchases-https://www.kaggle.com/c/instacart-market-basket-analysis

3. Restaurants-https://towardsdatascience.com/how-to-build-a-restaurant-recommendation-system-using-latent-factor-collaborative-filtering-ffe08dd57dca

4. Cuisines: https://github.com/hasan-kamal/Cuisine-Prediction

>带有电影海报的推荐页面

>希望读者自行在电影或其他相关数据集上DIY自己的Web应用程序。如果您遇到任何问题或有其他有趣的想法，欢迎交流。

>其他可用于推荐的数据集：

>1. 图书 https://www.kaggle.com/zygmunt/goodbooks-10k

>2. 电子商务产品 https://www.kaggle.com/c/instacart-market-basket-analysis

>3. 餐厅 https://towardsdatascience.com/how-to-build-a-restaurant-recommendation-system-using-latent-factor-collaborative-filtering-ffe08dd57dca

>4. 美食 https://github.com/hasan-kamal/Cuisine-Prediction

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://flyai.com/article/743

讨论

500字

表情

发送

删除确认

是否删除该条评论？

取消删除