Movies Clustering¶
Frame the Problem¶
Can we cluster movies that go together based on
- Name
- Genre
- Director
- Actor
- Year of Release
- Running Length
- Language
- Production House
- Ratings
Acquire the Data¶
IMDB for Western Movies
In [80]:
import pandas as pd
import numpy as np
In [54]:
url = "https://notes.pipal.in/2018/airwatch-ml/IMDB-Movie-Data.csv"
In [55]:
df = pd.read_csv(url)
In [56]:
df.columns
Out[56]:
In [57]:
df.shape
Out[57]:
In [58]:
colnames = ['Rank', 'Title', 'Genre', 'Description', 'Director', 'Actors', 'Year',
'Runtime', 'Rating', 'Votes', 'Revenue', 'Metascore']
In [59]:
df.columns = colnames
In [60]:
df.head()
Out[60]:
In [61]:
df.isnull().sum()
Out[61]:
In [62]:
df.Metascore.mean()
Out[62]:
In [63]:
import matplotlib.pyplot as plt
%matplotlib inline
In [64]:
df.corr()
Out[64]:
In [65]:
plt.scatter(df.Metascore, df.Rating)
Out[65]:
In [66]:
df_clean = df[['Rating', 'Metascore']].dropna()
In [67]:
from sklearn.linear_model import LinearRegression
metascore_model = LinearRegression()
metascore_model.fit(df_clean[['Rating']], df_clean.Metascore)
Out[67]:
In [68]:
metascore_model.coef_, metascore_model.intercept_
Out[68]:
In [69]:
df2 = df.copy(deep=True)
In [70]:
missing_indices = df2.Metascore.isnull()
df2['Metascore'][missing_indices] = metascore_model.predict(df2[['Rating']][missing_indices])
In [71]:
df2.isnull().sum()
Out[71]:
In [72]:
df.isnull().sum()
Out[72]:
In [73]:
df2.head()
Out[73]:
Transform the Data¶
In [100]:
# Number of Genres
df.Genre.str.split(",", expand=True).stack().unique().shape
Out[100]:
In [112]:
# One-Hot Encoding for Genres
genre = (df.Genre
.str
.split(",", expand=True)
.stack()
.str
.get_dummies()
.sum(level=0)
)
In [117]:
# Scale the Continuous Values
from sklearn.preprocessing import StandardScaler
In [118]:
df_cont = df2[["Runtime", "Rating", "Votes", "Metascore"]]
In [119]:
df_cont.head()
Out[119]:
In [121]:
SS = StandardScaler()
SS.fit(df_cont)
Out[121]:
In [122]:
df_cont_scaled = SS.transform(df_cont)
In [124]:
df_cont_scaled
Out[124]:
Model the Data¶
Simliar clusters of movies based on
- One-Hot Genres (20)
- Runtime
- Rating
- Votes
- Metascore
In [126]:
X = np.c_[np.array(genre), df_cont_scaled]
In [128]:
X.shape
Out[128]:
In [129]:
from sklearn.cluster import KMeans
In [130]:
model_km = KMeans()
In [131]:
model_km.fit(X)
Out[131]:
In [135]:
model_km.labels_
Out[135]:
In [139]:
df2[model_km.labels_ == 1].sort_values(by="Rating", ascending=False).head(10).Title
Out[139]:
In [189]:
def get_top_movies(cluster_id):
movies = (df2[model_km.labels_ == cluster_id]
.sort_values(by="Rating", ascending=False)
.head(10))
return [{"id": i, "title": movies.loc[i].Title, "director": movies.loc[i].Director}
for i in movies.index]
In [140]:
x = np.array([1, 2, 3, 4])
In [144]:
y = np.array(["A", "B", "B", "A"])
In [141]:
x[[1, 2]]
Out[141]:
In [142]:
x[[True, False, False, True]]
Out[142]:
In [145]:
x[y == "A"]
Out[145]:
In [147]:
get_top_movies(3)
Out[147]:
Q: How to get the cluster id?
In [152]:
X[:5]
Out[152]:
In [153]:
model_km.predict(X[:5])
Out[153]:
In [154]:
df2.head()
Out[154]:
In [161]:
get_top_movies(3)
Out[161]:
How to find the related movies given a movie?
In [162]:
def get_related_movies(movie_id):
cluster_id = model_km.labels_[movie_id]
return get_top_movies(cluster_id)
In [192]:
df2.head()
Out[192]:
In [193]:
get_related_movies(0)
Out[193]:
Save model¶
Save the data frame first.
In [194]:
df2.to_csv("movies.csv", index=False)
How to save the model?
In [195]:
import joblib
In [196]:
joblib.dump(model_km, "model_km.model")
Out[196]:
Let us move the required code to a python file.
In [201]:
%%file movies.py
import pandas as pd
import joblib
import sys
df = pd.read_csv("movies.csv")
model_km = joblib.load("model_km.model")
def get_top_movies(cluster_id):
movies = (df[model_km.labels_ == cluster_id]
.sort_values(by="Rating", ascending=False)
.head(10))
return [{"id": i, "title": movies.loc[i].Title, "director": movies.loc[i].Director}
for i in movies.index]
def get_related_movies(movie_id):
cluster_id = model_km.labels_[movie_id]
return get_top_movies(cluster_id)
def main():
movie_id = int(sys.argv[1])
print(get_related_movies(movie_id))
if __name__ == "__main__":
main()
In [202]:
!python movies.py 2
Running as a Service¶
Install firefly.
pip install firefly-python
And run the following in your terminal.
firefly movies.get_related_movies
That would start that function as an API.
Once it is running, you can use it as an API.
In [203]:
import firefly
In [204]:
api = firefly.Client("http://127.0.0.1:8000/")
In [205]:
api.get_related_movies(movie_id=5)
Out[205]: