Nashville Machine Learning Meetup 2015 Kickoff, part 1 of our Foundations of Supervised Machine Learning series.

Published on: **Mar 3, 2016**

Published in:
Data & Analytics

- 1. Naïve Bayes for the Superbowl John Liu Nashville Machine Learning Meetup January 27th, 2015
- 2. NashML Goals Create a hub for like-minded people to come together, share knowledge and collaborate on interesting domains. Experienced MLers ML Hackers Topic Presentations Collaborative Practice Common Platform
- 3. Platform ! IPython Notebook (Project Jupyter) ! Java, Scala, Python (others?) ! Scikit-learn ! PyLearn2/Theano ! iTorch ! AWS/Mahout/Spark/Mllib?
- 4. Rev. Thomas Bayes “An Essay towards solving a Problem in the Doctrine of Chances” published posthumously 1763 I now send you an essay which I have found among the papers of our deceased friend Mr. Bayes, and which, in my opinion, has great merit, and well deserves to be preserved. Experimental philosophy, you will ﬁnd, is nearly interested in the subject of it; and on this account there seems to be particular reason for thinking that a communication of it to the Royal Society cannot be improper…
- 5. Bayes’ Theorem P(A | B)P(B) = P(A∩B) = P(B | A)P(A) P(A | B) = P(A∩B) P(B) P(A | B) = P(B | A)P(A) P(B)
- 6. Statistical Inference P(A | B) = P(B | A)P(A) P(B) Likelihood Prior EvidencePosterior (What we want to find)
- 7. Intuition Behind Bayes A Priori Initial Belief (model) Evidence See the Data Likelihood How likely to see Data given Belief A Posteriori Updated Belief after seeing Data posterior = prior•likelihood evidence
- 8. Example Blackjack Insurance Bet: What is the probability that a dealer with an Ace showing has Blackjack?
- 9. Example Prior P(Dealer has Blackjack) 32/663 Evidence P(Ace showing) 1/13 Likelihood P(Ace showing|has Blackjack) 1/2 Posterior P(Blackjack|Ace showing) 16/51 16/51 = 31% or less than 1/3 of the time.
- 10. Are you a Bayesian? You read Burton Malkiel’s book “A Random Walk down Wall Street” and believe in the Efficient Market Hypothesis. Your broker gives you a tip to buy Tesla. You ignore the broker and Tesla rises 100 days in a row. As a Bayesian, do you believe: A) The stock is long due for a correction B) It is possible for Tesla to rise another 100 days in a row C) You were fooled by randomness
- 11. Naïve Bayes You observe outcome Y with some n features X1, X2, ..Xn. The joint density can be expressed using the chain rule: P(Y, X1, X2, ..Xn) = P(X1, X2, ..Xn|Y) P(Y) = P(Y) P(X1,Y) P(X2|Y,X1) P(X3|Y,X1,X2)… This is messy, but simplifies if we naively assume independence, P(X2|Y,X1) = P(X2|Y) P(X3|Y,X2,X1) = P(X3|Y) P(Xn|Y,Xn…X2,X1) = P(Xn|Y) Naïve Bayes Assumption
- 12. Naïve Bayes Classifier Let K classes be denoted ck. The (conditional) probability of class ck given that we observed features x1, x2,..xn is: A Naïve Bayes classifier simply chooses the class with highest probability (maximum a posteriori): P(ck | x1, x2, ..xn ) = P(ck ) P(xi | ck ) i=1 n ∏ cNB = argmax k∈K P(ck ) P(xi | ck ) i=1 n ∏
- 13. Gaussian Naïve Bayes When features xi are continuous valued, typically make the assumption they are normally distributed: Variance σki can be independent of xi and/or ck. P(xi | ck ) = 1 2πσki 2 e − 1 2 xi−µki σki " # $ % & ' 2 cNB = argmax k∈K P(ck ) P(xi | ck ) i=1 n ∏
- 14. Multinomial Naïve Bayes When features xi are the number of occurrences of n possible events (words, votes, etc…) pki = probability of i-th event occuring in class k xi = frequency of i-th event The multinomial Naïve Bayes classifier becomes: cNB = argmax k∈K logP(ck )+ xi •log pki i=1 n ∑ # $ % & ' (
- 15. Naïve Bayes Intuition ! Assumes all features are independent with each other ! Independence assumption decouples the individual distributions for each feature ! Decoupling can overcome curse of dimensionality ! Performance robust to irrelevant features ! Very fast, low storage footprint ! Good performance with multiple equally important features
- 16. Naïve Bayes Applications ! Document Categorization ! NLP ! Email Sorting ! Collaborative Filtering ! Sports Prediction ! Sentiment Analysis
- 17. Example: Doc Classification Want to classify documents into k topics. Document d consisting of words wi is assigned to the topic cNB: P(ck) = topic frequency = P(wi|ck) = word wi frequency in all topic ck docs cNB = argmax k∈K P(ck ) P(wi | ck ) i ∏ Ndocs(topic=ck ) Ndocs = Nword=wi (topic=ck ) Nword=wi (topic=ck ) i ∑
- 18. Laplace (add-1) Smoothing What happens with P(wi|ck) = 0 for a particular i, k? Solution is to add 1 to numerator & denominator: P(ck | x1, x2, ..xn ) = P(ck ) P(xi | ck ) i=1 n ∏ = 0! P(wi | ck ) = Nword=wi (topic=ck ) +1 Nword=wi (topic=ck ) +1( ) i ∑
- 19. NBC Application Roadmap ! Read Dataset ! Transform Dataset ! Create Classifier ! Train Classifier ! Make Prediction
- 20. Sports Prediction Who is favored to win the Superbowl? Given the 2014 season game statistics for two teams, how can we make a prediction on the outcome of the next game using a Naïve Bayes classifier?
- 21. Sentiment Analysis Which team is most favored in each state? What if we analyzed tweets by sentiment and location using a Naïve Bayes Classifier?
- 22. Starter Code Repo with Starter Code at: https://github.com/guard0g/NaiveBayesForSuperbowl IPython notebook: NB4Superbowl.ipynb Datasets: SeattleStats.csv NewEnglandStats.csv