1. A Tutorial on ML Basics and Embedding Chong Ruan ruanchong_ruby@163.com

2. Machine Learning Basics What is machine learning? Traditionally: Input + Model => Output What ML does: Input (+ Output) => Model

3. Machine Learning Basics Example: How to write a classifier which can distinguish between a watermelon and an apple? The traditional way: Use hand-crafted features and manually code how they are related to the expected output Feature: express a sample as some values (typically a vector) E.g.: The color, shape, weight of a fruit If (weight > 2kg) print “Watermelon”; else print “Apple”. Difficulty Sometimes the relation between inputs and outputs are too complicated to be formulated Say, how to write a program to recognize a cat?

4. Machine Learning Basics Example: How to write a classifier which can distinguish between a watermelon and an apple? The ML way: Collect some data with labels, say we have: (7.5, watermelon), (8.1, watermelon), (0.5, apple), (0.6, apple) Propose a hypothesis (model space): If (weight > T) print “Class A”; else print “Class B” (T: unknown parameter) You may want to use a more complicated hypothesis for some more difficult tasks Training/Learning the model: Typically achieved by optimize an objective function Find the best threshold T, such that most samples are classified correctly.

5. Machine Learning Basics The difficulty of the ML approach: How to collect (labelled) data What kind of hypothesis to use … The first (and the most important) step is: How to choose features

6. Machine Learning Basics Look back on the fruit example aforementioned: How to represent a fruit: Its weight, color, flavor, size, etc. Feature selection Which subset of features is useful for our purpose (classification)? In this example, weight (or/and size) is a great choice (due to our prior knowledge)

7. Machine Learning Basics More on feature selection: Sometimes we do not need to select (or construct) proper features Just feed all available features to a model Let the model learn automatically! But too many feature may: slow down the training and predicting procedure Cause model to overfit or misfit the dataset So we want to reduce the dimension of the collected data…

8. Machine Learning Basics Examples of huge feature set: In NLP: Use TF-IDF feature to express a document In music recommendation: Use user-music matrix to predict users’ preference Typically a really huge matrix …

9. Dimension Reduction PCA: Principle Component Analysis A well-known dimension reduction algorithm An example:

10. Dimension Reduction PCA: Principle Component Analysis A well-known dimension reduction algorithm An example:

11. Dimension Reduction PCA: Principle Component Analysis Algorithm:

12. Dimension Reduction PCA: Principle Component Analysis For the aforementioned example: Matrix X: Mean value of each dimension is already 0 Covariance matrix C:

13. Dimension Reduction PCA: Principle Component Analysis For the aforementioned example: Eigenvalue decomposition: Choose top k rows from the matrix P and left multiply X: In this case k=1

14. Dimension Reduction

15. Embedding

16. Graph embedding Suppose you have a graph with m vertices Each vertex represents a data point (say, a person, a word, etc.) And a similarity matrix W Wij measures to what extent vertex i and j are similar Say, # of mutual friends, mutual information, etc. Use the row it self to represent each sample is too cumbersome Say, a social network may have millions of users a corpus may have tens of thousands of words The purpose: Assign a low dimensional vector to each vertex of the graph that preserves similarities between the vertex pairs

17. Embedding Word embedding (Word vector) Suppose you have a corpus The pattern words occur in the corpus reflect their meanings Similar/Related words are likely to appear together The purpose: Assign a low dimensional vector to each word which preserves the information of the word Say, its polarity, grammatical function, semantic property, etc. If two words are similar, there word embeddings are similar

18. Embedding The advantages of embedding: Compact Word embedding vs. one-hot expression Handy for numerical operation Say, calculate similarity between samples Easy to visualize Use PCA/LDA/… to transform data points to a 2/3 dimensional space and plot them

19. Embedding Obtain embeddings with neural networks: Wishful thinking: Consider: how do you write a recursive function? Suppose we already have an embedding (randomly initialization or use some heuristics) Set a proper objective function and optimize it The embeddings will converge to a reasonable positions An example: PageRank When considering a (probabilistic) model, be sure to distinguish between: Representation (Modelling) Learning (Training) Inference (Predicting)

20. Neural Networks (Modelling) What is a neural network: A special kind of function Inspired by neurons and their connection To recap, what is ML: Collect data Propose a hypothesis Fit the model to the data Optimize a objective function Say, minimize the misclassification rate (for a classification problem), minimize intra-cluster distance while maximize inter-cluster distance (for a clustering problem), minimize reconstruction error (representation learning), etc.

21. Neural Networks (Modelling)

23. Illustration (Photo credit: Andrew Ng):

24. Neural Networks (Modelling)

26. Validation: What if the relation between inputs and outputs are not of this form? No need to worry: A 3-layer neural network is a universal approximator, can approximate any continuous function within any accuracy Analogy: If you have some 2-dimensional data and you want to fit them You can always use a polynomial Any accuracy can be achieved given the degree of the polynomial is high enough So we can always use a neural network (of more than 3 layers) to fit any data Say, the function that predict the next word given its previous words, if exists

27. Neural Networks (Training) Set an objective function (cost function) Say, mean square error, cross entropy, etc. Find a group of parameters which can best interpret the data Numerical optimization methods, typically gradient descent Need to compute cost function and its gradient

28. Neural Networks (Training) Gradient descent (illustration)

29. Neural Networks (Training) Calculate the cost function Forward Propagation z: value before activation a: value after activation

30. Neural Networks (Training)

31. Calculate the gradient of the objective function Back Propagation: the chain rule

32. Neural Networks (Training)

34. Remark on Back Propagation Intuition: Propagate error over the graph A more formal and neat way to understand Back Propagation: Then use the chain rule

35. Neural Networks (Training) Take-home message: Modelling: Propose a hypothesis that the relation between the inputs and outputs can be approximated by a special kind of function with parameters Training: Find a group of parameters which can interpret the observed data best Achieved by optimization algorithms, typically gradient descent Gradient can be computed efficiently by Back Propagation (The Chain Rule) Use random initialization (not all zeros!)

36. Neural Networks (Inference) Given a new example, feed it to the trained neural network Use forward propagation to compute the output Take the output as your prediction

37. Neural Networks (Application) Multiple classification

38. Neural Networks (Application) What is flexible: The network architecture

39. Neural Networks (Application)

40. Neural Networks in NLP How to express a word as a vector: A unique integer as word ID: Can be used for indexing Not a semantic embedding, because improper order is introduced One-hot expression Use a vector of length |V| to represent a word Only one component is 1, the rest are 0 Compare: 酸，甜，苦，辣 不辣，微辣，中辣，特辣

41. Neural Networks in NLP Distributed expression Express a word as an n-dimensional real vector (more on this later) Pros: Compact: a dimension of several hundred works well Can code semantic property of words: similar words have similar vectors Cons Hard to interpret the “meaning” of its components

42. Neural Networks in NLP How to obtain word vector? Hypothesis: Words in a sentence are not chosen randomly There is a formula that: Given first n words, this formula gives the distribution of next word A.k.a. Language Model (which models the probability of a given sentence) And the formula can be approximated by a neural network A.k.a. neural network language model (NNLM)

43. Neural Networks in NLP More on language model: Suppose we have: 我 爱 … What should be the next word? P( 你 ) = 0.003 P( 北京 ) = 0.00065 P( 睡觉 ) = 0.00087 P( 我 ) = 3.8e-9 P( 在 ) = 1.2e-10 … Useful for machine translation, input method, spell checking, etc.

44. Neural Networks in NLP

45. Neural network language model: Use a neural network to define the distribution of the next word The network structure is flexible (up to you)

46. Neural Networks in NLP

47. An example (2003, Bengio): Hidden layer: concatenate the input word embeddings

48. Neural Networks in NLP

49. An example (2003, Bengio): Objective function: Maximize the likelihood of the corpus

50. Neural Networks in NLP Don’t be too afraid of it Keep in mind: A neural network is a special kind of function We want to find a group of parameters to optimize the objective function Now we have two kinds of parameters: network weight and word embeddings The learning algorithm: Randomly initialization Use gradient descent to update parameters Note: include both network weight and word embeddings Use back propagation to calculate gradients Pros: Automatically smoothing

51. Neural Networks in NLP Visualize word embeddings using t-SNE: http://metaoptimize.s3.amazonaws.com/cw-embeddings- ACL2010/embeddings-mostcommon.EMBEDDING_SIZE=50.png http://metaoptimize.s3.amazonaws.com/cw-embeddings- ACL2010/embeddings-mostcommon.EMBEDDING_SIZE=50.png

52. Neural Networks in NLP Visualize word embeddings using t-SNE: http://metaoptimize.s3.amazonaws.com/cw-embeddings- ACL2010/embeddings-mostcommon.EMBEDDING_SIZE=50.png http://metaoptimize.s3.amazonaws.com/cw-embeddings- ACL2010/embeddings-mostcommon.EMBEDDING_SIZE=50.png

53. Neural Networks in NLP Visualize word embeddings using t-SNE: Bilingual embedding Socher et al. 2013

54. Neural Networks in NLP A more efficient network Word2vec (Mikolov et al., 2013) Continuous bag of words (CBOW) + hierarchical softmax (HS)

55. Neural Networks in NLP CBOW Given context, predict current word

56. Neural Networks in NLP

57. HS An example: For leaf “ 足球 ”: Then:

58. Neural Networks in NLP HS Use the Huffman tree: Intuitively, frequent words should have larger probability So make their paths shorter

59. Neural Networks in NLP The whole model

60. Neural Networks in NLP Learning procedure: SGD (details are omitted) Quite efficient Computation cost from the hidden layer to the output layer are reduced greatly Open source: For Linux: http://word2vec.googlecode.com/svn/trunk/http://word2vec.googlecode.com/svn/trunk/ Works on Mac, with minor modification For windows: https://github.com/jdeng/word2vechttps://github.com/jdeng/word2vec Require C++11 support See http://blog.csdn.net/heyongluoyao8/article/details/43488765 for referencehttp://blog.csdn.net/heyongluoyao8/article/details/43488765

61. Neural Networks in NLP Something more appealing: Analogy property Man – Woman = King – Queen = …

62. Something more crazy Embedding words and images into the same space Socher et al. 2013

63. Something more crazy Embedding sentences/paragraphs/documents… Cho et al. 2014

64. Something more crazy Embedding sentences/paragraphs/documents… Cho et al. 2014

65. The End Thanks for your listening!