1. Sentence Modeling
Representation of sentences is the heart of Natural Language Processing
A sentence model is a representation and analysis of semantic content of a sentence for classification or generation
The sentence modeling task is at the core of many tasks such as sentiment analysis, paraphrase detection, entailment recognition, summarization, discourse analysis, machine translation, grounded language learning and image retrieval
The aim of sentence modeling is a feature function that guides the process by which features of a sentence are extracted.

2. One Dimensional Convolution

3. M-gram Dot product Vector of weights Size: m Filter Input Sequence
Sentence

4. Produces a sequence c

5. Narrow Convolution
Size of c : s – m + 1 It requires that s ≥ m

6. Wide Convolution
Size:
s + m - 1

7. Wide Convolution Size of c s+m-1 No requirement on s or m
Out of range values are taken to be 0
Result of narrow convolution is subsequence of result of wide convolution

8. Advantages of Wide Convolution
Guarantees that a valid non empty c will always be produced
All weights in the filter reach the entire sentence
Holds no limit on the size of m or s

9. Time Delay Neural Network
A key feature for TDNN’s are the ability to express a relation between inputs in time.
The sequence s is viewed as having a time dimension and the convolution is applied over the time dimension.

12. Max TDNN

13. Properties of Max TDNN Sensitive to order of the words
Does not depend on external language specific features
Largely uniform importance to the signal from each of the words
Range of feature detectors is limited
Higher order and long range feature detectors cannot be incorporated
Multiple occurrences of features and the sequence ignored
Pooling factor: s-m+1

14. k-Max Pooling

15. k – Max Pooling
Given a value k and a sequence P of length p, k-max pooling selects the subsequence p-max of the k highest values of p.
The order of the values in p-max corresponds to the original order in p.

16. k-Max Pooling k most active features
Features may be number of positions apart
Preserves the order of the features
But is insensitive to their specific positions
Can detect multiple occurrences of feature

17. What should k be?
Why not let it decide for itself?

18. Dynamic k-Max Pooling Suppose length of sentence = 18 L = 3 Ktop = 3

19. Second Order Feature Map
Multiple Feature Maps
Feature Map
Second Order Feature Map
Convolution
K-max Pooling Layer
Non linear function

20. To increase the number of learnt feature detectors of a certain order, multiple feature maps
may be computed in parallel at the same layer.

21. Folding
Feature detectors in different rows are independent of each other.

22. Properties of Sentence Model
The subsequence of n-grams extracted by the pooling operation induces invariance to absolute positions, but maintains their order and relative positions.
DCNN feature graphs have a global range of the pooling operations
DCNN has internal input dependent structure and does not rely on externally provided parse trees.

23. Experiments

24. Sentiment Prediction in Movie Reviews
Concerns prediction of sentiment of movie reviews in Stanford Sentiment Treebank
Output is binary in experiment 1 and “negative, somewhat negative, neutral, somewhat positive, positive” in experiment 2
Binary: MultiClass:

26. Question Type Classification
TREC question dataset
Six Different Question Types

28. Twitter Sentiment Prediction with Distant Supervision
Large dataset of tweets
Tweet is labelled positive or negative automatically based on emoticon
Tweets are preprocessed

30. Conclusion Dynamic CNN defined, which uses Dynamic k-max Pooling
Feature Graph captures word relation of varying size
High performance on sentiment prediction and question classification without requiring external features