1. Recurrent Neural Networks: In Class Exercises BMI 707 Andrew Beam, PhD Department of Biomedical Informatics Harvard Medical School 04/16/2020 @AndrewLBeam

2. OUTLINE This week’s goals: Today: Overview of RNNs, language models, and word embeddings Thursday: Worked example of RNNs for text classification Image source: Wikipedia and LeCun et. al (1989)

3. NEWSGROUPS CLASSIFICATION

4. 20 NEWS GROUPS Post Category? 'alt.atheism': 'comp.graphics': 'comp.os.ms-windows.misc': 'comp.sys.ibm.pc.hardware': 'comp.sys.mac.hardware’: 'comp.windows.x': 'misc.forsale': 'rec.autos': 'rec.motorcycles': 'rec.sport.baseball': 'rec.sport.hockey': 'sci.crypt': 'sci.electronics': 'sci.med': 'sci.space': 'soc.religion.christian': 'talk.politics.guns': 'talk.politics.mideast': 'talk.politics.misc': 'talk.religion.misc':

5. LSTM IN KERAS Implement a RNN in keras to predict the category in the 20 news groups dataset. Colab notebook: https://bit.ly/2H2fJsQ

6. TRANSFER LEARNING This dataset isn’t huge, so can we use other data to help? Yes! We can use pre-trained word embeddings that were trained on large data sources. Use pre-trained embeddings in your RNN. Colab notebook: https://colab.research.google.com/drive/1JsdhsiJQP5JPEEG WWFtOMpQajBj4w1KA

7. CNNS FOR TEXT Can CNNs solve this problem in an equivalent way? Yes! Our text is now an (s x d) matrix, where s is the sequence length and d is the embedding dimension.

8. CNNS FOR TEXT Image Source: http://www.aclweb.org/anthology/D14-1181

9. CNNS FOR TEXT Dilated convolutions can learn long range dependencies

10. CNNS FOR TEXT Best practices are quickly evolving Reference: https://arxiv.org/abs/1803.01271

11. CNNS FOR TEXT Image Source: http://www.aclweb.org/anthology/D14-1181 Use a CNN to solve this task. Colab notebook: https://bit.ly/2J5RPgF