ANLP, Spring 2019 – Elena Filatova

Elena Filatova, efilatova@citytech.cuny.edu

Office: GC 4410

Class is held in room: 7395

Tools, Books, Related Courses

Grading Policy

Class (tentative) schedule:

	Topic	Reading Assignment
1. Jan. 28	Introduction Overview: classification, feature engineering, NLP problems casted as classification problems (text classification, POS tagging, sentiment analysis, etc.). Why deep learning? Why now? How is deep learning different from previously popular classification techniques? Lecture notes	Deep learning by Yann LeCun, Yoshua Bengio, Geoffrey Hinton . In Nature volume 521, pages 436–444 (28 May 2015)
2. Feb. 4	Neural nets: neuron analogy, one-layer NN as logistic regression classification, forward backward propagation. Lecture notes	Sebastian Raschka: What is the relation between Logistic Regression and Neural Networks
3 Feb. 11	Gradient decent, activation functions, regularization, loss functions, vectorization for IPython notebooks Lecture notes Assignment 1 is out	Ian Goodfellow and Yoshua Bengio and Aaron Courville Deep Forward Networks
4 Feb. 25	Word Vector representation: one-hot representation, word2vec, GloVe, embeddings Notes on Information Extraction (IE) snowBallPresentation Lecture notes	Word vectors: Distributed representations of words and phrases and their compositionality Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean Proceeding NIPS’13 Proceedings of the 26th International Conference on Neural Information Processing Systems Volume 2, Pages 3111-3119 (Shweta) Efficient Estimation of Word Representations in Vector Space Mikolov, Tomas and Chen, Kai and Corrado, G.s and Dean, Jeffrey Proceedings of Workshop at ICLR, 2013 (Qihui) Compare these papers and the SnowBall paper: Snowball: extracting relations from large plain-text collections Eugene Agichtein, Luis Gravano Proceeding DL’00 Proceedings of the fifth ACM conference on Digital libraries Pages 85-94 GloVe GloVe: Global Vectors for Word Representation Jeffrey Pennington and Richard Socher and Christopher D. Manning Empirical Methods in Natural Language Processing (EMNLP) 2014, pages = 1532–1543 (Kiwook)
5 Mar. 11	TensorFlow, Keras discussion, discussion of possible projects	Distributed Representations of Sentences and Documents, Le and Mikolov, ICML’14 Proceedings of the 31st International Conference on International Conference on Machine Learning (Xiaomeng) Deconfounded Lexicon Induction for Interpretable Social Science Reid Pryzant et al. 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Mingyan)
6 Mar. 18	RNN, CNN Assignment 2 is out (on Mar. 15) Project description	Visualizing and Understanding Recurrect Networks Karpathy et al, ICLR 2016 (Zhaoheng ) RNN effectiveness (Karpathy’s blog)
7 Mar. 25	RNN, CNN Lecture notes Lecture notes 2	Opinion Mining with Deep Recurrent Neural Networks Irsoy, Cardie EMNLP, 2014 (Sara) A Convolutional Neural Network for Modelling Sentences Nal Kalchbrenner, ACL 2014
8 Apr. 1	Project discussion (presentation of the task)
9 Apr. 8	Deep learning for classification Lecture notes 3 Machine Translation NMT (Stanford lecture)	Sequence to Sequence Learning with Neural Networks Ilya Sutskever et al, NIPS 2014 (Yeonju ) Neural Summarization by Extracting Sentences and Words Jianpeng Cheng Mirella Lapata, ACL 2016 Philipp Koehn, Statistical Machine Translation, 2017
10 Apr. 15	Speech recognition Assignment 3 is out (due May 4)	Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups Geoffrey Hinton et al (Anthony) Towards an Unsupervised Entrainment Distance in Conversational Speech using Deep Neural Networks Md Nasir, Interspeech 2018 (Andreas) Listen, Attend and Spell William Chan, Navdeep Jaitly, Quoc V. Le, Oriol Vinyals. IEEE 2016, (Hassan)
11 Apr. 29	WSD problem and Deep Learning Generative Adversarial Nets (GAN)	(Mehrdad) Towards End-to-End Speech Recognition with Recurrent Neural Networks Alex Graves, et al. ICML 2014 (Viet Anh) context2vec: Learning Generic Context Embedding with Bidirectional LSTM. Oren Melamud et al. SIGNLL Conference on Computational Natural Language Learning (CoNLL), 2016
12 May 6	Text generation Jiwei Li’s presentation on Deep Learning in Open Domain Dialogue Generation	Deep Reinforcement Learning for Dialogue Generation Jiwei Li et al. EMNLP 2016
13 May 13	Additional topics
14 May 15 (instead of the March 4 “snow day”)	Final project presentations, wrap-up	Sara Shweta and Kiwook Fuxian Li Andreas and Zhaoheng
15 May 20	Project presentations, wrap-up

Topic

Reading Assignment

Jan. 28

Introduction

Overview: classification, feature engineering, NLP problems casted as classification problems (text classification, POS tagging, sentiment analysis, etc.).

Why deep learning? Why now? How is deep learning different from previously popular classification techniques?

Lecture notes

Deep learning by Yann LeCun, Yoshua Bengio, Geoffrey Hinton . In Nature volume 521, pages 436–444 (28 May 2015)

Feb.

Neural nets: neuron analogy, one-layer NN as logistic regression classification, forward backward propagation.

Lecture notes

Sebastian Raschka: What is the relation between Logistic Regression and Neural Networks

Feb. 11

Gradient decent, activation functions, regularization, loss functions, vectorization for IPython notebooks

Lecture notes

Assignment 1 is out

Ian Goodfellow and Yoshua Bengio and Aaron Courville Deep Forward Networks

Feb. 25

Word Vector representation: one-hot representation, word2vec, GloVe, embeddings

Notes on Information Extraction (IE)

snowBallPresentation

Lecture notes

Word vectors:

Distributed representations of words and phrases and their compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean
Proceeding NIPS’13 Proceedings of the 26th International Conference on
Neural Information Processing Systems
Volume 2, Pages 3111-3119 (Shweta)
Efficient Estimation of Word Representations in Vector Space
Mikolov, Tomas and Chen, Kai and Corrado, G.s and Dean, Jeffrey
Proceedings of Workshop at ICLR, 2013 (Qihui)
Compare these papers and the SnowBall paper:
Snowball: extracting relations from large plain-text collections
Eugene Agichtein, Luis Gravano
Proceeding DL’00 Proceedings of the fifth ACM conference on Digital libraries
Pages 85-94

GloVe

GloVe: Global Vectors for Word Representation
Jeffrey Pennington and Richard Socher and Christopher D. Manning
Empirical Methods in Natural Language Processing (EMNLP)
2014, pages = 1532–1543 (Kiwook)

Mar. 11

TensorFlow, Keras discussion, discussion of possible projects

Distributed Representations of Sentences and Documents, Le and Mikolov, ICML’14 Proceedings of the 31st International Conference on International Conference on Machine Learning (Xiaomeng)
Deconfounded Lexicon Induction for Interpretable Social Science Reid Pryzant et al. 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Mingyan)

Mar. 18

RNN, CNN

Assignment 2 is out (on Mar. 15)

Project description