IFT6010, Winter 2021
Modern Natural Language Processing
Time and Place
Tuesdays 9:30am - 11:30am
Online via Zoom
Fridays 3:30pm - 5:30pm
Online via Zoom
Please check the
Studium platform for the link to Zoom and Slack workspace of this course.
Note that
the lectures will be recorded. The link to each recorded lecture will also be posted on
Studium after class.
Instructor
Bang Liu
Office hours: Tuesdays 3:00pm - 5:00pm. (Online via Zoom/Slack workspace. Same meeting room with our lectures)
Email: firstname.lastname@umontreal.ca
Course description
Natural Language Processing (NLP) is a branch of artificial intelligence that deals with the interaction between computers and humans using natural language. It is one of the most important technologies of the information age and is used everywhere: search engines, advertising, chatbots, language translation, virtual agents, and so on. Deep Learning approaches have obtained very high performance across many different NLP tasks in recent years. In this course, students will gain a thorough introduction to the basics of NLP, as well as cutting-edge research in Deep Learning for NLP. We will focus on modern techniques for NLP, as well as introduce the applications in our daily lives. Students are encouraged to do some pretty cool research projects based on NLP techniques, e.g., writing poetry, detecting spam emails, building chatbots, machine reading comprehension, and so on. Through lectures, assignments, and a term project, students will learn the necessary skills to design, implement, and understand their own models for NLP tasks.
Prerequisites
- Proficiency in Python
All class assignments will be in Python (e.g., using NumPy and PyTorch). If you have a lot of programming experience but in a different language (e.g. C/C++/Matlab/Java/Javascript), you will probably be fine.
- College Calculus, Linear Algebra
You should be comfortable taking (multivariable) derivatives and understanding matrix/vector notation and operations.
- Basic Probability and Statistics
You should know the basics of probabilities, gaussian distributions, mean, standard deviation, etc.
- Foundations of Machine Learning
If you already have basic machine learning and/or deep learning knowledge, the course will be easier.
Reference Texts
No textbook is required. But the following texts that can be read free online are helpful.
The following book is helpful to give you more background about neural networks.
Marking scheme
- Readings (10%):
10 readings assignments worth 1% each
- Assignments (45%):
3 programming assignments worth 15% each
- Term project (45%):
Project proposal (up to 2 pages): 5%
Midway report (up to 4 pages): 5%
Final presentation: 5%
Final report (up to 8 pages): 30%
- Bonus: class participation (3%):
Students are encouraged to actively participate in the class, slack, etc.
Late policy: each student has 6 free late days to use (including all the three assignments, project proposal, project mid-way report, and project final report. No late day for the final project presentation and reading assignments). You can use up to 2 late days per assignment. A late day extends the deadline 24 hours. Submissions after 2 late days or the deadline (if you have used all your late days) won't be accepted. We don't count hours, e.g., if you submit an assignment after 25 hours, it will be considered as 2 late days.
Tentative Time Table
Note: tentative schedule is subject to change.
Videos of the course can be found here:
ITF 6010 Winter 21 recording
Date | Topic |
Section I: Introduction / background |
Lecture 1 (Jan 19)
|
Introduction to NLP
|
Lecture 2 (Jan 22)
|
Basics of Deep Learning: Backpropagation and Neural Networks
|
Section II: NLP core techniques |
Lecture 3
|
Language Modeling and Recurrent Neural Networks
|
Lecture 4
|
Word Meaning and Word Embedding
|
Lecture 5
|
Sentence Embeddings, Convolutional Neural Networks
|
Lecture 6 & 7
|
Graph Representations for NLP, Graph Convolutional Network
|
Lecture 8
|
Machine Translation, Seq2Seq and Attention
|
Lecture 9
|
Transformer and BERT
|
Lecture 10
|
Pre-trained Language Models (student mini lectures)
|
Lecture 11
|
Constituency Parsing
|
Lecture 12
|
Syntactic Dependency Parsing
|
Section III: Cutting-edge research topics. |
Lecture 13
|
Data, Knowledge, and Logic: Modeling and Reasoning for Natural Language Understanding
|
Lecture 14
|
Unsupervised Text Generation (Guest lecture: Lili Mou, University of Alberta)
|
Lecture 15
|
Knowledge Graph (Guest lecture: Yanghua Xiao, Fudan University)
|
Lecture 16 & 17
|
Open-domain Question Answering (Conference tutorial: ACL 2020)
|
Lecture 18 & 19
|
Neural Approaches to Conversational AI (Conference tutorial: ICML 2019)
|
Lecture 20 & 21
|
Commonsense Knowledge Representation and Reasoning in Natural Language Processing (Conference tutorial: ACL 2020) |
Lecture 22 & 23
|
Course project presentations and discussions.
|
Resources
Software
- PyTorch an open-source deep learning library.
- DGL an open-source library for deep learning on graphs.
- HuggingFace Transformers an open-source library containing PyTorch and Tensorflow implementations, pre-trained model weights, usage scripts and conversion utilities for a variety of pre-trained language models.