Intro to NLP
Lecture: Word embeddings
● Distributional semantics. Count-based (pre-neural) methods. Word2Vec: learn vectors. GloVe: count, then learn. N-gram (collocations) RusVectores. t-SNE.
● Practical: word2vec, fasttext
Lecture: RNN + CNN, Text classification
● Neural Language Models: Recurrent Models, Convolutional Models. Text classification (architectures)
● Practical: Classification with LSTM, CNN
Lecture: Language modelling and NER
●Task description, methods (Markov Model, RNNs), evaluation (perplexity), Sequence Labelling (NER, pos-tagging, chunking etc.) N-gram language models, HMM, MEMM, CRF
● Practical: NER
Lecture: Machine translation, Seq2seq, Attention, Transformers
● Basics: Encoder-Decoder framework, Inference (e.g., beam search), Eval (bleu). Attention: general, score functions, models. Bahdanau and Luong models.
● Transformer: self-attention, masked self-attention, multi-head attention.
Lecture: Transfer learning in NLP
● Bertology (BERT, GPT-s, t5, etc.), Subword Segmentation (BPE), Evaluation of big LMs.
● Practical: transformers models for classification task,
● Practical: Transfer learning
Lecture & Practical: How to train big models?
Part1. Distributed training, Part2. RuGPT3 Training
● Training Multi-Billion Parameter Language Models. Model Parallelism. Data Parallelism.
● Practical: DDP example
Lecture: Syntax parsing
● Practical: Syntax
Lecture: Question answering
● Practical: seminar QA, seminar chatbots
● Squads (one-hop, multi-hop), architectures, retrieval and search, chat-bots
Lecture: Summarization, simplification, paraphrasing
● Practical: summarization seminar
Lecture: Knowledge Distillation in NLP