Understanding Knowledge Distillation in Neural Sequence Generation
5 536
23.7
Microsoft Research323 тыс
Следующее
17.01.20 – 9831:30:09Популярные
43 дня – 9 11826:38
96 дней – 7115:33Опубликовано 17 января 2020, 18:47
Sequence-level knowledge distillation (KD) -- learning a student model with targets decoded from a pre-trained teacher model -- has been widely used in sequence generation applications (e.g. model compression, non-autoregressive translation (NAT), low-resource translation, etc). However, the underlying reasons behind this success have, as of yet, been unclear. In this talk, we will try to tackle the understanding of KD particularly in two scenarios: (1) Learning a weak student from a strong teacher model while keeping the same parallel data used for training the teacher; (2) Learning a student from a teacher model of equal size while the targets are generated from additional monolingual data.
Talk slides: microsoft.com/en-us/research/u...
See more on this and other talks at Microsoft Research: microsoft.com/en-us/research/v...
Talk slides: microsoft.com/en-us/research/u...
See more on this and other talks at Microsoft Research: microsoft.com/en-us/research/v...
Свежие видео
4 дня – 3011:23
5 дней – 76711:12
7 дней – 1 329 74110:51
9 дней – 5841:07
9 дней – 4 43957:02
10 дней – 3 1630:11Случайные видео
10 дней – 8 9500:20
269 дней – 45 9693:43
30.03.23 – 7 8960:48
18.03.21 – 88 2575:39
16.05.20 – 7 3845:32
15.04.17 – 18 8651:54:42
1 день – 5670:07
2 дня – 15 1208:43
3 дня – 35 0090:22
3 дня – 16 67610:18
6 дней – 978 4938:57
8 дней – 3 181 1620:31
11 дней – 5 4750:20
