Task-Specific Architectures for Language Models: A Comprehensive Guide

Some tasks require significant additional architecture on top of a language model, as illustrated in Figure 2 (center). With sufficient training data and computational power, researchers may choose to train both a substantial task-specific architecture and also fine-tune the language model. This is the preferred choice for structure prediction tasks, in particular parsing tasks and occasionally sequence tagging tasks. Examples of sequence tagging models using this approach include BERT-CRF for NER (Souza et al., 2020b; Taher et al., 2019), though notably Devlin et al. (2019) show that the Conditional Random Field (CRF) layer is not necessarily needed for NER with BERT. Examples of parsing models using this approach include UDapter for dependency parsing (Ust"un et al., 2020).

Any sequence-to-sequence task that uses a pre-trained language model as its encoder may employ this approach. An interesting example is Zhu et al. (2020)'s formulation of machine translation. They use a pre-trained BERT model as the encoder and a task-specific architecture as the decoder, resulting in significant improvements in translation performance over traditional machine translation models.

Overall, the choice of approach depends on the complexity of the task, the amount of available training data, and the computational resources available. While some tasks may only require fine-tuning a pre-trained language model, others may require additional task-specific architecture on top of the language model.