Abstract

We systematically study how three large language models with code capabilities - CodeT5, Codex, and ChatGPT - generalize to out-of-domain data. We consider two fundamental applications - code summarization, and code generation. We split data into domains following its natural boundaries - by an organization, by a project, and by a module within the software project. We establish that samples from each new domain present all the models with a significant challenge of distribution shift. We study how established methods adapt models to better generalize to new domains. Our experiments show that while multitask learning alone is a reasonable baseline, combining it with few-shot finetuning on examples retrieved from training data can achieve very strong performance. Moreover, this solution can outperform direct finetuning for very low-data scenarios. Finally, we consider variations of this approach to create a more broadly applicable method to adapt to multiple domains at once. We find that for code generation, a model adapted to multiple domains simultaneously performs on par with those adapted to a single domain

Metadata

publication

arXiv preprint arXiv:2303.09128, 2023

year

2023

publication date

2023/3/16

authors

Shushan Arakelyan, Rocktim Jyoti Das, Yi Mao, Xiang Ren

link

https://arxiv.org/abs/2303.09128

resource_link

https://arxiv.org/pdf/2303.09128

journal

arXiv preprint arXiv:2303.09128