Discover our article on JaCoText: A Pretrained Model for Java Code-Text Generation published in the journal "World Academy of Science, Engineering and Technology".
Our research team also presented its work at the International Conference on Code Generation and Implementation: Watch the replay
Thanks to the Novelis Research Team for their knowlegde and experience.
Authors: Jessica Lòpez Espejel, Mahaman Sanoussi Yahaya Alassan, Walid Dahhane, El Hassane Ettifouri
Abstract
Pretrained transformer-based models have shown high performance in natural language generation task. However, a new wave of interest has surged: automatic programming language generation. This task consists of translating natural language instructions to a programming code. Despite the fact that well-known pretrained models on language generation have achieved good performance in learning programming languages, effort is still needed in automatic code generation. In this paper, we introduce JaCoText, a model based on Transformers neural network. It aims to generate java source code from natural language text. JaCoText leverages advantages of both natural language and code generation models. More specifically, we study some findings from the state of the art and use them to (1) initialize our model from powerful pretrained models, (2) explore additional pretraining on our java dataset, (3) carry out experiments combining the unimodal and bimodal data in the training, and (4) scale the input and output length during the fine-tuning of the model. Conducted experiments on CONCODE dataset show that JaCoText achieves new state-of-the-art results.
About the article
"In this paper, we present JaCoText, a pretrained model based on Transformers [5]. First, we initialize our model from pretrained weights of CoTexT-1CC and CoTexT-2CC, instead of performing a training from scratch. Later, we conduct an additional pretraining step using data that belongs to a specific programming language (Java in our case). Moreover, unlike works that based their pretraining on CodeSearchNet [18] such as CodeBERT [19] and CoTexT [2], we use more java data in the pretraining stage of our model, as [13] and [14] have shown that Transformers neural network improves its performance significantly from increasing the amount of pretraining data. Furthermore, we carry out experiments to measure the impact of the input and output sequences length on code generation task. Finally, we test the unimodal data and study its impact on the model’s performance. This study is crucial to evaluate the model in the pretraining stage."
World Academy of Science, Engineering and Technology is a predatory publisher of open access academic journals.
