Modèles de langage profonds low-cost : Enquête et évaluation des performances sur la génération de code Python

Découvrez la première version de notre publication scientifique "Modèles de langage profonds low-cost : Enquête et évaluation des performances sur la génération de code Python" publié dans arxiv et soumis au journal Engineering Applications of Artificial Intelligence. Cet article rédigé en anglais est déjà disponible au public. 

Merci à l’équipe de recherche de Novelis - notamment Jessica López Espejel, Mahaman Sanoussi Yahaya Alassan, Merieme Bouhandi, Walid Dahhane, El Hassane Ettifouri - pour son savoir-faire et son expertise.

A propos

"Large Language Models (LLMs) have become the go-to solution for many Natural Language Processing (NLP) tasks due to their ability to tackle various problems and produce high-quality results. Specifically, they are increasingly used to automatically generate code, easing the burden on developers by handling repetitive tasks. However, this improvement in quality has led to high computational and memory demands, making LLMs inaccessible to users with limited resources. In this paper, we focus on Central Processing Unit (CPU)-compatible models and conduct a thorough semi-manual evaluation of their strengths and weaknesses in generating Python code. We enhance their performance by introducing a Chain-of-Thought prompt that guides the model in problem-solving. Additionally, we propose a dataset of 60 programming problems with varying difficulty levels for evaluation purposes. Our assessment also includes testing these models on two state-of-the-art datasets: HumanEval and EvalPlus. We commit to sharing our dataset and experimental results publicly to ensure transparency."

arXiv est une archive ouverte de prépublications électroniques d'articles scientifiques dans différents domaines techniques, tels que la physique, les mathématiques, l'informatique et bien plus encore, gratuitement accessible par Internet.