Missions

The main task associated to this position is the evaluation of large multilingual generative models (mLLMs), i.e. generative models that support more than one language. The main question that arises is how to measure a model's level of “multilingualism” - to put it simply, “how many (and which) languages can the model handle (and with what level of quality)”? This methodological analysis is part of the European LLM4EU project, which aims to develop evaluation models and frameworks for all the official languages of the EU.

Background

In just a few years, lnatural anguage processing tools based on large language models have reached very high levels of performance for complex tasks. Today, they are widely used in our digital work environments to access, analyze and reformulate information, or to generate original content. With the widespread use of these technologies, the analysis of the actual performance, risks and limitations of these models is becoming increasingly important. When these models are multilingual, one dimension of the evaluation must concern the level of multilingualism of a model.

This evaluation is difficult, as it cannot (usually) be directly derived from the analysis of the training data and protocols, and must therefore be carried out in a “black box setting”, using queries submitted to a trained model. Multiple monolingual tasks and benchmarks exist, especially for English, but the quality and variety of these benchmarks is very unevenly distributed between languages. A second difficulty relates to the need to compare performance between languages: in this comparison, multiple factors may come into play, some of which are linked to the model, but others are intrinsic to the languages under consideration.

In the first stage, the work will aim to build robust metrics applicable to models for which the internal representations and probability distributions that are being manipulated can be accessed: for these models, we will study the validity of several information-theoretic-inspired metrics (compression, perplexity), and deploy them on an open infrastructure. In a second phase, the work will aim to study methods for minimizing differences between languages and making metrics more directly comparable -- for example, by using “universal” transcodings that are fairer between languages (by transliteration, phonetization, etc.). At the same time, we will look at possible generalization of the metrics proposed for open models to closed models.

Activités

The successful applicant will collaborate with members of the laboratory working on language processing; more broadly, he/she will maintain collaborations within the “LLM4EU” project (with other CNRS laboratories, and more generally with other project partners), to evaluate large multilingual language models, in particular to measure their linguistic abilities; and then to report on this work in scientific articles and communications. A significant part of the activity will be empirical in nature, involving experiments on state-of-the-art (large) language models.

Compétences

We are looking for a highly motivated candidate:
- holding a recent master's degree or engineering school diploma in artificial intelligence with skills in deep learning, natural language processing, information retrieval or machine translation.
- mastering Python and deep learning platforms and frameworks for manipulating language models and text generation algorithms;
- fluent in French and scientific English (written and spoken), a genuine interest in languages is a plus.

Contexte de travail

This position is affiliated with the Institut des Systèmes Intelligents et de Robotique, a multi-disciplinary laboratory jointly operated by Sorbonne Université and of CNRS, within the “Machine Learning and Deep Learning for Information Access” team (MLIA - https://www.isir.upmc.fr/equipes/mlia/). MLIA focuses on machine learning and its applications, particularly in language processing. Located on the Jussieu campus in central Paris, ISIR has over 250 members and is a major player in AI and Robotics in Europe (https://www.isir.upmc.fr).

Contraintes et risques

Office work with display screen equipment, no other specific risk.