Description of the thesis topic

The goal is to improve the traceability, readability, and explanatory fidelity of responses generated by LLMs while maintaining their performance. This research is part of a perspective of interpretability and query engineering, and will lead to analysis tools, generation protocols, and more explainable enriched models.

The thesis topic raises several research challenges. A critical point is the implicit translation of prompts into English, even when they are initially written in French. This operation, often invisible to the user, can introduce semantic, linguistic and cultural biases (idiomatisms, reformulations, loss of meaning), imprecisions or errors in the results. On the other hand, the intermediate steps of the chain of thought are rarely made explicit or controlled, which harms the transparency and traceability of the inference. In parallel, the introduction of special tokens in queries (e.g. reasoning, step, conclusion, lang=fr, etc.) or in the model's responses can be used to: structure the reasoning produced (beginning/end, intermediate steps, justification, answer choice), guide the model's behavior (forcing explanation or format), improve explainability by facilitating the extraction and visualization of key parts of the query processing.
The main objectives of the thesis are:
1/ To study the strengths, limitations and biases of CoT, as a multi-query approach exploring the knowledge space of an LLM.
2/ To explain the different steps involved in the processing of a complex prompt: Implicit translation (and its biases), Number and nature of decomposition steps, Use of paraphrase or intermediate reformulations.
3/ To develop mechanisms to guide, annotate or control these steps, by introducing special tokens, linguistic annotations, or structured prompts.
4/ To evaluate the impact of these elements on: the quality of the final result, the computational cost, the perceived and measurable explainability of the reasoning.
5/ Propose more robust, multilingual, and more transparent CoT variants.

The proposed methodology:
Phase 1: Literature review on CoT, structured prompts, implicit translation biases, and explainability methods.
Phase 2: Design of guided CoT variants: hierarchical prompts, explicit decompositions, controlled paraphrases.
Phase 3: Integration into existing and open architectures (LLaMA, DeepSeek, ...) and instrumentation of prompt processing to trace internal steps.
Phase 4: Experimental evaluation on complex QA and reasoning tasks (GSM8K, HotpotQA, CosmosQA), with comparison to standard baselines and qualitative analysis of the introduced linguistic biases.
Phase 5: Proposal of an interpretability protocol based on the reconstruction of the reasoning path followed by the model.

Bibliography:
1. Chain-of-Thought Prompting:
[1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., ... & Le, Q. (2022). Chain of Thought Prompting Elicits Reasoning in Large Language Models. arXiv preprint arXiv:2201.11903. https://arxiv.org/abs/2201.11903
[2] Nye, M., Lin, K., Lee, J., Chen, X., & Schulman, J. (2021). Show Your Work: Scratchpads for Intermediate Computation with Language Models. NeurIPS.https://arxiv.org/abs/2112.00114
Zhou, D., Schärli, N., Hou, L., Wei, J., & Le, Q. V. (2022). Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. arXiv preprint arXiv:2205.10625. https://arxiv.org/abs/2205.10625
[3] Yao, S., Zhao, J., Yu, D., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv preprint arXiv:2305.10601. https://arxiv.org/abs/2305.10601
2. Explicability:
[4] Andreassen, A., Reif, E., & Hewitt, J. (2024). Inseq: A Python Library for Interpretability Analyses of Sequence Generation Models. arXiv preprint arXiv:2407.15248. https://arxiv.org/abs/2407.15248
[5] Kim, S., Hwang, Y., Yoon, J., & Lee, K. (2023). Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention. arXiv preprint arXiv:2312.15033. https://arxiv.org/abs/2312.15033
[6] Zhu, Y., Ma, L., Lu, H., Zhang, H., & Zhang, M. (2024). SEER: Self-Explainability Enhancement of Large Language Models' Representations. arXiv preprint arXiv:2502.05242. https://arxiv.org/abs/2502.05242
[7] Wang, W., Li, J., & Gao, Y. (2023). Proto-LM: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models. arXiv preprint arXiv:2311.01732. https://arxiv.org/abs/2311.01732
3. Bias in LLM
[8] Ruder, S., et al. (2021). Beyond English-Centric Multilingual Machine Translation. Findings of EMNLP 2021. https://arxiv.org/abs/2103.06508
[9] Costa-jussà, M. R., et al. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv preprint arXiv:2207.04672. https://arxiv.org/abs/2207.04672
[10] Shen, Y., et al. (2023). How Far Can We Go with Multilingual Prompting? ACL 2023. https://arxiv.org/abs/2302.03983

Work Context

Large language models (LLMs) such as chatGPT, Gemini, Claude, LLaMA, LeCHAT, and DeepSeek have achieved remarkable performance on complex human-AI query processing tasks. Recent techniques that have enhanced their capabilities include Chain-of-Thought prompting (CoT), which breaks down a question into successive steps. This technique, popularized by LangChain, is similar to a multiple query method aimed at exploring more deeply the internal knowledge of an LLM. It thus promotes better structuring of human-AI interaction and improves the explainability of the generated answers. However, several challenges remain. Another critical point is the implicit translation of prompts into English, even when they are initially written in French. This operation, often invisible to the user, can introduce semantic, linguistic and cultural biases (idiomatisms, reformulations, loss of meaning), imprecisions or errors in the results. On the other hand, the intermediate steps of the chain of thought are rarely made explicit or controlled, which harms the transparency and traceability of the inference. In parallel, the introduction of special tokens in queries (e.g. reasoning, step, conclusion, lang=fr, etc.) or in the model's responses can serve to: structure the produced reasoning (start/end, intermediate steps, justification, answer choice), guide the model's behavior (forcing explanation or format), improve explainability by facilitating the extraction and visualization of key parts of the query processing. The goal is to improve the traceability, readability, and explanatory fidelity of responses generated by LLMs while maintaining their performance. This research is part of a perspective of interpretability and query engineering, and will lead to analysis tools, generation protocols, and more explainable enriched models.

The position is located in a sector under the protection of scientific and technical potential (PPST), and therefore requires, in accordance with the regulations, that your arrival is authorized by the competent authority of the MESR.

Constraints and risks

The subject has constraints in terms of computing capacity. A platform will be available to the student to conduct their experiments, which they will have to use wisely. We will take care to work on the frugality of the algorithms and measure the costs.

Additional Information

The thesis is a joint supervision financed by a grant from La Sorbonne between STIH-Sorbonne (C. Montacie) and LISN-CNRS (L. Devillers).