research paper

Do Language Models Know That They're Being Evaluated?

Do Language Models Know That They're Being Evaluated?

Picture this scenario: You’re very new to AI, exploring chatgpt by testing its capabilities on various topics, expecting honest answers unaware that behind the scenes, it already figured out that it’s being tested and is subtly changing its behaviour to ace your tests. This feels like a subtle

From Turn 1 to Turn 10: How LLMs Get Lost In Multi-Turn Conversations

From Turn 1 to Turn 10: How LLMs Get Lost In Multi-Turn Conversations

Real-world interactions between humans and LLMs are rarely single‑shot. Rather, users start with vague requests, iterate, clarify, and refine over multiple turns. Yet, most LLM benchmarks assume a fully‑specified, single‑turn setting which is different from how people actually chat. Prior analyses of conversation logs confirm that underspecification

SuperBPE: Rethinking Tokenization for Language Models

SuperBPE: Rethinking Tokenization for Language Models

In the domain of language models, tokenization i.e. the process of breaking down text into manageable units plays a pivotal role. Traditionally, models rely on subword tokenization, where words are split into smaller units. However, this approach often overlooks the semantic significance of multi-word expressions and varies across languages.

Image generated using Meta AI

Ensuring responsible AI: An overview of DeepMind’s FACTS framework

Introduction DeepMind introduces the FACTS Grounding, an online leaderboard and benchmark designed to evaluate the factual accuracy of language models' long-form responses based on a given context in user prompts. The benchmark requires models to generate text grounded in a provided document, which can be up to 32,000

Image credit: Meta

Innovative training of LLMs in continuous latent spaces, by Meta AI

LLMs have made significant progress in tasks based on language processing and understanding. However, their reasoning capabilities, particularly in complex scenarios, often fall short. Traditional approaches like Chain-of-Thought (CoT) reasoning, which asks the model to reason before answering by thinking step-by-step, face inherent limitations. A recent paper introduces a novel