Airbnb Introduces CASTLE for Training Natural Language Search Models with Synthetic Data

Airbnb researchers have introduced CASTLE, a framework designed to address the cold-start problem in natural language search. The approach uses de-identified booking sessions and large language models (LLMs) to jointly generate synthetic search queries and relevance labels, enabling model training before sufficient real search traffic is available.

Written by
Robert Krzaczyński Show moreShow less
Published by
InfoQ
Published
Length
536 words · 2 min
Airbnb Introduces CASTLE for Training Natural Language Search Models with Synthetic Data

Airbnb researchers have introduced CASTLE, a framework designed to address the cold-start problem in natural language search. The approach uses de-identified booking sessions and large language models (LLMs) to jointly generate synthetic search queries and relevance labels, enabling model training before sufficient real search traffic is available.

Natural language search allows guests to describe their preferences in everyday language, such as "romantic cabin near skiing" or "pet-friendly beach house that sleeps eight", rather than relying exclusively on predefined filters. Training such systems presents two challenges: generating queries that reflect actual user behavior and determining which property listings are relevant to each query.

Previous approaches, including InPars and Promptagator, address query generation and relevance labeling as separate tasks. They use LLMs to generate queries from documents or examples, followed by additional model-based judgments to assign relevance labels. According to the researchers, this separation can introduce additional sources of error and differences between synthetic and real-user query distributions.

CASTLE uses contrastive listing pairs derived from de-identified booking sessions. When a guest books a property after viewing several alternatives, the selected listing becomes a positive example, while another viewed but unbooked property serves as a negative example. An LLM is then prompted to generate a query that matches the positive listing but not the negative one. This allows relevance labels to be established through the construction process rather than through a separate evaluation step.

The framework combines three query-generation strategies: template-based prompts, few-shot prompts using approximately 500 seed queries collected through targeted user research, and attribute-grounded prompts focusing on distinguishing listing characteristics. A variety mechanism also encourages variation in query length and phrasing.

For listing pairs without sufficient booking history, including newly listed properties, CASTLE uses a Virtual Judge, an LLM-based evaluator that assigns relevance labels to additional query-listing pairs. The researchers also developed CASTLE-R, a variant intended for production environments where real search traffic has started to accumulate.

In the evaluation, CASTLE achieved a Kullback–Leibler (KL) divergence of 1.01 against real-user query distributions, compared with 9.33 for the best baseline, Promptagator. This represents a 9.2-fold reduction in divergence. At the attribute-type level, CASTLE achieved a KL divergence of 0.08, compared with 0.09 for manually curated survey seed queries.
Source: Medium Blog

The researchers also conducted a human evaluation in which three independent subject matter experts reviewed 200 sampled triplets. Agreement with CASTLE's relevance assignments ranged from 91% to 93% across the three evaluators.

The evaluation methodology also prompted discussion on LinkedIn. Mark Tsai, an AI Engineer specializing in LLM fine-tuning and evaluation, raised a question about potential overlap between training and evaluation data:

Do the training and evaluation splits keep seed templates and listing IDs disjoint? For synthetic triplets, I’d compare a random split with that stricter split, then evaluate on untouched real queries. It would help separate learning reusable intent–attribute matches from recognizing familiar templates or properties.

Wendy Ran Wei, the article's author, responded that the team evaluated the model on untouched real queries and implemented leakage guardrails when constructing evaluation sets. She also clarified that the paper does not report a controlled comparison between random splits and stricter splits with disjoint seed templates and listing IDs, identifying this as a possible direction for further investigation.

The full paper is available on arXiv.

Where this came from

This story was reported by Robert Krzaczyński Show moreShow less and first published by InfoQ on 16 Nov 2026. HUE Legacy Ventures did not write it.

Carried in full with attribution and a link to the original. Rights remain with the publisher, who may request removal at any time.

Read it at infoq.com →