When NLP meets LLM

This book looks at conversational search in intelligent dialogue systems, as it investigates and addresses the challenges pertinent to effective context incorporation in conversational question answering (ConvQA). The authors explore the possibility of designing a scalable Conversational Question Answering Agent that can handle the challenges of incomplete/ambiguous questions, better able to relate to co-references to cope with the problems of effective weights and optimal threshold selection in vehicular networks. A fundamental emphasis is the understanding of ambiguous follow-up questions and the generation of contextual and question entities to fill in the missing information gaps. Key topics are studied, such as ‘hard history selection’ to filter out the context that is not relevant and performing a re-ranking of the selected turns based on their significance to answer the question as a part of the soft history selection process.

This book aims to demonstrate that the history selection and modelling approaches proposed can effectively improve the performance of ConvQA models in different settings. The proposed models are compared with the state-of-the-art vis-à-vis different conversational datasets and provide new insights into conversational information retrieval. Through a systematic study of structured representations, entity-aware history selection, and open-domain passage retrieval using contrastive learning, this book presents a robust framework for advancing multi-turn QA systems.

It is an essential resource for researchers, practitioners, and graduate students working at the intersection of NLP, dialogue systems, and intelligent information access.