Situation
The development of large language models (LLMs) has surged in recent years, prompting many organisations to question how they can benefit from these advancements. However, before embarking on the LLM journey, it is crucial to identify the specific use cases where LLMs can add the most business value.
Without this understanding, organisations risk disappointing results, ineffective products, and wasted investments. ProRail, the Dutch railway management company, faced this challenge and sought to understand the value of LLMs for their operations.
Approach
To address this challenge, ProRail initiated five proof of concept projects, known as ‘pressure cooker’ projects, to explore the potential of LLMs in various business scenarios. Valcon was tasked with conducting one of these projects, focusing on improving the Rail Infra Catalogue (RIC), a large database containing design regulations and other documents essential for contractors, architects, and inspectors. The existing search engine for RIC was inefficient, making it difficult for users to quickly find the information they needed.
Valcon’s team of data scientists and business change consultants developed RICO, a chatbot prototype, in just one week. RICO is a Retrieval Augmented Generation (RAG) chatbot designed to answer questions about the design regulations in the RIC. It operates in two steps: first, it searches for relevant passages in the rail catalogue, and then it uses an LLM to generate easy-to-understand responses based on the retrieved information. RICO is specifically instructed to use only information from the catalogue and to avoid answering questions outside its scope.
Results
The implementation of RICO demonstrated significant business value for ProRail. After one week, Valcon showcased RICO to end users, who found it to be a substantial improvement over the existing search engine. RICO could accurately answer most questions and provide references to the original passages for information validation. This capability helped users quickly find and understand the information they needed, improving their efficiency.
Valcon also identified areas for further improvement, such as enhancing the document management system to increase the quality of answers and ensuring users are aware of the need to validate RICO’s responses due to its probabilistic nature. The successful collaboration between ProRail and Valcon highlighted the effectiveness of the pressure cooker approach in rapidly validating innovative LLM solutions. This low-risk method provided ProRail with a swift understanding of the potential impact and business value of AI, paving the way for future advancements.
