Situation

In the last couple of years, one of Valcon’s retail clients saw a tremendous change in the way consumers buy their products and how their products need to be distributed. These changes are partly related to the changing demand of the new generation of consumers and the increased wish to buy products online. It became clear that the business model that made the client successful in the past had to be transformed into a more digital model. Therefore, the client started a digital transformation in which processes were made more data-driven and automated.

One of their data-driven efforts revolved around understanding the competitive landscape. This has always been key to a successful business, especially in today’s retail sector which has become more competitive, multi-channelled and global.

To gain these competitive insights, Valcon’s client already had an existing third-party solution in place. However, this solution did not cover all of the client’s markets and was complex to maintain due to the many rules that are in the solution (matches between a product of the client and a product of its competitors were made on the basis of a set of business rules). Scaling the solution to accommodate all product categories and all markets our client operated in would be extremely expensive.

To solve these challenges, a new solution using Artificial Intelligence (AI) was to be designed that would offer the required quality, scalability and flexibility. Because of Valcon’s experience with implementing end-to-end AI solutions in the retail sector, Valcon was asked to make a proposal for the delivery of such a solution.

Solution

Valcon developed a solution that monitors competitor products and service information from retailers around the world. The solution consists of the following steps (see also the figure below):

1. Data acquisition: using web scraping for publicly available product data from competitor websites and extracting data from internal applications containing product information.
2. Feature engineering: extracting relevant information from the acquired and combined data which consists of images, text and structured data.
3. Product matching: matching competitor products with internal products using leading-edge AI techniques (e.g. transformer models and deep neural networks).
4. Match validation: using human input to incorporate a feedback loop into the model training process.
5. Insights dashboard: visualising the insights based on product matches to enable data-driven decision-making.

Together, these components enable timely action on competitor moves.

 

Approach

The acquisition of data by means of web scraping is an important part of the solution. After developing the proof of concept, a third-party web scraping specialist was contacted to improve the scalability and maintainability of the web scraping component. This allowed Valcon to focus on AI development.

After acquiring the right data, Valcon focused on improving the product-matching models. The solution was based on leveraging state-of-the-art AI techniques (e.g. transformer models) that assess the similarity of products to find product matches. To improve this matching model, the right training data is required. This training data was improved by providing the model with examples of correct and incorrect product matches via a customised front-end app. This front-end app allowed people to quickly validate product pairs suggested by the model. Based on this human feedback, the training data set was improved in an iterative process.

The models used in this solution are computationally intensive. That is why, after developing the matching models, Valcon focused on improving the scalability of the solution. Several improvements were implemented such as simplifying the architecture of the models and using the latest database techniques (e.g. Databricks Delta Lake).

During the development of the solution, Valcon learned which data input was important, how to efficiently process these and how to train the AI model for their use case. This helped Valcon to prepare for the next phase, which is scaling up to other markets and categories. Currently, Valcon and its client are in this scaling phase.

 

Result

With the developed solution, the client is now able to gain insights into the competitor landscape and determine their price position in the market to adjust their strategy accordingly. Due to its success, the solution is planned to be rolled out globally in the upcoming years.

It has been a challenging journey for the Valcon team to build such an extensive end-to-end solution, using the latest cloud technology and AI techniques.

If you are interested to know more about our Competitive Intelligence service, read more about it here.