Discover how we helped a large US e-commerce company establish a robust evaluation framework, identify key metrics, and automate model deployment to drive measurable sales growth and improve search performance.
Organization Supported: A Leading US-Based E-Commerce Company
Number of Employees: ~3000
Service: NLP & Machine Learning Consulting
Objective: Optimizing E-Commerce Search Algorithms with Machine Learning Model Evaluation
Challenge: Ineffective Model Evaluation for Search Algorithm Improvement
A large US-based e-commerce company, which generates a significant portion of its sales through its search engine, faced a challenge in optimizing its product search algorithms. The company’s software team developed a series of machine learning (ML) models aimed at improving the search quality for specific product categories. However, after deploying these models, they were unsure about how to objectively evaluate model and business performance.
The main issue was a lack of a clear, quantitative framework to assess the effectiveness of the machine learning models. Decisions were being made based on intuition and gut feel, rather than data-driven insights. The company also struggled to define the right metrics to track and measure the lift in sales after deploying the models. This uncertainty left them in a challenging position, where they couldn’t confidently determine whether the models delivered value to their e-commerce platform.
Solution: Building a Robust Model Evaluation and Metrics Framework
Given the complexity of the issue, the first step in our approach was to address the foundational aspects of their process. In collaboration with the company’s software team, we conducted a thorough analysis to understand the following:
- The existing metrics that the company was tracking related to search performance and sales.
- The process around machine learning model development, evaluation, and deployment.
- How the ML models fit into the overall search engine workflow.
- How the company tracked the impact of search algorithm changes on sales.
After gaining a deep understanding of the company’s existing workflow and challenges, we formulated a plan to address the core issues and provide a clear path forward.
Step 1: Identifying Key Metrics
We worked with the team to shortlist the most relevant metrics to track, particularly those tied to search performance and sales outcomes. These metrics were essential for measuring the effectiveness of the ML models and determining the lift in sales once the models were deployed.
We guided the company in tracking these key performance indicators (KPIs) and provided a methodology for measuring the impact of the models both with and without their operation. This allowed the company to establish a clear benchmark for comparison.
Step 2: Developing a Model Evaluation Workflow
With the right metrics in place, we turned our attention to improving the process for evaluating and deploying machine learning models. We worked closely with the company to develop a robust, repeatable workflow for ensuring that only models that meet the required criteria are deployed.
Key aspects of the workflow included:
- Production Activity Signals: We introduced a method to use real-time production activity data as a source for model evaluation. This ensured that the company’s evaluation process was grounded in actual performance data from the live environment.
- Evaluation Metrics: We provided a suite of evaluation metrics that aligned with the company’s business goals, enabling them to assess the impact of the ML models on search results and sales.
- Automated Decision-Making Process: We developed an automated workflow to help the company determine whether a model is fit for deployment. This process leveraged the production activity data and evaluation metrics to provide an objective, data-driven decision.
Step 3: Tracking Model Degradation
Finally, we set up a framework to track model degradation over time. As machine learning models can lose their effectiveness due to changing user behavior, seasonality, or other factors, it was critical for the company to be able to monitor model performance on an ongoing basis. With the metrics, evaluation data, and automated workflow in place, the company can now proactively detect and address model degradation, ensuring that search performance remains optimal.
Results: Improved Model Evaluation and Sales Impact
As a result of our collaboration, the e-commerce company gained:
- Clarity on Key Metrics: The company now has a clear understanding of which metrics to track in order to evaluate both the performance of its search engine and the impact of machine learning models on sales. This data-driven approach replaced gut-feel decision-making with objective, measurable insights.
- Automated Model Evaluation: With the new model evaluation workflow, the company can automatically assess whether a model is fit for deployment. This workflow, which leverages production activity data, has significantly streamlined their process and increased confidence in model-related decisions.
- Ability to Track Model Degradation: The new evaluation framework not only helps assess the initial effectiveness of the models but also enables ongoing monitoring to detect potential degradation over time. This proactive approach ensures that the company can continue to optimize its search algorithms and adapt to changes in user behavior.
Overall, the company is now equipped with a reliable, data-driven approach to evaluate and deploy machine learning models, ensuring continuous improvement in search performance and driving measurable sales growth.
Conclusion
By helping the company establish a comprehensive evaluation framework for their machine learning models and aligning metrics with business goals, we empowered them to make more informed decisions about their search algorithms. The new workflow and metrics tracking system not only optimized their models but also provided the foundation for sustained improvements in e-commerce search performance.
Book a Free 20-Minute Strategy Call With Opinosis Analytics
Good Methodology and Foundation for our Large Company
The methodology Opinosis Analytics came with was a great involvement of all the team. We worked with them. We defined with them. So they were very engaged. And I think this is, in the long term, very good for the company. We now have a very detailed document of all our processes. So it’s a very good and valuable start. If we want to start developing the software recommended, we know where to start. We have a very good foundation to start conversations with potential software developers and so on, and that was not existing before…We look forward to our ongoing collaborations with Opinosis Analytics.
Opinosis Analytics Helped Us Mitigate Risks…
Working with Opinosis Analytics has been a highly positive experience. Their collaborative approach, combined with a strategic mindset, ensured that we were aligned every step of the way. Opinosis took the time to understand our unique business challenges, delivering tailored AI solutions that met our needs and helped us mitigate risks. Their open communication, reliability, and professionalism were exceptional throughout the project. We’re extremely satisfied with the outcome and would absolutely recommend Opinosis Analytics to others.
Impressive Work Quality…
Dr. Ganesan’s knowledge of Natural Language Processing (NLP) and text mining methodologies and their application to solve real-world problems is very impressive. She is able to understand the business goals and then apply and recommend the algorithms that would be most effective. She is also able to plan and estimate her work very well, which is very helpful and reassuring, especially when one is exploring an unfamiliar territory. Her work is of very high quality. Her communication skills are excellent and she is able to explain and support her work and her recommendations very well.
We’re Internationally Recognized AI for Business Transformation Experts
