Domain: AI in Market Research
113 Industries is a Market Research Firm helping Fortune 500 companies like OceanSpray and Hersheys innovate. They use AI and a team of expert analysts to reveal critical insights into consumer behavior.
The insights are discovered leveraging large amounts of consumer generated content including user reviews, survey comments and social media posts.
Problem in the workflow
While 113 Industries uses a third-party analysis tool for much of their analysis, they have found it inadequate for an in-depth analysis, requiring their analysts to perform tireless amounts of manual work. One such problem is the ability to research on related domain-specific concepts or concepts that occur in conjunction with other concepts. You can think of these concepts as descriptive keywords.
For example, if Safeway (a grocery store) was a topic of analysis, they also wanted to analyze other grocery stores like HEB and Smiths as well as concepts related to grocery stores such as farmers market and grocers . Another example is to be able to surface products that are used in conjunction with other products. For example, peanut butter is often used in conjunction with jelly.
While it’s easy for a human to enumerate related concepts for a handful of topics, scaling this up to thousands of different topics as well as industries requires Natural Language Processing automation. Otherwise, this process would be highly time and labor-intensive — which is something they wanted to reduce.
Leveraging AI in market research
To help 113 Industries achieve their goal of automatically uncovering synonymous and related concepts for market research, we developed a custom Natural Language Processing pipeline that feeds into their existing analysis workflow. The underlying technology uses Python, Apache Spark, Gensim, and custom algorithms designed by our team.
The solution we developed was general in that it works for different industries and does not require labeled training data. It can also uncover surprising concepts that a human may not typically think about, providing analysts with a variety of concepts to work with.
To ensure that the results made sense from a utility perspective, we internally performed evaluation across several industries. 113 also performed their own evaluation to vet the quality of the related concepts. Finally, to ensure correct use and integration of the pipeline, we provided hands-on training to the 113 team on:
- How to retrain the system on new data
- How to evaluate data quality prior to retraining models
- How to deal with data sparsity
- Limitations of the system
The Results
Because of our partnership, 113 Industries is now able to perform research on any number of concepts from different industries instantaneously. In their old method of listing related concepts manually, it used to take them hours to collect concepts for each research question. It now takes them minutes.
Also, because the tool that we developed is not dependent on labeled data, their cost is contained to the investment they had put into developing the AI system and integration thereafter.
