Automating Concepts Extraction for Market Research

113 Industries is a Market Research Firm helping Fortune 500 companies like OceanSpray and Hersheys innovate. They use AI and a team of expert analysts to reveal critical insights into consumer behavior.

The insights are discovered leveraging large amounts of consumer generated content including user reviews, survey comments and social media posts.

Problem in the workflow

While 113 Industries uses NetBase for much of their analysis, they have found it inadequate for in-depth analysis, requiring their analysts to perform tireless amounts of manual work. One such problem is the ability to research on related domain specific concepts or concepts that occur in conjunction with other concepts.

Ibotta Adds 5 New Retailers - HEB, Meijer, and MORE

For example, if Safeway  (a grocery store) was a topic of analysis, they also wanted to analyze other grocery stores like HEB  and Smiths as well as concepts related to grocery stores such as farmers market and grocers . Another example is to be able to surface products that are used in conjunction with other products. For example, peanut butter is often used in conjunction with jelly.

While it’s easy for a human to enumerate related concepts for a handful of topics, scaling this up to thousands of different topics as well as industries requires Natural Language Processing automation. Otherwise, this process would be highly time and labor intensive — which is something they wanted to reduce.

Our solution

To help 113 Industries achieve their goal of automatically uncovering synonymous and related concepts, we developed a custom Natural Language Processing pipeline that feeds into their existing analysis workflow. The underlying technology uses Python, Apache Spark, Gensim and custom algorithms designed by our team.

The solution we developed was general in that it works for different industries and does not require labeled training data. It can also uncover surprising concepts that a human may not typically think about.

Figure 1 shows an example of concepts related to grocery store , walmart and safeway surfaced by our AI system. Concepts like convenience store is not something 113’s analysts typically come up with, but is captured by our system.

Figure 1: Example of concepts related to grocery stores uncovered by our AI system
Figure 1: Example of concepts related to grocery store uncovered by our AI system

To ensure that the results made sense from a utility perspective, we internally performed evaluation across several industries. 113 also performed their own evaluation to vet the quality of the related concepts. Finally, to ensure correct use and integration of the pipeline, we provided hands-on training to the 113 team on:

  1. How to retrain the system on new data
  2. How to evaluate data quality prior to retraining models
  3. How to deal with data sparsity
  4. Limitations of the system

The Results

Because of our partnership, 113 Industries is now able to perform research on any number of concepts from different industries instantaneously. In their old method of listing related concepts manually, it used to take them hours to collect concepts for each research question. It now takes them minutes.

Also, because the tool that we developed is not dependent on labeled data, their cost is contained to the investment they had put into developing the AI system and integration thereafter.

See Also

 

 

Scroll to Top