Machine Learning with Unstructured Content: Text Analytics
What is Text Analytics?
Often when we think of machine learning, we think of models that predict an outcome... we don't often think of how these models can be used to predict the sentiment and themes of text data. Text Analysis leverages machine learning and natural language processing capabilities to help businesses extract insight from text data. By converting raw text or unstructured data to structured content using machine learning algorithms, allows business users to understand the key constructs in their underlying text. Below are some of the key functions that text analytics performs:
Text Preprocessing: Cleaning and formatting the text data to prepare it for analysis. This may include removing stop words, punctuation, and stemming words to their root forms.
Sentiment Analysis: Determining the emotional tone and polarity of the text (e.g., positive, negative, neutral) to gauge customer sentiment. Good Text Analytics tools will also provide a numeric sentiment score so that users can ascertain the extent of the polarity (i.e., very positive or very negative).
Entity Recognition: Identifying and categorising entities such as names, locations, and products mentioned in the text. Text Analytics can extract entities such as brands, countries, products, etc.
Topic Modeling: Discovering prevalent themes and topics within the text data. This is a critical component of Text Analytics, as it allows businesses to identify key themes, as well as define rules for the classification of words and/or phrases (sometimes referred to as "Queries").
Note: Text Analytics is not a word cloud. Word clouds are a very simple representation of commonly mentioned words in a given text data set, but does not provide the richer capabilities mentioned above. There are better ways to analyse and visualise outputs from Text Analytics; please get in touch with our team if you want to learn more about these.
Why Analyse Text Data?
Text analysis plays a pivotal role in three key areas for businesses: Customer Insights and Feedback Analysis, Competitive Intelligence, and Operational Efficiency and Risk Management. Firstly, it enables organisations to gain valuable insights from customer reviews, feedback, social media mentions, and customer experience interactions. These can help with facilitating informed decisions to enhance product development, marketing strategies, and customer experience.
Secondly, text analysis aids in monitoring competitors by analysing public sources, allowing businesses to identify market gaps and refine their strategies. Understanding what your target customer perceives about your competitors and/or their products is incredibly valuable, especially if this data can be obtained from mechanisms that carry less bias (such as review sites for example).
Lastly, it can improve internal operations by analysing employee feedback and communication, while also serving as a tool for proactive external risk management by monitoring news articles and social media for emerging threats.
These applications collectively empower businesses to make data-driven decisions and improve overall performance, using unstructured text data that would have otherwise been cumbersome or impossible to analyse.
Text Analysis for Customer Insights and Feedback Analysis
Understanding customer sentiment and preferences is paramount for any business. Text analysis allows organisations to delve deep into customer reviews, feedback forms, social media mentions, and customer support interactions. By analysing this unstructured text data, businesses can gain valuable insights into what their customers like, and dislike, and what problems they face. This information can inform product development, marketing strategies, and customer experience improvements, ultimately leading to higher customer satisfaction and loyalty (and spend) while reducing churn.
Example: A call centre is an important component in enhancing customer experience and decreasing customer churn. Conventional models for predicting customer churn primarily rely on customer data gathered from past transactions and demographic information. However, this approach overlooks the inclusion of important context provided directly from customers, including their needs, desires, wishes, and emotions. Text Analytics is capable of generating categories like billing problems and product issues, which can then be incorporated into predictive models. This valuable information provides insights into the purpose behind each customer interaction and customers' motivations, resulting in a 2 times faster reduction in customer churns on average. (Source: Lexalytics)
Competitive Intelligence with Text Mining and Text Analytics
Staying ahead of the competition is a constant challenge in today's fast-paced business environment, particularly as businesses fight for share of a decreasing wallet. Text analysis can help businesses monitor their competitors' activities by analysing public sources like competitor reviews, news articles, and social media discussions. By tracking and analysing competitor sentiment and customer feedback, organisations can identify gaps in the market, capitalise on their competitors' weaknesses, and refine their own strategies to gain a competitive advantage.
Driving Operational Efficiency and Risk Management with Text Analysis Techniques
Text analysis can be applied internally to improve operational efficiency and risk management. By analysing employee feedback, internal communications, and performance reviews, businesses can identify issues within the organisation, such as low morale, communication breakdowns, or operational bottlenecks. This insight can be used to streamline processes, enhance employee satisfaction, and reduce operational risks.
Additionally, text analysis can also be used for monitoring external risks. By analysing news articles, social media, and industry reports, businesses can stay vigilant to emerging threats, such as public relations crises or market disruptions, allowing them to take proactive measures to mitigate these risks.
Example: As a food and beverage research company, Technomic’s data analysts had the challenge for categorising ingredients provided by a data vendor. Using a trained machine learning model, they were able to automatically put ingredients in the appropriate sub-categories at over 98% accuracy, saving Technomic up to 40 people-hours of labour per category. (source: Lexalytics)
Incorporating Text Analysis Outputs into Machine Learning Models
Text analysis can be a valuable component in building predictive models, especially when dealing with unstructured text data. For example, when predicting a customer's propensity to repurchase, a business could utilise unstructured feedback from multiple channels (email, call centre, and surveys) to identify key themes and sentiment indicators that can be incorporated into a predictive model. Here are some other areas where Text Analysis can be incorporated into Machine Learning models:
Feature Engineering: Text analysis helps convert unstructured text into structured features that can be used in predictive models. This involves techniques like sentiment analysis, entity recognition, and topic modelling. These extracted features can be combined with other structured data to improve the predictive power of models.
Sentiment Analysis for Customer Churn Prediction: Suppose a telecom company wants to predict customer churn. By analysing customer comments and reviews, sentiment analysis can be applied to gauge customer sentiment towards the company. Positive or negative sentiment scores can be used as features in predictive models to forecast the likelihood of a customer leaving the service.
Text Classification for Spam Detection: In email or message filtering systems, text classification can be employed to distinguish between spam and legitimate messages. By analysing the content of emails and messages, the model can predict whether an incoming message is likely to be spam or not, enabling automatic filtering.
Topic Modeling for Content Recommendation: Streaming platforms like Netflix use topic modelling to recommend content to users. By analysing the text descriptions, reviews, and user feedback for movies and TV shows, they can create predictive models that suggest content based on a user's viewing history and preferences.
Customer Support Ticket Resolution Time Prediction: Customer support teams can use text analysis on support tickets and queries. By examining the text for keywords, sentiments, and complexity, predictive models can estimate the time required, as well as the best team to resolve a customer issue, enabling better resource allocation and improved service levels.
Market Sentiment Analysis for Stock Price Prediction: Financial institutions can use text analysis to analyse news articles, social media posts, and press releases related to publicly traded companies. Sentiment scores extracted from the text can be integrated into predictive models to forecast stock price movements.
Review Ratings for Product Sales Prediction: E-commerce platforms can utilise sentiment analysis on product reviews to predict future sales trends. If a product consistently receives positive reviews, the predictive model can forecast higher sales for that product in the coming months.
Bringing it all Together
It is important to create and understand the relationships between the insights extracted from Text Analysis. For example, it is not enough for a retailer to understand that their customers are talking negatively (sentiment) about shoes (topic). The retailer would also need to understand other themes that are related to or occurring at the same time as these, such as sizing or availability, in order to provide richer context. Similarly, when analysing competitor reviews online it would be more useful to understand all the relevant context together. This may also require the integration of other operational data that could impact decision-making related to the Text Analysis.
Getting Started
Text Analytics is an exciting application of machine learning and NLP, that is also mature from a technology perspective. Even more exciting is combining the results with other traditional machine learning models and predictive analytics. While the technology exists, it is important to ensure that it is implemented appropriately.
Leverage our team's experience with machine learning and text analytics; get in touch with us to book a free consultation to understand the opportunities you have to implement unstructured text analytics and drive better decision-making in your business.
