sentiment analysis (opinion mining)

What is sentiment analysis (opinion mining)?

Sentiment analysis, also referred to as opinion mining, is an approach to natural language processing (NLP) that identifies the emotional tone behind a body of text. This is a popular way for organizations to determine and categorize opinions about a product, service or idea. Sentiment analysis involves the use of data mining, machine learning (ML), artificial intelligence and computational linguistics to mine text for sentiment and subjective information such as whether it is expressing positive, negative or neutral feelings.

Sentiment analysis systems help organizations gather insights into real-time customer sentiment, customer experience and brand reputation. Generally, these tools use text analytics to analyze online sources such as emails, blog posts, online reviews, customer support tickets, news articles, survey responses, case studies, web chats, tweets, forums and comments. Algorithms are used to implement rule-based, automatic or hybrid methods of scoring whether the customer is expressing positive words, negative words or neutral ones.

In addition to identifying sentiment, sentiment analysis can extract the polarity or the amount of positivity and negativity, subject and opinion holder within the text. This approach is used to analyze various parts of text, such as a full document or a paragraph, sentence or subsentence.

Vendors that offer sentiment analysis platforms include Brandwatch, Critical Mention, Hootsuite, Lexalytics, Meltwater, MonkeyLearn, NetBase Quid, Sprout Social, Talkwalker and Zoho. Businesses that use these tools to analyze sentiment can review customer feedback more regularly and proactively respond to changes of opinion within the market.

How does sentiment analysis work?

Sentiment analysis uses machine learning models to perform text analysis of human language. The metrics used are designed to detect whether the overall sentiment of a piece of text is positive, negative or neutral.

Sentiment analysis generally follows these steps:

1. Collect data. The text being analyzed is identified and collected. This involves using a web scraping bot or a scraping application programming interface.
2. Clean the data. The data is processed and cleaned to remove noise and parts of speech that don't have meaning relevant to the sentiment of the text. This includes contractions, such as I'm, and words that have little information such as is, articles such as the, punctuation, URLs, special characters and capital letters. This is referred to as standardizing.
3. Extract features. A machine learning algorithm automatically extracts text features to identify negative or positive sentiment. ML approaches used include the bag-of-words technique that tracks the occurrence of words in a text and the more nuanced word-embedding technique that uses neural networks to analyze words with similar meanings.
4. Pick an ML model. A sentiment analysis tool scores the text using a rule-based, automatic or hybrid ML model. Rule-based systems perform sentiment analysis based on predefined, lexicon-based rules and are often used in domains such as law and medicine where a high degree of precision and human control is needed. Automatic systems use ML and deep learning techniques to learn from data sets. A hybrid model combines both approaches and is generally thought to be the most accurate model. These models offer different approaches to assigning sentiment scores to pieces of text.
5. Sentiment classification. Once a model is picked and used to analyze a piece of text, it assigns a sentiment score to the text including positive, negative or neutral. Organizations can also decide to view the results of their analysis at different levels, including document level, which pertains mostly to professional reviews and coverage; sentence level for comments and customer reviews; and sub-sentence level, which identifies phrases or clauses within sentences.

Types of sentiment analysis

Sentiment analysis systems fall into several different categories:

Fine-grained sentiment analysis breaks down sentiment indicators into more precise categories, such as very positive and very negative. This approach is similar to opinion ratings on a one to five star scale. This approach is therefore effective at grading customer satisfaction surveys.

Emotion detection analysis identifies emotions rather than positivity and negativity. Examples include happiness, frustration, shock, anger and sadness.

Intent-based analysis recognizes motivations behind a text in addition to opinion. For example, an online comment expressing frustration about changing a battery may carry the intent of getting customer service to reach out to resolve the issue.

Aspect-based analysis examines the specific component being positively or negatively mentioned. For example, a customer might review a product saying the battery life was too short. The sentiment analysis system will note that the negative sentiment isn't about the product as a whole but about the battery life.

Why is sentiment analysis important?

Sentiment analysis is an important way for organizations to understand how customers perceive and experience their products and brands. Increasingly, customer feedback is given online through a variety of unconnected platforms, such as Amazon product reviews and posts on social media platforms.

Organizations typically don't have the time or resources to scour the internet and read and analyze every piece of data relating to their products, services and brand. Instead, they use sentiment analysis algorithms to automate this process and provide real-time feedback.

Organizations use this feedback to improve their products, services and customer experience. A proactive approach to incorporating sentiment analysis into product development can lead to improved customer loyalty and retention.

What is sentiment analysis used for?

Sentiment analysis tools are used in nearly every industry for a variety of applications:

• Social media monitoring, a key strategy that tracks customer sentiments across social media platforms, such as Facebook, Instagram and Twitter.
• Monitoring brand awareness, reputation and popularity at a specific moment or over time.
• Analyzing consumer reception of new products or features to identify possible product improvements.
• Evaluating the success of a marketing campaign.
• Pinpointing a target audience or demographic.
• Conducting market research, such as emerging trends and competitive insights.
• Categorizing customer service requests and automating customer service.
• Customer support analysis to assess the effectiveness of customer support and monitor trending issues.

Benefits of sentiment analysis

The benefits of sentiment analysis include the following:

• Collecting large amounts of unstructured data from various sources.
• Tracking real-time customer feedback and sentiment about an organization's brand, products and services.
• Providing feedback on ways to improve products, services and customer experience.
• Getting data and feedback on problems with products and services.
• Gathering data and feedback that keeps customer support staff up to date on customer issues and improves their ability to respond.
• Tracking the effectiveness of customer support through support tickets and other online feedback.
• Automating customer service by identifying customers' sentiments and automatically sending them to relevant FAQ responses for resolution.
• Identifying emerging marketing trends, and understanding and improving what marketing strategies resonate with customers.
• Gaining competitive insights by monitoring comments about competitors.
• Establishing consistent criteria for evaluating sentiment instead of relying on subjective human analysis.
• Identifying and reacting to emerging negative sentiments before they escalate.
• Freeing employee time and energy for other tasks.
• Improving the empathy of organizations and customer service.

Challenges with sentiment analysis

Challenges associated with sentiment analysis typically include the following:

Neutral sentiments. Comments with a neutral sentiment tend to pose a problem for systems and are often misidentified. For example, if a customer received the wrong color item and submitted a comment, "The product was blue," this could be identified as neutral when in fact it should be negative.

Unclear language. Sentiment is challenging to identify when systems don't understand the context or tone. Answers to polls or survey questions like "nothing" or "everything" are hard to categorize when the context is not given; they could be labeled as positive or negative depending on the question. This is known as lexical ambiguity. Similarly, it's difficult to train systems to identify irony and sarcasm, and this can lead to incorrectly labeled sentiments. Algorithms have trouble with pronoun resolution, which refers to what the antecedent to a pronoun is in a sentence. For example, in analyzing the comment "We went for a walk and then dinner. I didn't enjoy it," a system might not be able to identify what the writer didn't enjoy -- the walk or the dinner.

Unclassifiable language. Computer programs have difficulty understanding emojis and irrelevant information. Special attention must be given to training models with emojis and neutral data so they don't improperly flag texts.

Ambiguous sentiments. People can be contradictory in their statements. Most reviews will have both positive and negative comments. This situation can be managed by analyzing sentences one at a time. However, sentences that contain two contradictory words, also known as contrastive conjunctions, can confuse sentiment analysis tools. For example, "The packaging was terrible but the product was great."

Named-entity recognition. This is when an algorithm cannot recognize the meaning of a word in its context. For instance, the use of the word "Lincoln" may refer to the former United States President, the film or a penny.

Small data sets. Sentiment analysis tools work best when analyzing large quantities of text data. Smaller data sets often won't provide the insight needed.

Language evolution. Language is constantly changing, especially on the internet where users are continually creating new abbreviations, acronyms, and using poor grammar and spelling. This level of variation and evolution can be difficult for algorithms.

Fake reviews. Algorithms can't always tell the difference between real and fake reviews of products, or other pieces of text created by bots.

Need for human intervention. Gartner finds that even the most advanced AI-driven sentiment analysis and social media monitoring tools require human intervention in order to maintain consistency and accuracy in analysis.

Machine learning techniques and technology underpin sentiment analysis models. Learn about 10 other uses for machine learning applications in business.