Twitter Insight

Get insight about twitter data for any topic over a period of time

Overview

Twitter represents an important data source. It's a great place to collect real-time and historical data to analyze and get insights about any real-life situations such as the opinion of people about a specific topic, product, event, etc.

For this, we built Twitter Keyword Insight App to extract Twitter data for a specific keyword/keywords of any topic over a period of time. Then analyze sentiments for the extracted tweets and get insights about that topic.

Architecture

The application built based on the following nodes:

Extract tweets node:

Extract Twitter data for a specific keyword/keywords of any topic over a period of time using Twint library.

keyword:

  • The keyword you search for can be a hashtag or any word in tweets in general. So, If you want tweets with a hashtag only, put # with the keyword.

  • You can search for more than one keyword:

    • If you want tweets which contain all input keywords together, use AND:

    keyword1 AND keyword2 AND keyword3

    • If you want tweets which contain any of the input keywords, use OR:

    keyword1 OR keyword2 OR keyword3

Period:

  • start_date, end_date: must be in the format yyyy-mm-dd

  • The start date is included, but the end date is excluded.

Example: start_date: 2020-01-20 end_date: 2020-01-24

The extracted tweets will be for the days 20, 21, 22, and 23

  • limit: maximum number of extracted tweets.

In case there are no tweets for the specified period, the execution stopped and the following message will be printed: "The required keyword has not any tweet in the selected period."

Process twitter data node:

Apply some processing to the extracted tweets' texts such as removing stop words, punctuation, links, hashtags, and user mentions. in addition to lemmatize words.

Predict tweets sentiment node:

Predict the sentiment of each tweet (positive/neutral/negative) using VADER (Valence Aware Dictionary and sEntiment Reasoner) which is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media.

Extract tweets insight node:

General statistics over the sentiments of the tweets (positive, neutral, negative); this plot is dynamic, that will show stats for years, months, or days based on the selected period.

Tweets word cloud node:

Draw word cloud for the most common words in each sentiment(positive/negative/neutral).

Top Negative tweets node:

Find the top 5 negative tweets and take a screen shots for them.

Frequent hashtags node:

Extract the top 10 hashtags associated with the input keyword with the occurrence of each hashtag.

Sample Run - APP Editor

Assume we want to search for #StarTrekPicard for the period 07-12/1/2020 with maximum 1000 tweets, then select the Extract tweets node to enter the inputs in the right-side panel as following:

Then click on Execute APP button and wait until the app finish and give the app successfully executed. Then select each of the output nodes, in the output tab the following figures will be shown:

Frequent hashtags node output:

Extract insight node output:

Tweets word cloud node output:

Top negative tweets node output:

Here is a sample screenshot for one of the top negative tweets:

Sample Run - Form

If you want to run any application without going into the app details, Baseet.ai offers this option via Run an Example form in the overview section in the app details page.

For this app, the required inputs are the keyword/keywords you search on, start date & end date, and max number of retrieved tweets (limit). Default values are filled, and you can change them.

Click on Run Example. After the app runs successfully, the output section will be shown as follows:

The output of each output node will be shown, click on any image in the output to enlarge it. The same images as in the Sample Run - APP Editor section since the same input values are used.