How to Scrape Twitter Data Without Coding Using Power Automate

Outline

1. Preamble
2. What is Power Automate
3. Get a Microsoft 365 Developer Account
4. Create a Streaming Dataset in Power BI
5. Create Power Automate Flow and Test the Solution
6. Visualize in Power BI
7. Conclusion

Preamble

Gone are the days when you need to know how to use Python, R, Java, etc before you can scrape data from the web, especially Twitter Data.

Some libraries will even require you to create a Twitter developer account which not everyone can follow through those processes. These various libraries are still important and they have their unique advantages however, for someone looking for a simpler way to interact with Twitter data, I present to you a No-Code approach to web scraping using Microsoft Power Automate.

What is Power Automate?

Microsoft Power Automate is part of the Power Platform Family and it is specialize in automating business processes. This is a valuable tool in today’s business environment as you can bring a digital solution to the business in a timely manner.

Here is the summary

1. Power Apps
2. Power Automate
3. Power BI
4. Power Pages
5. Power Virtual Agents:

Get A Microsoft 365 Developer Account

There is no substitute for hands-on experience and most of the services needed to acquire these skills and experiences aren’t free. This is the same for Microsoft 365 Apps and Services including Power BI Service (Power BI Desktop is free but Service is a paid version).

Microsoft 365 Developer Account gives you access to E5 License which comes with Office 365 Apps + other services including Power BI Pro as can be seen in the image below.

Image of Microsoft 365 Apps and Services for a Microsoft Developer Account

Watch this video to Get a Microsoft E5 License for FREE.
1. Register for an Outlook Account or Use an existing one
2. Signup for Microsoft 365 Developer Account. Don

Create a Streaming Dataset in Power BI

We need to first create a Streaming Dataset. This is where the tweets data will be stored in real-time. Follow the instructions below to create a Power BI streaming dataset.

Step 1: Log in to Power BI https://app.powerbi.com

Step 2: Navigate to your profile at the bottom left to locate “My Workspace” and follow the prompts to create streaming data as shown below

reload the page to see this gif replay. it only plays twice per view

Step 3: Follow the guide below to create columns that we will capture while scraping the data. for demo purposes, we will limit it to 3 columns (Date, Tweet, Location).

reload the page to see this gif replay. it only plays twice per view

Now that we have successfully created a streaming dataset in Power Bi, it is time to go to Power Automate to create our Twitter Data Scraping flow

Create Power Automate Flow

You can launch Power Automate directly from the Microsoft office portal https://office.com. follow the instruction below to launch Power Automate.

Note: You need to log in with your Microsoft 365 Developer Account you earlier created or use work or school credentials to log in (provided there is a Microsoft license on the work or school account).

Following the instruction below, you are creating the Flow that will scrape the data. we are creating an “Instance Cloud Flow” because we only need the flow to run when we trigger it ourselves.

reload the page to see this gif replay. it only plays twice per view

Now, Let’s add actions to the trigger above.

1. We will add action to search Twitter for Tweets with the hashtag #WakeUpNinjas
2. We will add another action to add the search tweets to the streaming Power BI Dataset we earlier created “TwitterDataScrape”

If you are using the “Search Tweet” Action for the first time, it will require that you log in to your Twitter account. follow the prompts to log in

reload the page to see this gif replay. it only plays twice per view

Let’s add the results from the tweet to the columns in the dataset. this will connect the tweets retrieved to the Power BI Dataset.

We will also test the flow by manually running it. Follow the guide below to do this

reload the page to see this gif replay. it only plays twice per view

Visualize in Power BI

It’s time to visualize this in Microsoft Power BI. Follow the guide below

reload the page to see this gif replay. it only plays twice per view

In total, 29 Tweets were scraped with the hashtag #WakeUpNinjas

Conclusion

In this post, I was able to show you how to scrape tweet data without writing code. This is a Low-Code way of getting people to learn how to do some technical stuff without a technical background.

Limitations
The Search Tweet can only retrieve a maximum of 100 Tweets. including retweets.

Next Article

My next article will show you how to create a pipeline that collects real-time tweet and save them automatically in Power BI Dataset.

Do follow me, clap, and share this post if you find it useful 😊