Due to recent ASO optimization work by our team, my colleague and I need to collect a large number of app reviews. The conventional method is to use the comment export function of third-party ASO optimization platforms, but this function can only export limited data. After searching online, I learned a method to scrape app reviews from the App Store using Python.
Conventional Method for Obtaining App Reviews#
Taking Qimai ASO Platform as an example, search for the desired app, such as WeChat, and then select "Rating & Reviews" on the Qimai details page of WeChat. Finally, view the "Review Details" and click the "Export Data" button to export the app review data.
The advantages and disadvantages of this method are as follows:
- Advantages: Reviews can be filtered by date, star rating, and keywords, and it is relatively more compliant with anti-fraud requirements.
- Disadvantages: Cannot directly export review data and requires manual text processing of the review data.
Python Scraping App Store App Reviews#
After ruling out the above conventional methods, we considered whether there is a method to directly scrape competitor's review data from the App Store. After some searching, I found a Python script on Zhihu that can scrape 500 reviews using the official Apple API.
Original post: How to get all the comments of an app on iTunes? Spider? Apple's provided API?
I tried this method and successfully scraped the review data of any ID product on the App Store and exported it in the form of an Excel spreadsheet. The usage is as follows:
Step 1: Install Python 3 environment, which can be achieved by downloading and installing Anaconda (Tsinghua University mirror).
Step 2: Use the pip command to install the XlsxWriter module. The code is as follows:
pip install XlsxWriter
Step 3: Search on Baidu: appstore + app name, enter the official website, and check the address bar to get the application ID.
Step 4: Save the following Python script as *.py format, for example, comments.py.
import urllib.request
import json
import xlsxwriter
print("This is an online tool to get the comment list of any app in appstore")
print("After running, a file named 'app评论.xlsx' will be generated")
page=1;
appid=input("Please enter the application ID:");
#appid=1182886088
workbook = xlsxwriter.Workbook('app评论.xlsx')
worksheet = workbook.add_worksheet()
format=workbook.add_format()
format.set_border(1)
format.set_border(1)
format_title = workbook.add_format()
format_title.set_border(1)
format_title.set_bg_color('#cccccc')
format_title.set_align('left')
format_title.set_bold()
title=['Nickname','Title','Comment']
worksheet.write_row('A1',title,format_title)
row=1
col=0
count=0
#Default loop 10 times
while page<11:
myurl="https://itunes.apple.com/rss/customerreviews/page="+str(page)+"/id="+str(appid)+"/sortby=mostrecent/json?l=en&&cc=cn"
response = urllib.request.urlopen(myurl)
myjson = json.loads(response.read().decode())
print("Generating data file, please wait......"+str(page*10)+"%")
if "entry" in myjson["feed"]:
count+=len(myjson["feed"]["entry"])
#Loop to write in column 1: Nickname
for i in myjson["feed"]["entry"]:
worksheet.write(row,col,i["author"]["name"]["label"],format)
row+=1
#Loop to write in column 2: Title
row=1+(page-1)*50
for i in myjson["feed"]["entry"]:
worksheet.write(row,col+1,i["title"]["label"],format)
row+=1
#Loop to write in column 3: Content
row=1+(page-1)*50
for i in myjson["feed"]["entry"]:
worksheet.write(row,col+2,i["content"]["label"],format)
row+=1
page=page+1
row=(page-1)*50+1
else:
print("Generating data file, please wait......100%")
break
if count==0:
print("Finished running, no data obtained. Please check if the input is correct!")
else:
print("Generation complete, please check the relevant file. A total of "+str(count)+" data obtained")
workbook.close()
Step 5: If you want to save the final generated app review spreadsheet in the root directory of the e:\ drive, you can run this script in the DOS window in that directory.
cd e:\
python comments.py
Then follow the prompts to enter the application ID and automatically scrape the reviews of the corresponding ID app in the App Store, as shown in the figure below.
