w3hello.com logo
Home PHP C# C++ Android Java Javascript Python IOS SQL HTML videos Categories
crawl dynamic webpage for data using scrapy

Scrapy can't run javascript so you will have to analizy javascript code and do something similar in Python and Scrapy or recognize how javascript get data from server (which urls and parameters it use) and use it your script. It can be a lot work - first with Firebug in Firefox, then with Python and Scrapy.

If you have no idea how to do this than better use Selenium (or something similar) which simulate real browser and can run javascript. You will have to only say to Selenium which button press on page, what text put in forms, etc.

import requests
import json

# set request as GET
response =

# change json into dictionary
data =  json.loads(response.text)

#print data

import pprint


for x in data['resultSets']:
    print x['rowSet']

© Copyright 2018 w3hello.com Publishing Limited. All rights reserved.