w3hello.com logo
Home PHP C# C++ Android Java Javascript Python IOS SQL HTML videos Categories
How can I get the search result (in html) for a given keyword and iterate it for all search result pages in Talend?

You'll want to use a tLoop component set to use a for loop to loop through all the possible pages.

So you'd want to connect the tLoop in between your tFlowToIterate component and then concatenate the loop iteration variable into your URL with something like:

"http://www.absolventa.de/stellenangebote?page=" +
((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION")) +
"&query%5Bcity%5D=&query%5Bradius%5D=100&query%5Btext%5D=SAP&utf8=%E2%9C%93"

I'm not sure how you could make it end when the page would return nothing (you'd need a while loop that's condition could be set post the loop in processing) but if you just set your output file to append then if nothing is returned then it won't add anything to the file. You'd still have the problem that you'd need to set the amount of loops in your tLoop component to be high enough to cover any possible occurrence but then this would be making lots of pointless requests to fetch those empty pages.

You can maybe rework it into a while loop with some extra effort but I don't have much experience in this.





© Copyright 2018 w3hello.com Publishing Limited. All rights reserved.