我正在使用 Express 和 Request 创建一个抓取工具.URL 数组被传递给 request,然后通过 for 循环进行迭代.解析完所有数据并解析所有数据后,回调将调用 res.send.
I'm creating a scraper using Express and Request. An array of URL's is passed to request which is then iterated through a for loop. Once all the data is parsed and all data is resolved the callback calls res.send.
我正在尝试将其转换为 promise,但我相信我使用的 for 循环不允许我这样做.如果循环导致了问题,是否还有其他方法可以对此进行编码并获得相同的结果?
I'm trying to convert this into promises but I believe the for loop I am using will not allow me. If the loop is causing the issue, is there another way I can code this and achieve the same result?
回调方法
function scrape(callback){ for(var i = 0; i < urls.length; i++){ request(urls[i], function(error, response, html){ if(!error && response.statusCode == 200){ // LOAD Cherio (jQuery) on the webpage var $ = cheerio.load(html); try{ var name = $(".name").text(); var mpn = $(".specs.block").contents().get(6).nodeValue.trim(); var jsontemp = {"MPN": "", "Name": "", "PriceList": {}}; jsontemp.MPN = mpn; jsontemp.Name = name; // Traverse the DOM to get tr tags and extract info $(".wide-table tbody tr").each(function (i, row) { var $row = $(row), merchant = $row. attr("class").trim(), total = $row.children(".total").text(); jsontemp.PriceList[merchant] = merchant; jsontemp.PriceList[merchant] = total; }); } catch(err){ console.log('Error occured during data scraping:'); } list.push(jsontemp); } else{ console.log(error); } count++; callback(); }); } } }); scrape(() => { console.log(count); if(count == urls.length){res.send(list)} });Promise 实现尝试
var urls = [ "test/", "test/2" ]; var list = []; var count = 0; scrape().then((data) => { list.push(data) if(count == urls.length){res.send(list)} }) .catch(error => console.log(error)) function scrape(){ for(var i = 0; i < urls.length; i++){ return new Promise(function (resolve, reject) { request(urls[i], function(error, response, html){ if(!error && response.statusCode == 200){ var $ = cheerio.load(html); try{ var name = $(".name").text(); var mpn = $(".specs.block").contents().get(6).nodeValue.trim(); var jsontemp = {"MPN": "", "Name": "", "PriceList": {}}; jsontemp.MPN = mpn; jsontemp.Name = name; // TRAVERSING DOM FOR DATA // $(".wide-table tbody tr").each(function (i, row) { var $row = $(row), merchant = $row. attr("class").trim(), total = $row.children(".total").text(); jsontemp.PriceList[merchant] = merchant; jsontemp.PriceList[merchant] = total; }); } catch(err){ console.log('Error occured during data scraping:'); } resolve(jsontemp); } else{ console.log(error); return reject(error); } count++; }); } } 推荐答案你需要将这些 promise 存储在一个列表中,然后调用 Promise.all 来获取所有的单个 promise:
You need to store these promises in a list, and then call Promise.all to get a single promise for everything:
function scrape() { var promises = []; // array of promises for(var i = 0; i < urls.length; i++) { var url = urls[i]; var promise = new Promise(function(resolve, reject) { // ... }; // add to array promises.push(promise); } // return a single promise with an array of the results // by using Promise.all return Promise.all(promises); }此外,在使用 var 时,不要在循环内的函数内使用循环变量(如 i).相反,您应该在 promise 回调函数之外声明一个 url 变量,或者用更新的 let 替换 var.
Also, don't use loop variables (like i) inside a function inside a loop when using var. Instead, you should declare a url variable outside the promise callback function, or replace var with the newer let.
更多推荐
从 For 循环返回多个 Promise
发布评论