谷歌搜索网络抓取与python中的关键字列表

本文介绍了谷歌搜索网络抓取与python中的关键字列表的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我正在尝试通过使用名称列表作为输入并在DataFame中获取数据集来在Google搜索上进行网络抓取.之前，我曾使用硒进行网络抓取，但我很难建立语法使用循环运行名称列表作为输入来获取结果并抓取每个页面.这是下面的我的Python代码:

I'm trying to do web scraping on Google search by using a list of names as inputs and get dataset in a DataFame. I used selenium for web scraping before, I am having a difficult time building syntax using loops to run a list of names as an input to get the results and scrap each page. Here is my Python code below:

baseUrl = 'www.google/search?q=' pluseUrl = input('CEO: ') url = baseUrl + quote_plus(pluseUrl) browser = webdriver.Chrome(r"C:\Users\...\chromedriver.exe") browser.get(url) table = browser.find_elements_by_css_selector('div.ifM9O') df = pd.DataFrame(columns = ['ceo', 'value']) values =[] for row in table: ceo = str(([c.text for c in row.find_elements_by_css_selector('div.kno-ecr-pt.PZPZlf.gsmt.i8lZMc')])).strip('[]').strip("''") value = str(([c.text for c in row.find_elements_by_css_selector('div.Z1hOCe')])).strip('[]').strip("''") ceo = pd.Series(ceo) value = pd.Series(value) df = df.assign(**{'ceo': ceo, 'value': value}) print(df)

这是将比尔·盖茨作为输入后的结果:

And here is the result after putting Bill Gates as an input:

CEO: Bill gates ceo value 0 Bill Gates Born: October 28, 1955 (age 64 years), Seattle...

任何建议将不胜感激.

推荐答案

尝试一下:

baseUrl = 'www.google/search?q=' browser = webdriver.Chrome(r"C:\Users\...\chromedriver.exe") input_list = ["Bill Gates", "Elon Musk", "Warren Buffet"] output = {} def scrape_ceo_list(list_of_ceo): for ceo in list_of_ceo: browser.get(baseUrl + ceo) // query selectors, dataframes etc as per original code // ... output[ceo] = df

output现在是数据帧的字典，以CEO名作为字典键.

output is now a dictionary of data frames, with CEO names as dictionary keys.