How to write a selenium loop in Python? -


i want web-scrape data many different websites contains javascript code (thus why using selenium method information). working great, when try load next url long error message : 

> traceback (most recent call last):   file "c:/python27/air17.py", line 46, in <module>     scrape(urls)   file "c:/python27/air17.py", line 28, in scrape     browser.get(url)   file "c:\python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 268, in     self.execute(command.get, {'url': url})   file "c:\python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 254, in execute     response = self.command_executor.execute(driver_command, params)   file "c:\python27\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 464, in execute     return self._request(command_info[0], url, body=data)   file "c:\python27\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 487, in _request     self._conn.request(method, parsed_url.path, body, headers)   file "c:\python27\lib\httplib.py", line 1042, in request     self._send_request(method, url, body, headers)   file "c:\python27\lib\httplib.py", line 1082, in _send_request     self.endheaders(body)   file "c:\python27\lib\httplib.py", line 1038, in endheaders     self._send_output(message_body)   file "c:\python27\lib\httplib.py", line 882, in _send_output     self.send(msg)   file "c:\python27\lib\httplib.py", line 844, in send     self.connect()   file "c:\python27\lib\httplib.py", line 821, in connect     self.timeout, self.source_address)   file "c:\python27\lib\socket.py", line 575, in create_connection     raise err error: [errno 10061]

the data first website in csv file, when code tries open next website freezes, , error message. doing wrong?

from bs4 import beautifulsoup selenium import webdriver import time import urllib2 import unicodecsv csv import os import sys import io import time import datetime import pandas pd bs4 import beautifulsoup import mysqldb import re import contextlib import selenium.webdriver.support.ui ui  filename=r'output.csv'  resultcsv=open(filename,"wb") output=csv.writer(resultcsv, delimiter=';',quotechar = '"', quoting=csv.quote_nonnumeric, encoding='latin-1') output.writerow(['time','flight','from','airlane','aircraft','ihavetodeletethis','status'])   def scrape(urls):     browser = webdriver.firefox()     url in urls:         browser.get(url)         html = browser.page_source         soup=beautifulsoup(html,"html.parser")         table = soup.find('table', { "class" : "table table-condensed table-hover data-table m-n-t-15" })         datatable=[]         record in table.find_all('tr', class_="hidden-xs hidden-sm ng-scope"):             temp_data = []             data in record.find_all("td"):                 temp_data.append(data.text.encode('latin-1'))             datatable.append(temp_data)          output.writerows(datatable)          resultcsv.close()         time.sleep(10)          browser.quit()  urls = ["https://www.flightradar24.com/data/airports/bud/arrivals", "https://www.flightradar24.com/data/airports/fco/arrivals"] scrape(urls)

not sure browser.quit() have @ end of method such idea. according selenium doc :

quit()

quits driver , close every associated window.

i think browser.close()(as documented here) enough in loop. keep browser.quit() outside loop.


Comments

Post a Comment

Popular posts from this blog

python - Selenium remoteWebDriver (& SauceLabs) Firefox moseMoveTo action exception -

while performing simple hover test on firefox through selenium (3.4.0) python bindings (3.4.3) using of available os in saucelabs (except linux, latest available firefox version old), performing following actionchain: hover = actionchains(driver) hover.move_to_element(elm_men_menu).perform() it throws following error: taceback: file [...] in testhover hover.perform() file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/action_chains.py", line 80, in perform action() file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/action_chains.py", line 290, in <lambda> command.move_to, {'element': to_element.id})) file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute self.error_handler.check_response(response) file "/usr/local/lib/python2.7/...
Read more

html - How to custom Bootstrap grid height? -

why bootstrap grid has height 591.067 ? did height came ? .main-post { padding: 20px; background-color: #fff; border: 1px solid #d8d8d8; margin-bottom: 100px; } .main-post .post-categories { margin-bottom: 10px } .main-post h3 { margin: 0 0 10px; color: #777; letter-spacing: -1px; font-weight: bold } .main-post .post-author, .main-post .post-date, .main-post .post-comments { font-size: 12px } .main-post img { display: block; margin: 10px 0; } .main-post .post-summary { line-height: 1.7; color: #888 } .main-post { color: #999 } <div class="col-sm-6"> <div class="main-post"> <h3 class="post-title"> <a href="#"> post title </a> </...
Read more

transpose - Maple isnt executing function but prints function term -

im using maple im in worksheetmode tried using maple input , 2d input , want transpose matrix a: := `<|>`(`<,>`(1, .5, -2), `<,>`(.5, 9/4+b, 5+3*b), `<,>`(-2, 5+3*b, 18+9*b+4*a)); b:= transpose(a); when exectute sheet dont translated values, there same input. matrix looks same input matrix plus function term. you can see picture in following link: why arent functions executed? meanwhile b:=a^+ doing right way , transposed matrix. other functions return function body instead needed values... if using 2d input mode (the default) space you've got between transpose , bracketed (b) interpreted multiplcation. rid of such space. also, either load package @ start of document like, with(linearalgebra): before calling transpose command package, or call it's full name like, linearalgebra:-transpose(b);
Read more