python - How to efficiently use webdriver to scrape and store it into a dataframe? -


i have data-frame hundreds of urls. try visit each url selenium , store source code in corresponding series. however, seems slow start web driver each visit. thinking starting web driver before apply function, don't know how pass 'driver' apply function. there better way this? 

def selenium_download (url):     driver = webdriver.phantomjs()     try:         driver.set_page_load_timeout(10)         driver.get(url)         source_page = driver.page_source         driver.quit()     except:         driver.quit()         source_page = 'time-out'      return source_page  def revisit_urls(df):     df['source_page'] = df['page_url'].apply(selenium_download)


Comments

Post a Comment

Popular posts from this blog

html - How to custom Bootstrap grid height? -

why bootstrap grid has height 591.067 ? did height came ? .main-post { padding: 20px; background-color: #fff; border: 1px solid #d8d8d8; margin-bottom: 100px; } .main-post .post-categories { margin-bottom: 10px } .main-post h3 { margin: 0 0 10px; color: #777; letter-spacing: -1px; font-weight: bold } .main-post .post-author, .main-post .post-date, .main-post .post-comments { font-size: 12px } .main-post img { display: block; margin: 10px 0; } .main-post .post-summary { line-height: 1.7; color: #888 } .main-post { color: #999 } <div class="col-sm-6"> <div class="main-post"> <h3 class="post-title"> <a href="#"> post title </a> </...
Read more

javascript - pass values from mssql to views in node -

being new node js , javascript . have been trying find solution problem . this app.js file var express=require ('express'); var bodyparser = require('body-parser'); var path = require ('path'); var sql = require('mssql'); var app = express(); const config = { user: 'sa', password: 'password', server: 'localhost\\sqlexpress', // can use 'localhost\\instance' connect named instance database: 'pcgdb', options: { encrypt: false // use if you're on windows azure } }; const request = new sql.request() //view engine app.set('view engine', 'ejs'); app.set('views', path.join(__dirname, 'views')) app.use(bodyparser.json()); app.use(bodyparser.urlencoded({extended: false})); app.get('/',function(req,res){ res.render('index') }); app.get('/new_project', function(req, res) { function(data,err){ ...
Read more

javascript - Understanding ExtJS Framework Syntax -

i have been given project uses sencha's extjs framework, no handover or documentation. trying come grips hoping people can pls assist me following syntax possibly point me specific site might explain code newcomer framework. i have been going through code , unsure doing, i.e.: if(val==1) { ext.getcmp('status').settext('awaiting approv'); } switch(flag) { case 1: if(val==0) { ext.getcmp('complex_first').el.replacecls('x-form-complex', 'x-form-simplex'); } else { ext.getcmp('complex_first').el.replacecls('x-form-simplex', 'x-form-complex'); } break; case 2: if(val==0) { ext.getcmp('complex_second').el.replacecls('x-form-complex', 'x-form-simplex'); } else { ext.getcmp('complex_second').el.replacecls('x-fo...
Read more