python - scrapy avoid crawler logging out -

-

i using scrapy library facilitate crawling website.

the website uses authentication , can login page using scrapy.

the page has url log out user , destroy session.

how ensure scrapy avoids logout page when crawling?

if using link extractors , don't want follow particular "logout" link, can set deny property:

rules = [rule(sgmllinkextractor(deny=[r'logout/']), follow=true),]

another option check response.url inside spider's parse method:

def parse(self, response):     if 'logout' in response.url:         return      # extract items

hope helps.


Comments

Post a Comment

Popular posts from this blog

html - Cut text on left side inside button while centering -

-
i need have button text (f.e. "add basket") fluid in width. if button width little show "add basket" must cut "basket" only. did achieve of splitting <span class="text">add basket</span> into 2 parts , reverse order them in dom <span class="text">basket</span><span class="text text-prefix">add to&nbsp;</span> and show them in correct order adding .text{ float: right; } however, text inside button must centered, not work far. here fiddle, i'd appreciate ideas: https://jsfiddle.net/5cavrrhc/ under circumstances, want avoid using js this. solution ok, here go: https://jsfiddle.net/6etabchg/4/ ok, co-worker , found solution works nicely. make button position: relative make wrapper display: flex; flex-wrap: wrap-reverse; justify-content: center; reverse order make both inner spans display: inline-block;flex-shrink: 0; make 1 span should remain or...
Read more

plugins - CodeIgniter support on netbeans 8.0 -

-
i new in codeigniter , trying project in netbeans 8.0. problem is, can't download plugin codeigniter support. shows error "some plugins require plugin php annotation api installed." i checked here solution , got - " http://ellislab.com/forums/viewthread/184716/ " but thing need - " http://alielwafdy.wordpress.com/2012/07/28/codeigniter-support-in-netbeans/ ". this supports netbeans 7.3.1 what should netbeans 8.0. please help i checked this question . of netbeans version - 7 . asking question find if there option in netbeans 8 , so not duplicate still there no direct plugin available though can use auto complete class helps auto complete words. copy php code http://blog.bobbyallen.me/wp-content/uploads/2012/08/netbeans_autocomplete_codeigniter.txt save file codeigniter project’s netbeans folder (in root, in same folder project.properties , project.xml) if have netbeans open you’ll need restart changes take effect!
Read more

php - Selecting items from MySQL for a status update feed efficiently while accounting for possible spam -

-
i creating "status update" feed users can post general messages can viewed other users (think status updates in facebook, twitter tweets, etc.). first of all, each post assigned rank score @ time post created , saved database. rank score combined time decay function used assign "true rank" each post @ time feed generated , presented user. true rank how items sorted. a complexity arose when introducing strategy combat "spam" (we don't want users able post rapidly , hog top spot in feed). strategy track how many times user has posted in, say, last hour. every subsequent post added within last hour, assigned rank score penalized. more posts user within last hour, greater rank score penalty. idea here push potential spam further down feed, while still giving chance display. following illustrates how feed sorted after 4 items posted same user if we're not accounting spam. item 1: posted 1 min ago. true rank = 0.99. user_id = 666. item 2: poste...
Read more