ruby - Mechanize script keeps stopping with `fetch': 503 => Net::HTTPServiceUnavailable -

-

i trying run local ruby script using mechanize logs me onto website , goes through 1500 of webpages , parses information each of them. parsing works, length of time; script runs 45 seconds or so, , stops , reports:

/users/myname/.rvm/gems/ruby-1.9.3-p374/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:306:in `fetch': 503 => net::httpserviceunavailable http://example.com/page;53 -- unhandled response (mechanize::responsecodeerror)

i can't tell sure, feel due connection timeout. tried resolving in script long timeout (this script take 15 minutes run), still doesn't change anything. let me know if have ideas.

this script:

require 'mechanize' require 'open-uri' require 'rubygems'  agent = mechanize.new  agent.open_timeout   = 1000 agent.read_timeout   = 1000 agent.max_history = 1  page = agent.get('examplesite.com')  myform = page.form_with(:action => '/maint')  myuserid_field = myform.field_with(:id => "username") myuserid_field.value = 'myusername'   mypass_field = myform.field_with(:id => "password") mypass_field.value = 'mypassword'   page = agent.submit(myform, myform.buttons.first)  urlarray = [giant array of webpages here]  urlarray.each |term|     page = agent.get('' + term + '')     page.encoding = 'windows-1252'     puts agent.page.parser.xpath("//tr[4]/td[2]/textarea/text()").text + 'newlinehere' end

try calling sleep(1) in each loop. it's target server overwhelmed requests without pause.


Comments

Post a Comment

Popular posts from this blog

javascript - Count length of each class -

-
i have below code in this fiddle . html <span class="myname">john smith</span> <span class="myname">john smith</span> <span class="myname">john smith</span> <span class="myname">john smith</span> <span class="myname">john smith</span> <span class="myname">john smith</span> javascript $('.myname').each(function(){ var text = $(this).html().split(' '), len = text.length, test = $(this).val().length, result = []; console.log('outside: ' + test); for( var = 0; < len; i++ ) { result[i] = '<span class="name-'+[i]+'" >' + text[i] + '</span>'; console.log('inside: ' + test); } $(this).html(result.join(' ')); console.log('after
Read more

What design pattern is this code in Javascript? -

-
this question has answer here: why write “.call(this)” @ end of javascript anonymous function? [duplicate] 4 answers what kind of design pattern , significance of using closure? (function(){ // code here }).call(this); edit then difference between above code , following this keyword still refer same object in both ways. (function(){ // code here })(); thats invoked function expression. more info here: http://benalman.com/news/2010/11/immediately-invoked-function-expression/ the purpose run code while protecting scope (so variables declared within don't leak out global scope. update call sets value of this function being applied on. without it, value set window object, there set outer scope.
Read more

hadoop - Restrict secondarynamenode to be installed and run on any other node in the cluster -

-
i have have cloudera cdh3u4 cluster setup. have secondarynamenode daemon running on 1 of nodes. i have ensure user has root access node on cluster (for example, datanode not have other daemons running) should not able start instance of secondarynamenode daemon on node (thereby having 2 secondarynamenodes on cluster) how can achieve this? thanks in advance. add property hdfs-site.xml namedfs.secondary.http.address valuesecondarynamenode-host address or ip:50090
Read more