Speeding up get URL

[Jim Ault]

Noel is correct.

Even Google will ban IP addresses of those machines that will execute too
many searches in a short time.  One answer is to use proxy servers, but that
is a more complex process.

One suggestion is to send an email to the support group for the "one domain"
and ask if there is a better way to get the info you want.  You would have
to say you are willing to register and play by their rules, but this could
get all the data you want in 20-30 minutes.  They might even allow you to
download one file from their FTP site.

If you are in the business of making more money for them, they will likely
help you.  They may already have such a service.

Jim Ault
Las Vegas

On 8/3/08 9:56 AM, "Noel" <noelf at nomigraphics.com> wrote:

> Yes, something like what you are describing could easily be confused
> with a DOS attack.
> 
> DOS attacks are done by flooding a server with requests for webpages
> to the point that the server crashes due to its inability to process
> all the requests.
> 
> Even if you are not considered a DOS attack, the company may not
> appreciate the bandwith that is being used for you to continually
> index their site and may deny your IP address access at some point.
> 
>   - Noel
> 
> At 10:38 AM 8/3/2008, you wrote:
>> It's always one domain, the same domain, and I have no control over
>> the domain or its hosting company.  The domain itself probably has
>> millions of pages.  Anybody can sell products thru them, and they
>> make it very easy to do so.  So there are probably thousands (or
>> more) folks with massive quantities of products for sale.
>> 
>> The worst thing will be if 40,000 products takes 4.5 hours (my last
>> two runs confirmed that), once it gets to hundreds of thousands it
>> could take *days*.  That would be bad.
>> 
>> I'm having other options for partial runs, but there's always got to
>> be the ability for a full run.
>> 
>> I'm not familiar with the ins and outs of a DOS attack, you mean
>> something like this could be confused for one?
>> 
>>> Just a thought...  One factor might be if your list has the same domain
>>> appearing as a contiguous block, the web server may be detecting that you
>>> are not a human browsing and slow down the transfer rate.  One of my hosting
>>> companies does this because they had bad experiences with denial of service
>>> attacks.
>>> 
>>> Hope this helps.
>>> 
>>> Jim Ault
>>> Las Vegas
>> 
>> 
>> --
>>   Dogs and bears, sports and cars, and patriots t-shirts
>>   http://www.villagetshirts.com
>>  WlND0WS and MAClNT0SH shareware games
>>  http://www.gypsyware.com
>> _______________________________________________
>> use-revolution mailing list
>> use-revolution at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-revolution
> 
> _______________________________________________
> use-revolution mailing list
> use-revolution at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution