Speeding up get URL

[Bernard Devlin]

I think you've had a lot of good suggestions for solving this problem.
However, depending on the kind of data you're trying to parse out (and the
frequency with which that data changes), you might be better to let Google
or Yahoo do the search (using the kind of advanced search like:

"some meaningful phrase" OR "another meaningful phrase" allinurl:
somedomain.com

The search engine would then return the results to you, and you could then
proceed to download the actual pages that match, and parse them further.

I believe most of the major search engines have APIs for returning search
results as XML.  I certainly used Yahoo to to this before.

You'd need to look at the APIs for search to see if they contain features
that would work for the kind of information you're trying to find.

Bernard