url no longer working as expected

[Richard Gaskin]

Likely the case. The expense of collating all that data and presenting it to their site visitors is considerable.  They use advertising to cover those costs.  If the data were easily scrapable, scrapers diminish revenue, putting the resource itself at risk.

Some data provides offer APIs.  When you see anti-scraping effects, look for API options (I saw none there but I didn't look deeply).  APIs take fewer resources to deliver, and may have strategic benefit for some data brokers.

But if they have scrape-prevention and no API, they're sending a clear signal: "We need to pay our bills, please send your traffic to our page so we can do that."

That said, I've come across stock APIs before, and while I don't recall many free ones there likely are some.

Richard Gaskin
FourthWorld.com



Paul Dupuis wrote:

> I get a response from Yahoos that is an html page with a 404
> information as part of it.
...
> I think this is Yahoo Finance not being able to detect the
> browser type and intentionally returning a 404 as a method
> of deterring screen scraping.
>
>
On 6/28/2024 1:04 PM, Hugh Senior via use-livecode wrote:
...
>> Problem:
>> Enter "https://uk.finance.yahoo.com/quote/SHEL.L/history/" into
>> any web browser and the page is displayed as expected.
>>
>> Use LC's URL command to access the same page direct returns a
>> 404
>> put url "https://uk.finance.yahoo.com/quote/SHEL.L/history/"
>>
>> Anyone got any insights?