Spurious characters from html files - text encoding issues?
keith.clarke at me.com
Mon May 17 10:58:55 EDT 2021
Thanks Ben, that’s really interesting. It never occurred to me that these html files might be anything other than simple plain text files, as I’d work with in Coda, etc., for years.
The local HTML files are storage of the HTML text pulled from the LiveCode browser widget, saved using the URL ‘file:’ option. I’d been working ‘live’ from the Browser widget’s html text until recently, when I’ve introduced these local files to split page ‘crawling’ and analysis activities without needing a database.
Reading the files back into LiveCode with the URL ‘file:’ option works quite happily with no text anomalies when put into a field to read. The problem seems to arise when I load the HTML text into a variable and then start to extract elements using LiveCode's text chunking. For example pulling the text between the offsets of say <p> & </p> tags is when these character anomalies have started to pop into the strings.
A quick test on reading in the local HTML files with the URL ‘binfile:’ option and then textDecode(tString, “UTF-8”) seems to reduce the frequency and size of anomalies, but some remain. So, I’ll see if re-crawling pages and saving the HTML text from the browser widget as binfiles reduces this further.
Thanks & regards,
> On 17 May 2021, at 12:57, Ben Rubinstein via use-livecode <use-livecode at lists.runrev.com> wrote:
> Hi Keith,
> The thing with character encoding is that you always need to know where it's coming from and where it's going.
> Do you know how the HTML documents were obtained? Saved from a browser, fetched by curl, fetched by Livecode? Or generated on disk by something else?
> If it was saved from a browser or fetched by curl, then the format is most likely to be UTF-8. In order to see it correctly in LiveCode, you'd need to two things:
> - read it in as a binary file, rather than text (e.g. use URL "binfile://..." or "open file ... for binary read")
> - convert it to the internal text format FROM UTF-8 - which means use textDecode(tString, "UTF-8"), rather than textEncode
> If it was fetched by LiveCode, then it most likely arrived over the wire as UTF-8, but if it was saved by LiveCode as text (not binary) then it _may_ have got corrupted.
> If you can see the text looking as you expect in LiveCode, you've solved half the problem. Then you need to consider where it's going: who (that) is going to consume the CSV. This is the time to use textEncode, and then be sure to save it as a binary file. If the consumer will be something reasonably modern, then again UTF-8 is a good default. If it's something much older, you might need to use "CP1252" or similar.
> On 17/05/2021 09:28, Keith Clarke via use-livecode wrote:
>> Hi folks,
>> I’m using LiveCode to summarise text from HTML documents into csv summary files and am noticing that when I extract strings from html documents stored on disk - rather than visiting the sites via the browser widget & grabbing the HTML text - weird characters being inserted in place of what appear to be ‘regular’ characters.
>> The number of characters inserted can run into the thousands per instance, making my csv ‘summary’ file run into gigabytes! Has anyone seen the following type of string before, happen to know what might be causing it and offer a fix?
>> I’ve tried deliberately setting UTF-8 on the extracted strings, with put textEncode(tString, "UTF-8") into tString. Currently I’m not attempting to force any text format on the local HTML documents.
>> Thanks & regards,
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
More information about the use-livecode