Query large CSV file as a data source?
matthias_livecode_150811 at m-r-d.de
matthias_livecode_150811 at m-r-d.de
Tue Nov 3 10:09:32 EST 2020
Regarding the loop.
you could do a
put the detailed files
to get the file size of that csv file.
With that size you would now how often you have to run the loop
-
Matthias Rebbe
Life Is Too Short For Boring Code
> Am 03.11.2020 um 15:14 schrieb Keith Clarke via use-livecode <use-livecode at lists.runrev.com>:
>
> Thank you Matthias, I wasn’t aware of that ability to open the file and read its contents straight from the disk.
>
> A quick experiment looks positive - no hanging and fast access - so I just need to work on the loop.
>
> Much obliged.
>
> Best,
> Keith
>
>> On 3 Nov 2020, at 13:37, matthias rebbe via use-livecode <use-livecode at lists.runrev.com> wrote:
>>
>> Couldn't you read the file partially
>>
>> you would open the file for read
>>
>> open file tFile for read
>>
>> then in a repeat loop you could read e.g. 10000 lines and process the data
>>
>> read from file tFile for 10000 lines
>>
>>
>> Just a thought.
>>
>>
>> -
>> Matthias Rebbe
>> Life Is Too Short For Boring Code
>>
>>> Am 03.11.2020 um 13:14 schrieb Keith Clarke via use-livecode <use-livecode at lists.runrev.com>:
>>>
>>> Hi folks,
>>> I’ve a 3GB CSV file with 18M rows from which I need to create a subset CSV file for the rows containing tString. The following script results in LiveCode becoming unresponsive, so I’m wondering if I’m missing an obvious trick or what alternative approach I might take...
>>> on mouseUp pButtonNumber
>>>
>>> -- Select CSV File
>>>
>>> answer file prompt as sheet
>>>
>>> put it into tFile
>>>
>>> put tFile into field "File"
>>>
>>> -- Process CSV extract
>>>
>>> put URL ("file:" & tFile) into tText
>>>
>>> put line 1 of tText into field "List"
>>>
>>> filter lines of tText with "*tString*"
>>>
>>> put tText into field tList
>>>
>>> end mouseUp
>>>
>>> I don’t normally deal with large data sets, so maybe LiveCode isn’t the best tool for this but Excel 365 seems to only offer CSV file import, not query.
>>>
>>> Any thoughts greatly appreciated.
>>>
>>> Thanks & regards,
>>> Keith
>>> _______________________________________________
>>> use-livecode mailing list
>>> use-livecode at lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>>
>> _______________________________________________
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
More information about the use-livecode
mailing list