Query large CSV file as a data source?

[Keith Clarke]

Hi folks,
I’ve a 3GB CSV file with 18M rows from which I need to create a subset CSV file for the rows containing tString. The following script results in LiveCode becoming unresponsive, so I’m wondering if I’m missing an obvious trick or what alternative approach I might take...
on mouseUp pButtonNumber

-- Select CSV File

answer file prompt as sheet

put it into tFile

put tFile into field "File"

-- Process CSV extract

put URL ("file:" & tFile) into tText

put line 1 of tText into field "List"

filter lines of tText with "*tString*"

put tText into field tList

end mouseUp

I don’t normally deal with large data sets, so maybe LiveCode isn’t the best tool for this but Excel 365 seems to only offer CSV file import, not query.

Any thoughts greatly appreciated.

Thanks & regards,
Keith