How can I scrape text from a web page that is generated by javascript?

Sat Jul 29 15:56:42 EDT 2017

FYI depending on what you are trying to get out you can also 

get the htmltext of widget "browser"; set the htmltext of templateField(); put the text of templateField()

and see if that may serve in some use case.

On 7/24/17, 12:41 PM, "use-livecode on behalf of Mike Bonner via use-livecode" <use-livecode-bounces at lists.runrev.com on behalf of use-livecode at lists.runrev.com> wrote:

    Something like autoit can probably be set up to automate things.  Set up an
    autoit script that does the scrape, then use os scheduling tools to run the
    script as often as you like.  I haven't done it in windows in a while, but
    I think "at" is deprecated.  Can't recall whats replaced it.

    On Mon, Jul 24, 2017 at 3:03 PM, Roger Eller via use-livecode <
    use-livecode at lists.runrev.com> wrote:

    > So... I can make it scrape, then save to a CSV, then read the CSV into LC
    > and use the data.  But how to automate the scraping and CSV saving at a
    > given interval, I'm not there yet.
    >
    > ~Roger
    >
    >
    > On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode <
    > use-livecode at lists.runrev.com> wrote:
    >
    > > The developer was also helpful when I was running into difficulty writing
    > > some of the parameter sets.
    > >
    > > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
    > > use-livecode at lists.runrev.com> wrote:
    > >
    > > > Scratch that.  It's amazing what watching the video or reading the docs
    > > can
    > > > do to enlighten the unenlightened.
    > > >
    > > >
    > > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <
    > > roger.e.eller at sealedair.com>
    > > > wrote:
    > > >
    > > > > I installed the plugin, but am unfamiliar with how to use it.  I
    > didn't
    > > > > see an obvious way to filter the page down to the current prices I
    > want
    > > > to
    > > > > grab.
    > > > >
    > > > > ~Roger
    > > > >
    > > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
    > > > > use-livecode at lists.runrev.com> wrote:
    > > > >
    > > > >> Don't know what to tell you.  I scrape a few hundred thousand pages
    > > at a
    > > > >> time from multiple sites with this plugin.
    > > > >>
    > > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
    > > > >> use-livecode at lists.runrev.com> wrote:
    > > > >>
    > > > >> > No workie. I create a new sitemap with a URL of
    > http://192.168.1.87
    > > > and
    > > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
    > > > >> > encounter that kind of problem. Even if I load the full url
    > default
    > > > page
    > > > >> > for that device, then copy paste the URL it STILL tells me it's
    > > > invalid.
    > > > >> >
    > > > >> > Bob S
    > > > >> >
    > > > >> >
    > > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
    > > > >> > use-livecode at lists.runrev.com> wrote:
    > > > >> > >
    > > > >> > > Roger,
    > > > >> > > I use a chrome plugin called "Web Scraper", and then I process
    > the
    > > > >> CSV in
    > > > >> > > LC - then again, I'm usually scraping many many many pages, and
    > > this
    > > > >> > > automates the process.
    > > > >> > > _______________________________________________
    > > > >> > > use-livecode mailing list
    > > > >> > > use-livecode at lists.runrev.com
    > > > >> > > Please visit this url to subscribe, unsubscribe and manage your
    > > > >> > subscription preferences:
    > > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
    > > > >> >
    > > > >> >
    > > > >> > _______________________________________________
    > > > >> > use-livecode mailing list
    > > > >> > use-livecode at lists.runrev.com
    > > > >> > Please visit this url to subscribe, unsubscribe and manage your
    > > > >> > subscription preferences:
    > > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode
    > > > >> >
    > > > >>
    > > > >>
    > > > >>
    > > > >> --
    > > > >> On the first day, God created the heavens and the Earth
    > > > >> On the second day, God created the oceans.
    > > > >> On the third day, God put the animals on hold for a few hours,
    > > > >>    and did a little diving.
    > > > >> And God said, "This is good."
    > > > >> _______________________________________________
    > > > >> use-livecode mailing list
    > > > >> use-livecode at lists.runrev.com
    > > > >> Please visit this url to subscribe, unsubscribe and manage your
    > > > >> subscription preferences:
    > > > >> http://lists.runrev.com/mailman/listinfo/use-livecode
    > > > >>
    > > > >
    > > > >
    > > > _______________________________________________
    > > > use-livecode mailing list
    > > > use-livecode at lists.runrev.com
    > > > Please visit this url to subscribe, unsubscribe and manage your
    > > > subscription preferences:
    > > > http://lists.runrev.com/mailman/listinfo/use-livecode
    > > >
    > >
    > >
    > >
    > > --
    > > On the first day, God created the heavens and the Earth
    > > On the second day, God created the oceans.
    > > On the third day, God put the animals on hold for a few hours,
    > >    and did a little diving.
    > > And God said, "This is good."
    > > _______________________________________________
    > > use-livecode mailing list
    > > use-livecode at lists.runrev.com
    > > Please visit this url to subscribe, unsubscribe and manage your
    > > subscription preferences:
    > > http://lists.runrev.com/mailman/listinfo/use-livecode
    > >
    > _______________________________________________
    > use-livecode mailing list
    > use-livecode at lists.runrev.com
    > Please visit this url to subscribe, unsubscribe and manage your
    > subscription preferences:
    > http://lists.runrev.com/mailman/listinfo/use-livecode
    >
    _______________________________________________
    use-livecode mailing list
    use-livecode at lists.runrev.com
    Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
    http://lists.runrev.com/mailman/listinfo/use-livecode