How can I scrape text from a web page that is generated by javascript?

Mike Bonner bonnmike at gmail.com
Mon Jul 24 18:41:20 EDT 2017


Something like autoit can probably be set up to automate things.  Set up an
autoit script that does the scrape, then use os scheduling tools to run the
script as often as you like.  I haven't done it in windows in a while, but
I think "at" is deprecated.  Can't recall whats replaced it.

On Mon, Jul 24, 2017 at 3:03 PM, Roger Eller via use-livecode <
use-livecode at lists.runrev.com> wrote:

> So... I can make it scrape, then save to a CSV, then read the CSV into LC
> and use the data.  But how to automate the scraping and CSV saving at a
> given interval, I'm not there yet.
>
> ~Roger
>
>
> On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode <
> use-livecode at lists.runrev.com> wrote:
>
> > The developer was also helpful when I was running into difficulty writing
> > some of the parameter sets.
> >
> > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
> > use-livecode at lists.runrev.com> wrote:
> >
> > > Scratch that.  It's amazing what watching the video or reading the docs
> > can
> > > do to enlighten the unenlightened.
> > >
> > >
> > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <
> > roger.e.eller at sealedair.com>
> > > wrote:
> > >
> > > > I installed the plugin, but am unfamiliar with how to use it.  I
> didn't
> > > > see an obvious way to filter the page down to the current prices I
> want
> > > to
> > > > grab.
> > > >
> > > > ~Roger
> > > >
> > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> > > > use-livecode at lists.runrev.com> wrote:
> > > >
> > > >> Don't know what to tell you.  I scrape a few hundred thousand pages
> > at a
> > > >> time from multiple sites with this plugin.
> > > >>
> > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> > > >> use-livecode at lists.runrev.com> wrote:
> > > >>
> > > >> > No workie. I create a new sitemap with a URL of
> http://192.168.1.87
> > > and
> > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> > > >> > encounter that kind of problem. Even if I load the full url
> default
> > > page
> > > >> > for that device, then copy paste the URL it STILL tells me it's
> > > invalid.
> > > >> >
> > > >> > Bob S
> > > >> >
> > > >> >
> > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> > > >> > use-livecode at lists.runrev.com> wrote:
> > > >> > >
> > > >> > > Roger,
> > > >> > > I use a chrome plugin called "Web Scraper", and then I process
> the
> > > >> CSV in
> > > >> > > LC - then again, I'm usually scraping many many many pages, and
> > this
> > > >> > > automates the process.
> > > >> > > _______________________________________________
> > > >> > > use-livecode mailing list
> > > >> > > use-livecode at lists.runrev.com
> > > >> > > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >> >
> > > >> > _______________________________________________
> > > >> > use-livecode mailing list
> > > >> > use-livecode at lists.runrev.com
> > > >> > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> On the first day, God created the heavens and the Earth
> > > >> On the second day, God created the oceans.
> > > >> On the third day, God put the animals on hold for a few hours,
> > > >>    and did a little diving.
> > > >> And God said, "This is good."
> > > >> _______________________________________________
> > > >> use-livecode mailing list
> > > >> use-livecode at lists.runrev.com
> > > >> Please visit this url to subscribe, unsubscribe and manage your
> > > >> subscription preferences:
> > > >> http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >>
> > > >
> > > >
> > > _______________________________________________
> > > use-livecode mailing list
> > > use-livecode at lists.runrev.com
> > > Please visit this url to subscribe, unsubscribe and manage your
> > > subscription preferences:
> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > >
> >
> >
> >
> > --
> > On the first day, God created the heavens and the Earth
> > On the second day, God created the oceans.
> > On the third day, God put the animals on hold for a few hours,
> >    and did a little diving.
> > And God said, "This is good."
> > _______________________________________________
> > use-livecode mailing list
> > use-livecode at lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>



More information about the use-livecode mailing list