Synchronisation of sound and vision

[Tom Glod]

This would be so fun to work on, let us know what approach you used to get
the job done. Good luck.

On Thu, Feb 13, 2020 at 5:22 AM Ben Rubinstein via use-livecode <
use-livecode at lists.runrev.com> wrote:

> I held off contributing to this discussion because it sounded like
> callbacks
> were a solid solution. However if that's not necessarily true it might be
> worth thinking about text tracks.
>
> This depends of course on what effect you want to achieve, and what
> platforms
> you're targeting. But way back when (cue more CD-ROM nostalgia) we
> produced a
> CD-ROM including some interviews. We put the transcript in a text track in
> Quicktime, but hid the text track from the player, and intercepted it in
> code
> so that we could present it in the way we wanted.
>
> I don't think LC let's you do that, but it does let you enable and disable
> tracks. So if you were happy with the default presentation of the text,
> that
> might be a very straightforward solution.
>
> Ben
>
> On 12/02/2020 18:57, Devin Asay via use-livecode wrote:
> > Tore,
> >
> > I would agree if callbacks were 100% reliable. I have tried them in the
> past and found that in some cases they were missed. I never had any trouble
> when using time indices. But I should say that I haven’t needed to do this
> for several years, and the callbacks in the new player object might be
> completely reliable.
> >
> > In other ways creating time indices makes your application more
> flexible, however. It’s dead simple, for instance, to set up an application
> where you can click on a line of text and play just that line. Set the
> startTime, set the endTime, set the playSelection to true, start playing.
> Done. That would be a little more challenging if all you had was callbacks.
> >
> > One of the great things about LiveCode is that there is almost always
> more than one way to do what you want.
> >
> > Regards,
> >
> > Devin
> >
> >
> > On Feb 12, 2020, at 9:55 AM, Tore Nilsen via use-livecode <
> use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com>>
> wrote:
> >
> > Using callbacks negate the need to fiddle with duration or  timescales
> and start or stop times. It uses the sampling intervals as is, regardless
> of time. In my opinion it is much easier than trying to calculate start and
> end times. You can easily handle large audio/video files using callbacks. I
> would recommend using one file per poem though, this simplifies the
> handling of the messages sent from the player. You can basically use the
> same message for all files, resetting a counter variable each time you load
> a new file to handle with line you would like to act upon.
> >
> > You could also store the callbacks for each audio file in a text file
> and set the callbacks as a part of the handler used to load each audio file.
> >
> > Regards
> > Tore
> >
> > 12. feb. 2020 kl. 16:49 skrev Devin Asay via use-livecode <
> use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com>>:
> >
> > Graham,
> >
> > Take a look at the duration and the timeScale properties of player
> objects. By dividing duration by timeScale you get the length of the video
> in seconds.
> >
> >
> > put the duration of player  “foo” / the timescale of player  “foo” into
> totalSeconds
> >
> > What you are contemplating is very doable, but you’ll have to do a fair
> amount of work to do to get the synching right. You can take one of several
> approaches:
> >
> > - Calculate times as above to predict when to show/highlight the next
> line. Can be tricky with long video files and rounding errors.
> >
> > - Check the currentTime property of the player to determine the
> startTime and endTime of each spoken line, and set the playSelection of the
> player to true. When the played segment ends, immediately load the
> following start and end times and play again. Something like this, from
> memory:
> >
> > set the startTime of player “foo” to 444
> > set the endTime of player “foo” to 999
> > set the currentTime of player “foo” to the startTime of player “foo”
> > set the playerSelection of player “foo” to true
> > start player “foo"
> > - Break up the video or audio file into separate files, one line per
> file, then play each succeeding file when the previous one reaches its end.
> The playStopped message is your friend here.
> >
> > Like I said, it’s doable, but takes a bit of thought and planning,
> creating segment indexes, that sort of thing.
> >
> > Hope this helps.
> >
> > Devin
> >
> >
> > On Feb 12, 2020, at 5:28 AM, Graham Samuel via use-livecode <
> use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com
> ><mailto:use-livecode at lists.runrev.com>> wrote:
> >
> > Thanks, that’s a start - I will look at the dictionary. I suppose the
> callbacks rely on one analysing how long each line/word takes the performer
> to say. It’s a lot of work, but there’s no way around it since potentially
> every line takes a different length of time to recite. If it’s too much
> work, I guess I can just display the whole text and have one callback at
> the end of each recording. Maybe that is really the practical solution for
> a large body of work (say all the Shakespeare sonnets, for example).
> >
> > Anyway thanks for the hint.
> >
> > Graham
> >
> > On 12 Feb 2020, at 12:16, Tore Nilsen via use-livecode <
> use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com
> ><mailto:use-livecode at lists.runrev.com>> wrote:
> >
> > You will have to use the callbacks property of the player to do what you
> want to do. The callbacks list would be your cues. From the dictionary:
> >
> > The callbacks of a player <> is a list of callbacks, one per line. Each
> callback consists of an interval number, a comma, and a message <> name.
> >
> >
> > Regards
> > Tore Nilsen
> >
> >
> > 12. feb. 2020 kl. 11:25 skrev Graham Samuel via use-livecode <
> use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com
> ><mailto:use-livecode at lists.runrev.com>>:
> >
> > Folks, forgive my ignorance, but it’s a long time since I considered the
> following and wondered what pitfalls there are.
> >
> > I have in mind a project where a recording of someone reading a poetry
> text (“old fashioned” poetry in metrical lines) needs to be synchronised to
> the display text itself on the screen, ideally so that a cursor or
> highlight would move from word to word with the speaker, although that
> would almost certainly involve too much work for the developer (me), or at
> least highlight lines as they are being spoken. I see that one would
> inevitably have to add cues to the spoken text file to fire off the
> highlighting, which is indeed an unavoidable amount of work, but can it be
> done at all in LC? For example, what form would the cues take?
> >
> > TIA
> >
> > Graham
> >
> >
> > Devin Asay
> > Director
> > Office of Digital Humanities
> > Brigham Young University
> >
> > _______________________________________________
> > use-livecode mailing list
> > use-livecode at lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>


-- 
Tom Glod
Founder & Developer
MakeShyft R.D.A (www.makeshyft.com)
Mobile:647.562.9411