Synchronisation of sound and vision

[Ben Rubinstein]

I held off contributing to this discussion because it sounded like callbacks 
were a solid solution. However if that's not necessarily true it might be 
worth thinking about text tracks.

This depends of course on what effect you want to achieve, and what platforms 
you're targeting. But way back when (cue more CD-ROM nostalgia) we produced a 
CD-ROM including some interviews. We put the transcript in a text track in 
Quicktime, but hid the text track from the player, and intercepted it in code 
so that we could present it in the way we wanted.

I don't think LC let's you do that, but it does let you enable and disable 
tracks. So if you were happy with the default presentation of the text, that 
might be a very straightforward solution.

Ben

On 12/02/2020 18:57, Devin Asay via use-livecode wrote:
> Tore,
> 
> I would agree if callbacks were 100% reliable. I have tried them in the past and found that in some cases they were missed. I never had any trouble when using time indices. But I should say that I haven’t needed to do this for several years, and the callbacks in the new player object might be completely reliable.
> 
> In other ways creating time indices makes your application more flexible, however. It’s dead simple, for instance, to set up an application where you can click on a line of text and play just that line. Set the startTime, set the endTime, set the playSelection to true, start playing. Done. That would be a little more challenging if all you had was callbacks.
> 
> One of the great things about LiveCode is that there is almost always more than one way to do what you want.
> 
> Regards,
> 
> Devin
> 
> 
> On Feb 12, 2020, at 9:55 AM, Tore Nilsen via use-livecode <use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com>> wrote:
> 
> Using callbacks negate the need to fiddle with duration or  timescales and start or stop times. It uses the sampling intervals as is, regardless of time. In my opinion it is much easier than trying to calculate start and end times. You can easily handle large audio/video files using callbacks. I would recommend using one file per poem though, this simplifies the handling of the messages sent from the player. You can basically use the same message for all files, resetting a counter variable each time you load a new file to handle with line you would like to act upon.
> 
> You could also store the callbacks for each audio file in a text file and set the callbacks as a part of the handler used to load each audio file.
> 
> Regards
> Tore
> 
> 12. feb. 2020 kl. 16:49 skrev Devin Asay via use-livecode <use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com>>:
> 
> Graham,
> 
> Take a look at the duration and the timeScale properties of player objects. By dividing duration by timeScale you get the length of the video in seconds.
> 
> 
> put the duration of player  “foo” / the timescale of player  “foo” into totalSeconds
> 
> What you are contemplating is very doable, but you’ll have to do a fair amount of work to do to get the synching right. You can take one of several approaches:
> 
> - Calculate times as above to predict when to show/highlight the next line. Can be tricky with long video files and rounding errors.
> 
> - Check the currentTime property of the player to determine the startTime and endTime of each spoken line, and set the playSelection of the player to true. When the played segment ends, immediately load the following start and end times and play again. Something like this, from memory:
> 
> set the startTime of player “foo” to 444
> set the endTime of player “foo” to 999
> set the currentTime of player “foo” to the startTime of player “foo”
> set the playerSelection of player “foo” to true
> start player “foo"
> - Break up the video or audio file into separate files, one line per file, then play each succeeding file when the previous one reaches its end. The playStopped message is your friend here.
> 
> Like I said, it’s doable, but takes a bit of thought and planning, creating segment indexes, that sort of thing.
> 
> Hope this helps.
> 
> Devin
> 
> 
> On Feb 12, 2020, at 5:28 AM, Graham Samuel via use-livecode <use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com><mailto:use-livecode at lists.runrev.com>> wrote:
> 
> Thanks, that’s a start - I will look at the dictionary. I suppose the callbacks rely on one analysing how long each line/word takes the performer to say. It’s a lot of work, but there’s no way around it since potentially every line takes a different length of time to recite. If it’s too much work, I guess I can just display the whole text and have one callback at the end of each recording. Maybe that is really the practical solution for a large body of work (say all the Shakespeare sonnets, for example).
> 
> Anyway thanks for the hint.
> 
> Graham
> 
> On 12 Feb 2020, at 12:16, Tore Nilsen via use-livecode <use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com><mailto:use-livecode at lists.runrev.com>> wrote:
> 
> You will have to use the callbacks property of the player to do what you want to do. The callbacks list would be your cues. From the dictionary:
> 
> The callbacks of a player <> is a list of callbacks, one per line. Each callback consists of an interval number, a comma, and a message <> name.
> 
> 
> Regards
> Tore Nilsen
> 
> 
> 12. feb. 2020 kl. 11:25 skrev Graham Samuel via use-livecode <use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com><mailto:use-livecode at lists.runrev.com>>:
> 
> Folks, forgive my ignorance, but it’s a long time since I considered the following and wondered what pitfalls there are.
> 
> I have in mind a project where a recording of someone reading a poetry text (“old fashioned” poetry in metrical lines) needs to be synchronised to the display text itself on the screen, ideally so that a cursor or highlight would move from word to word with the speaker, although that would almost certainly involve too much work for the developer (me), or at least highlight lines as they are being spoken. I see that one would inevitably have to add cues to the spoken text file to fire off the highlighting, which is indeed an unavoidable amount of work, but can it be done at all in LC? For example, what form would the cues take?
> 
> TIA
> 
> Graham
> 
> 
> Devin Asay
> Director
> Office of Digital Humanities
> Brigham Young University
> 
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>