Problem using AppleScript of "system events" to click a location on screen

Ben Rubinstein benr_mc at cogapp.com
Thu Apr 27 11:42:58 EDT 2023


I had a need to click on an element in a web page loaded in a browser widget 
on a card.

There might be an elegant way to do this using javascript injected into the 
widget, but I'm too ignorant to figure it out.

So to save time (hah!), I though I could use the accessibility functionality 
to just click on that bit of the screen. I could get the location to click 
within the stack, then use globalLoc to convert it to screen coordinates.

(This is just a personal stack to achieve an objective, nothing that's ever 
going to be shared with anyone else, so any filthy/fragile method is OK if it 
works.)

I tested this in Script Debugger:

	tell application "System Events"
		click at {917, 667}
	end tell

It worked fine.

So then I tried the same script in my stack, via
	do ... as "applescript"

And after a brief spinning pizza, got the dry result "execution error".

I expected that this would happen because I needed to give it permission to 
control the computer; but after granting that in System Preferences, there was 
no change.

Just for fun, I tried building a standalone from my stack. This demonstrated a 
different effect: first I got the same result "execution error". Then I 
granted it permission for this standalone app to 'control the computer'. Now, 
there's still no evidence that it sends a click; but it gives a different 
result, which I think is the path to the UI element corresponding to that 
location. If the point is over a native control in the stack, it's something like
  	window "Ben Test Stack (1)" of application process "Ben Test Stack"
	of application "System Events"

(I don't think LiveCode controls are recognisable by the accessibility 
system.) If the point is over the browser widget, 'the result' is something like:
	group 2 of UI element 1 of scroll area 1 of group 1 of group 1
  	of window "Ben Test Stack (1)" of application process "Ben Test Stack"
	of application "System Events"

("Ben Test Stack" is the name of my stack and of the standalone. My guess 
about the complicated control path is that it reflects the complicated web 
page loaded in the browser widget.)

So that does suggest that there is something which doesn't quite work about 
giving the IDE permission to use the Accessibility Framework, but which does 
for a standalone. But still doesn't explain what the problem is.

I finally solved my problem with an even uglier hack: compiling the script 
from Script Debugger as an 'application', and then in my stack, instead of 
executing the applescript directly, using "launch" on that application. Yeuchh.

Can anyone shed light, either on how to grant the IDE permission to use the 
Accessibility controls; or on why the applescript wouldn't work?

TIA,

Ben



More information about the use-livecode mailing list