[macOS] Notarization, hardened runtimes, LCB, and executables

[Trevor DeVore]

Hey list,

I just spent a good portion of the last few days troubleshooting some
notarization errors I started receiving. I'm going to document what I did
so that someone else out there might benefit.

# The Original Problem

I have been notarizing my application for a while now without any issues.
This week an error message started coming back from Apple: "The executable
does not have the hardened runtime enabled.". (I found the error my looking
at the log file that Levure creates when packaging an application. It
contains the responses from the Apple notarization service which in turn
contain a url pointing to a detailed report of why an app fails
notarization.) A search led me to the following web page which tells me to
pass the `--options=runtime` to the `codesign` command line call to fix
this issue.

https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution/resolving_common_notarization_issues?language=objc

Easy enough. I updated the Levure packager script (see links below) that
packages up applications so that it will automatically sign the app bundle
and the files inside using this option as part of the packaging process.

I ran into some problems when trying to sign the Sparkle.framework that
ships with my app. It has an Autoupdate.app app inside of the
`./Content/Resources` folder which in turn contains a `filop` executable
that wasn't being signed. I had to spend some time making additional
improvements to the Levure packager script so that it would dig down into
the necessary folders looking for files that need signing.

After making the necessary changes so that all files were properly signed
application was successfully notarized by Apple. Phew!

# The Problem Caused By The Solution To The Problem

Unfortunately the solution of hardening the runtime uncovered yet another
problem. My application started throwing an error when trying to load some
LCB code. The LiveCode engine was unable to bind to some Foreign Function
Interface definitions in one of my LCB files. The problematic binding is
defined as follows:

private foreign handler C_CurrentKbdLayoutInputSource() returns ObjcId
binds to "c:Carbon.framework>TISCopyCurrentKeyboardLayoutInputSource"

# Entitlements

Since none of my code had changed I looked in the Console application to
see if there were any error messages. I saw a message in the console about
my app needing the `com.apple.security.automation.apple-events` permission
so I chased that red herring for a while. As part of my research I came
across entitlements. Entitlements are a way for you to tell macOS that your
app needs special permissions. I've used them before in the Mac App Store
but not for apps used outside the store.

The following web page lists the entitlements that are available for a
macOS app:

https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_disable-library-validation?language=objc

As I scanned through the list a couple of the options caught my eye. For
instance, `com.apple.security.cs.allow-dyld-environment-variables` sounds
like something that might apply to LCB.

My first test involved assigning a number of entitlements to my app to see
if it would launch. Sure enough it did. I then started removing
entitlements until I found the smallest list that would work. Here is the
final entitlement file that I came up with that allowed my LCB code to run
without causing an error or causing a spinning beach ball of doom:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "
http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
  <true/>
  <key>com.apple.security.cs.allow-dyld-environment-variables</key>
  <true/>
</dict>
</plist>

The above XML goes in a file with the same name as my app with an
`.entitlements` extension. That file is added to the `./Contents/Resources`
folder of the app bundle and Levure then uses that file when signing the
app. Learn more:
https://github.com/trevordevore/levure/wiki/How-do-I-include-additional-files-and-folders-in-my-application-builds%3F#copy-files-into-the-contentsresources-folder-of-your-packaged-macos-application

This same entitlements file also fixed a problem I was seeing with the
browser widget in my app. It too would lock up.

# Mostly Smooth Sailing

With a hardened runtime and the proper entitlements in place my LiveCode
app was being notarized successfully AND launching. Good times! Feeling
confident I decided to merge in a branch I had been working on that
included an executable created using pyinstaller so I could send it off to
QA. That didn't go well.

After packaging up the app I did a quick test to make sure the new feature
worked on my machine. It didn't. The error message was about some code
trying to execute that wasn't signed and some note about library validation
or something or other (I didn't write it down). After some searches I found
some other people who experienced similar problems with executables built
with pyinstaller (and some other installers I believe). The solution?
Entitlements.

Here is the entitlement file that I used when signing the executable built
with pyinstaller:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "
http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>com.apple.security.cs.disable-library-validation</key>
  <true/>
</dict>
</plist>

# Conclusion

My app and the executable built with pyinstaller are now running just fine
on macOS in a signed and notarized state. I've made further updates to the
Levure packager script so that entitlement files can be included alongside
of an executable nested in your Levure app folder and the packager will
find it and apply it during the code signing process. I do love a good cup
of automation!

If you want to see the signing scripts and some of the logic that goes into
deciding what to sign you can follow these links to the source code:

https://github.com/trevordevore/levure/blob/develop/packager/packager.livecodescript#L2302
https://github.com/trevordevore/levure/blob/develop/packager/packager.livecodescript#L2380

-- 
Trevor DeVore
ScreenSteps
www.screensteps.com