Unsigned Mac binary leads to warning message "The application is damaged..."

[Phillipus]

I'm putting this here to highlight that this is a known issue and also to discuss the possibility of signing the Mac Archi binary.

The issue is due to Gatekeeper on MacOS. If an app that is not signed is run on a Mac it can lead to one of two error dialog messages:

“Archi can’t be opened because it is from an unidentified developer.”

This is easy to solve by Control-click or by right clicking the app and selecting “Open” from the contextual menu that appears, then click “Open” in the dialog box.

Or the message may say:

"Archi is damaged and can’t be opened. You should move it to the Trash."

Normally one can solve this by following these instructions.

However, on MacOS Catalina (10.15) users are getting the "Archi is damaged..." message even after applying the sudo spctl --master-disable command.

This can be solved by following these instructions.

One thing to note - this last case applies to Archi 4.6 but not Archi 4.5.1. I don't know why this is. Perhaps something to do with Archi 4.5.1 using Eclipe 4.11 and Archi 4.6 using Eclipse 4.13.

The information on how to solve this issue is present:

• On the download page
• On the FAQ page
• In a readme file distributed with the Mac binary

However. some users miss this information and they then report the issue. (I can't blame them for that, users shouldn't have to deal with this).

[Phillipus]

To solve this problem we could sign the Mac binary.

To sign an app we need to:

• Pay Apple $99 a year for a dedicated individual Apple developer ID and account
• Set up two factor authentication for that Apple ID
• Set up that account on a physical Mac
• Ensure that account is separate from any personal Apple ID on that machine
• Add the developer certificates to the keychain on that Mac
• Set up the signing process
• Create a DMG file for the Archi build on a Mac
• Sign the binary for the Archi build on a Mac
• Upload and Notarize the binary with Apple
• Test the build on another Mac

There is no way to test this process without committing to $99. I'm not even sure if it will work.

I am averse to this for many reasons, the main one being that I dislike that we have to pay Apple to distribute free and open source software. We don't have to do this for Windows or Linux.

[Phillipus]

What would be ideal is if someone who has an Apple Developer account could do a test to see if signing the Archi.app is possible before we commit to this. If they could do that and let us know what they did that would be very helpful. :-)

This process is horribly complicated and I'm not sure if it can be automated.

If Archi Mac users would like to see this done, then they might consider helping out with this. Until then things will remain as they are and it is up to end-users to apply the workaround outlined above.

[Phillipus]

For an individual user the "quarantine bit" workaround is do-able but an inconvenience. An organization or company that may not be able to do this has the option of building from the source code and creating their own binary distribution, or perhaps signing the existing binary with their own certificate and distributing that internally.

Fortunately, Archi is open source and free and so there are many options, including contributing to solve this issue.

[jbsarrodie]

I'll see with some clients if one of them can help us on this aspect.

[hkmsn]

I have followed the instructions, the bit appears unset. However still getting the same issue. Using Catalina V10.15.1. Any other suggestions?

[Phillipus]

I have followed the instructions, the bit appears unset. However still getting the same issue. Using Catalina V10.15.1. Any other suggestions?

Nothing else to suggest. You have to ensure the path to the Archi.app is correct:

xattr -r -d com.apple.quarantine /path/to/Archi.app

[hkmsn]

Yes, followed all the instructions -- not working. Will try downloading Eclipse and build from scratch

[hkmsn]

I can get it to run, by downloading Eclipse/source, then using Product Export Wizard to create an executable. Following the instructions didn't work- only seen this issue, with the 4.6 release.

[Phillipus]

I Googled this issue some more today. It seems the quarantine bit is set by the app that downloads it, in this case the Browser. If you get the zip file by some other means like wget or curl it's not a problem.

For example:

curl -o Archi-Mac-4.7.1.zip https://www.archimatetool.com/downloads/archi/Archi-Mac-4.7.1.zip

See also MacDownApp/macdown#1106

[Phillipus]

For anyone who wants to sign the Archi.app for themselves, here you go:

http://www.zarkonnen.com/signing_notarizing_catalina

Of you could simply type in the Terminal:

xattr -r -d com.apple.quarantine Archi.app

[kentxu]

If I am not mistaken, Archi is bundling JRE 8. JRE 8 will not pass the notarization process (used to work before 2020.02) because it is considered legacy with reference to obsolete/unsafe Apple SDK API or something like that. Basically, all dylib need to be signed and pass notarization. Upgrade to Java 9+ should solve that problem.

It does pass notarization without much problem once you remove jre from the package.
result.txt
result_without_jre.txt

Apple's process is a bit complicated but it is totally possible to automate it with some script help. The basic flow is: build app -> sign code -> package app -> sign package -> upload for notarization -> wait for result -> verify the result -> stamp the package.

[kentxu]

Here is the script used in the testing. I wasn't sure if Archi needs any extra app.entitlements, but I have one ready from other projects where all options set to the least strict mode.

ACCOUNT_ID=developer_id
CERT_NAME=ABC
TOOLPATH=~/git/build_osx


codesign --verbose --force --timestamp --options runtime --deep  --entitlements $TOOLPATH/app.entitlements  --sign $CERT_NAME Archi/Archi.app

(cd Archi/; ditto -c -k --sequesterRsrc --keepParent Archi.app archi.app.zip )  

codesign -f -o runtime --timestamp $TOOLPATH/app.entitlements  -s $CERT_NAME Archi/archi.app.zip

# 
xcrun altool --verbose --notarize-app --primary-bundle-id "om.archimatetool.editor" -u $ACCOUNT_ID -p @keychain:mypass -t osx -f Archi/archi.app.zip

# check result
# xcrun altool --notarization-info $NOTARIZATION_REQID -u $ACCOUNT_ID  -p @keychain:mypass

#stamp if successful
xcrun stapler staple Archi/Archi.app

[Phillipus]

If I am not mistaken, Archi is bundling JRE 8.

No, it's AdoptOpenJDK 11.

[Phillipus]

@kentxu Thanks for looking at this. Can you enumerate the exact steps required?

For example:

1. Get Apple dev account
2. What to do for certificates?
3. What to do for keychain?
4. Etc....

[kentxu]

If I am not mistaken, Archi is bundling JRE 8.

No, it's AdoptOpenJDK 11.

My bad, when it failed, I just deleted the JRE folder without look into it much. Now you mentioned, the error message would be different if it is caused by legacy JRE.

Anyway, the JRE likely just needs to be resigned first or the notarization info is already cached. Or, the jre portion is already signed but not good enough and not recursively signed its dylib files. This usually calls for a search and deletion of __MACOSX. Either way, it is usually a good thing if the only problem Apple complains about is the bundled java and the app is already able to run on java 9+

In terms of the env setup steps, I only have a fuzzy idea now because I did those following apple's official doc last year.

1. sign up for apple developer account, pay $99 /year
2. you will be able to download the cert from dev site
3. install the cert and you should be able to start signing stuff
4. To notarize the signed package, you need to use acrun command to upload to the notarization server. This is where you need to supply a user id and password. You need to obtain a per app credential from apple (dev site) so that you do not expose the dev account info.
5. The keychain is optional. It is only needed so that you do not have to put the id/secret pair on the command line (you could do that though). What this does is to let the CLI retrieve a password from the keychain without directly spell it out. This is very useful for build systems where the build script is often shared by a group of users.

There is no other tools needed other than the latest xcode.

[ronaldoussoren]

The message "Archi is damaged and can’t be opened. You should move it to the Trash." is caused by having a partially signed bundle, in particular Archi.app/Contents/MacOS/Archi contains a signaturethat fails verification.

$ codesign --display -vv Archi.app
Executable=/Users/ronald/Downloads/Archi/Archi.app/Contents/MacOS/Archi
Identifier=SigningServlet-2317633887306001519-unsigned-eclipse
Format=app bundle with Mach-O thin (x86_64)
CodeDirectory v=20500 size=543 flags=0x10000(runtime) hashes=7+5 location=embedded
Signature size=9079
Authority=Developer ID Application: Eclipse Foundation, Inc. (JCDTMS22B4)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=19 Aug 2020 at 11:40:19
Info.plist=not bound
TeamIdentifier=JCDTMS22B4
Runtime Version=10.10.0
Sealed Resources=none
Internal requirements count=1 size=212

The easiest workaround is to strip the signature from the application.

I did this using the following command:

$ codesign_allocate -i Archi.app/Contents/MacOS/Archi -o Archi.app/Contents/MacOS/Archi -a x86_64 0

After that the application is not signed at all:

$ codesign --display -vv Archi.app
Archi.app: code object is not signed at all

I now get the "application from an unidentified developer" message, but that's easy to get around (right-click the app and select "Open" from the context menu for the first launch instead of double clicking).

The manual page says codesign_allocate ... -r should also work instead of ``codesign_allocate ... -a x86_64 0```, I haven't tested that.

Note that supporting the new M1 Macs will require signing the arm64 binaries, but that can be done using an ad-hoc signature (Developer ID is not required), the Xcode 12 release notes describe how to ad-hoc sign a binary.

[Phillipus]

@ronaldoussoren Thank-you for that very useful information, it answers a lot of questions.

The Archi executable binary is generated by Eclipse when exporting the product. It seems that the Eclipse Foundation started to sign this binary in Eclipse version 4.12 or 4.13. This explains why the "Archi is damaged" message started with Archi 4.6 when we moved to using Eclipse 4.13 (earlier versions of Archi don't have this problem).

Following your instructions I applied the codesign_allocate -i Archi -o Archi -r command to just the Archi binary executable file and re-packaged it in a new build of Archi 4.8.0 for Mac that's available to download now.

As you say, the warning message with this is now less daunting and can be bypassed by right-clicking and selecting "Open".

I've updated the FAQ to explain this:

https://www.archimatetool.com/faq/#mac

What's interesting is that only the Archi binary executable file needs to have its signing removed. Inside of the Archi.app bundle there is a jre folder containing a copy of the AdoptOpenJDK Java Runtime. Many of the files in that folder are signed and are executable.

Note that supporting the new M1 Macs will require signing the arm64 binaries

The current build of Archi for Mac is Intel only so I'm not sure if this required yet? At the moment there is no ARM version of Eclipse (or an AdoptOpenJDK JRE for ARM), so perhaps this is something that needs to be done when that is available?

[Phillipus]

What's interesting is that once Archi has been launched on Mac you'll not see the "Archi cannot be opened..." message any more but the quarantine bit is still set on the bundle.

xattr Archi.app gives an output of com.apple.quarantine

[ronaldoussoren]

@ronaldoussoren Thank-you for that very useful information, it answers a lot of questions.

My pleasure, glad I can help.

[...]

What's interesting is that only the Archi binary executable file needs to have its signing removed. Inside of the Archi.app bundle there is a jre folder containing a copy of the AdoptOpenJDK Java Runtime. Many of the files in that folder are signed and are executable.

If this is a complete copy of that JRE signatures will stay valid. The code signature for the binary in an application bundle generally covers not just the binary itself but the entire application bundle. That's what likely causing problems here. I can't give a good explication at this time, I'm starting to look into this for my own programs (written in Python).

Note that supporting the new M1 Macs will require signing the arm64 binaries

The current build of Archi for Mac is Intel only so I'm not sure if this required yet? At the moment there is no ARM version of Eclipse (or an AdoptOpenJDK JRE for ARM), so perhaps this is something that needs to be done when that is available?

Signatures are only required for arm64 binaries, so no changes are required for now. When an arm64 (or Universal 2) version of AdaptOpenJDK is created you'll have to resign the arm64 binary (or the arm64 half of the binary for Universal 2). In theory running codesign -s - -a arm64 Archi.app/Contents/MacOS/Archi after stripping the signatures should be enough.

[ronaldoussoren]

What's interesting is that once Archi has been launched on Mac you'll not see the "Archi cannot be opened..." message any more but the quarantine bit is still set on the bundle.

xattr Archi.app gives an output of com.apple.quarantine

See https://eclecticlight.co/2019/04/15/sandboxing-makes-quarantine-flags-almost-meaningless/ for more information the xattr. In short the value for the attribute contains amongst others flags that tell the system if the application should be check when launched, those flags are cleared after the first successful launch.

[Phillipus]

@ronaldoussoren Thanks for the additional information.

Ideally, what is needed is for someone with an Apple account to take the current Archi.app bundle, (perhaps also remove the quarantine bit) and then see if they can sign the whole bundle with their certificate.

The bundled JRE might give problems (as @kentxu mentioned in an earlier comment). Or maybe we have to sign our own JRE bundle? I'd prefer to bundle the JRE with Archi for Mac so that users don't have to download it separately or use the wrong one.

If I knew exactly what I had to do, and that it would definitely work, I might (begrudgingly, as this is open source) pay for an Apple developer account. On the other hand, asking the user to right-click on the app to launch it the first time is really not too much to ask for a free product. :-) But then, we might need to do this in the future as Apple tightens its grip...

[Phillipus]

In theory running codesign -s - -a arm64 Archi.app/Contents/MacOS/Archi after stripping the signatures should be enough.

As an experiment I applied the following on the Archi executable to add an ad-hoc signature:

codesign -s - -a x86_64 Archi.app/Contents/MacOS/Archi

I then uploaded the zip file containing Archi.app to the server and downloaded it via a Browser. This time the warning message is:

Opening from the right-click context menu works, but the message is probably more scary than before, so I'll stick with the unsigned version.

[Phillipus]

Note to self - there's a difference between Catalina and Big Sur when copying the Archi zip file via Finder. On Catalina I can copy the Archi zip file from a Windows network drive, unzip it, and run Archi with no warning messages. On Big Sur, I get the warning message.

[Phillipus]

Note to self - this also works:

codesign --remove-signature Archi

[Kechmade]

I’ve always opened it manually on different versions of iOS.
All you have to do is try to open the app (then it is blocked and listed on the general tab on the security & privacy panel in system preferences)

All you have to do is head to that to allow installation
Go to System Preferences > Security & Privacy and click on Open anyway.

Hope this helps

[Phillipus]

Hope this helps

Yes, that's another way to do it, thanks for pointing it out!

[tsanov]

Hi Phil,

Please find below the steps through which I managed to produce a signed and notarized DMG. Also, I am willing to contribute these DMGs to the project if you see any opportunity to delegate collaboratively this fun effort ...

1. Prerequisites
   1.1.I did it under BigSur with Xcode 12.5.1 installed but it should work with any Xcode 11+ command line tools only (I guess).
   1.2.Apple Dev subscription.
   1.3. We must create and download Developer ID Application certificate inside the Apple dev account (the pic shows the first step)

1.4. We should import the certificate in our Keychain and make sure it shows something like the below in the shell (I’ve got two more certs which shouldn’t be necessary for this specific job)

1.5.We must also create an app specific password as per https://support.apple.com/en-us/HT204397 and we must make sure we wrote it down.

2. Building, signing and notarizing
   2.1. We build the binaries as per https://github.com/archimatetool/archi/wiki/Building-the-Binaries
   2.2. We create a new DMG-construction folder Archi-Mac-4.8.1, then we copy com.archimatetool.editor.product/target/products/com.archimatetool.editor.product/macosx/cocoa/x86_64/Archi.app into it, then we create adjacent to Archi alias of Applications so that we can have the drag-and-drop “installation” experience when the DMG is ready.

2.3. I also downloaded the ZIP from https://www.archimatetool.com/download/ and I copied&pasted the jre dir, as well as the files Info.plist, Archi.ini and artifacts.xml into the respective locations inside my DMG wannabe dir.

2.4. We create “entitlement” file, plain XML, e.g. archi-entitlements, somewhere outside the DMG-construction dir, with the below contents (thanks to this thread https://developer.apple.com/forums/thread/117816):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
<key>com.apple.security.cs.disable-executable-page-protection</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<key>com.apple.security.cs.allow-dyld-environment-variables</key>
<true/>
</dict>
</plist>

2.5. In the shell, we can go to [whatever-the-DMG-construction-dir-is]/Archi.app/Contents/MacOS/ and we can sign the binary like this (we may opt not to cd of course):
codesign --entitlements [some-dir-per-step-2.4]/archi-entitlements --options runtime --force --sign "Developer ID Application: Nikolay Tsanov ([whatever-it-is-in-step-1.4])" ./Archi

2.6. We build a DMG from [whatever-the-DMG-construction-dir-is] with Disk Utility like this:

2.7. We codesign the DMG like this
codesign --force --sign "Developer ID Application: Nikolay Tsanov ([whatever-it-is-in-step-1.4])" ./Archi-Mac-4.8.1.dmg

2.8. We then send a notarization request from the shell like this (this takes a while, Request UUID is returned by Apple)

2.9. Apple will notify by email about the outcome, we can also check in the shell like this

3. I tested on another Mac under Catalina, no issues for the time being.

I hope this helps,
Nik

[tsanov]

Problem in stage 2.8:

xcrun: error: unable to find utility "altool", not a developer tool or in PATH

I don't have Xcode installed only the Command line tools.

@Phillipus try

xcode-select -r

then see if it finds it

[Phillipus]

xcode-select -r

Tried it. No luck.

[tsanov]

xcode-select -r

Tried it. No luck.

What is your Xcode tools version, i.e. xcodebuild -version ? I think I spotted somewhere that in v 13 (which is still in beta) they replaced altool with a new utility ...

[Phillipus]

xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance

I think I have to install XCode for Catalina. Downloading 11gb...

(11gb for one command line tool... 😢 )

...now extracting the Xcode_12.4.xip archive file...(this might take some time...)

[tsanov]

I hear you, 11 GB sounds like a stretch but you know what they say - there is no joy without suffering LoL

On a different note, I kind of managed to notarize a ZIP with --primary-boundle-id com.archimatetool.editor in the sense that Apple sent me confirmation of success by email, however the shell verification returns status "invalid" so it remains to test on another Mac which I'll do later today (I am in EST Canada).

zip -r Archi-Mac-4.8.1.zip ./Archi-Mac-4.8.1
xcrun altool --notarize-app --primary-bundle-id "com.archimatetool.editor" --username "[email protected]" --password "************" --asc-provider "**********" --file ./Archi-Mac-4.8.1.zip

[Phillipus]

I'm going with DMG... (50% extracted .xip file...)

[Phillipus]

Files in the DMG file are:

• Archi.app
• "docs" folder
• README!.txt file

But no shortcut to "Applications". I thought user might want to put somewhere else or put the "docs" folder somewhere else...

[tsanov]

You need to create alias to Applications manually, I created it by right-click in Finder. As for the docs and readme, it's not typical for DMGs to have anything except alias to Applications and the .app folder, otherwise end-users are getting confused... how is it in the Windows installation package? Anyway, you can put them in DMG, this won't break the notarization (I guess)

[Phillipus]

We have to ship those docs (User Guide PDF and change log) and readme, same as for Windows and Linux. I've seen DMG files with these extra files in.

[Phillipus]

xcrun altool has a new error:

(The username <my-email-address> is not a member of the provider <TEAM ID>.

[tsanov]

xcrun altool has a new error:

(The username <my-email-address> is not a member of the provider <TEAM ID>.

xcode-select -r ?

[Phillipus]

xcode-select -r ?

No, xcrun altool is running. It either doesn't like my email address or the ID from step 1.4

[Phillipus]

I've tried everything I can think of, but the process fails at the notarization stage.

I do this:

xcrun altool --notarize-app --primary-bundle-id "com.archimatetool.editor" --username "my_apple_email_address" --password "app-password" --asc-provider "TEAM ID" --file ./Archi-Mac-4.8.1.dmg

And get:

The username <my_apple_email_address> is not a member of the provider <TEAM ID>. Contact your team admin for assistance. (1296)

[tsanov]

I've tried everything I can think of, but the process fails at the notarization stage.

I do this:

xcrun altool --notarize-app --primary-bundle-id "com.archimatetool.editor" --username "my_apple_email_address" --password "app-password" --asc-provider "DevID" --file ./Archi-Mac-4.8.1.dmg

And get:

The username <my_apple_email_address> is not a member of the provider <DevID>. Contact your team admin for assistance. (1296)

I can't recall I've ever seen this snag ... Apple support? Now you're paying for it!

[Phillipus]

xcrun altool --list-providers -u "<my_apple_email>" -p "<app-password>"

Gives this (numbers have been changed)

ProviderName               ProviderShortname        PublicID                             WWDRTeamID 
-------------------------- ------------------------ ------------------------------------ ---------- 
Phillip Beauvoir|123456789 PhillipBeauvoir123456789 xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx TEAMID

[tsanov]

xcrun altool --list-providers -u "<my_apple_email>" -p "<app-password>"

Gives this (numbers have been changed)

ProviderName               ProviderShortname        PublicID                             WWDRTeamID 
-------------------------- ------------------------ ------------------------------------ ---------- 
Phillip Beauvoir|123456789 PhillipBeauvoir123456789 xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx TEAMID

On my end ProviderShortname and WWDRTeamID are the same, matching the the (ID) after my name on step 1.4, are they different for you?

[Phillipus]

On my end ProviderShortname and WWDRTeamID are the same, matching the the (ID) after my name on step 1.4, are they different for you?

Different. See https://developer.apple.com/forums/thread/119445

I used "PhillipBeauvoir123456789" and it's now uploaded.

Should I get confirmation email?

Edit: I've since found out that the --asc-provider argument is optional and I don't need to add it. See https://apple.stackexchange.com/questions/396311/im-trying-to-use-xcrun-altool-to-get-my-app-notarised-what-is-asc-provider-pa

[tsanov]

Yes, within a few minutes, looks like the below (I did cut off the ids):

[Phillipus]

Tada!

https://www.archimatetool.com/downloads/test/Archi-Mac-4.8.1.dmg

Edit - link removed, now at https://www.archimatetool.com/download

[tsanov]

Tada!

https://www.archimatetool.com/downloads/test/Archi-Mac-4.8.1.dmg

Way to go! I will now replace my impure package with the genuine package downloaded directly from the manufacturer!

[Phillipus]

Nik, I can't thank you enough. I wouldn't have been able to do this without your clear step-by-step instructions and your help here in this thread. I'm opening up a beer. Do you have a copy of LightBox for Archi and jArchi BTW?

[tsanov]

Nik, I can't thank you enough. I wouldn't have been able to do this without your clear step-by-step instructions and your help here in this thread. I'm opening up a beer. Do you have a copy of LightBox for Archi and jArchi BTW?

Don't make me feel uncomfortable please - writing down a step-by-step I've already known anyway is nothing compared to what you've done with Archi during the last 10+ years! And yes, I have the plug-ins (I am a patron)

[Phillipus]

I am a patron

Oops, so you are! And a long term patron, too. I appreciate that. In fact your patronage has more than paid for the Apple Developer fee.

Steps for me now are:

• Upload the DMG file for Archi downloads
• Try to understand this process better
• Document this process
• Automate this process

Until then Archi beta Mac versions might not be signed but I shall certainly sign the official builds.

Thanks again!

[Phillipus]

The best part of this will be removing this from the FAQ:

And closing this issue (but not yet...just in case)

[Phillipus]

The Archi Mac binary has been available as a signed and notarized DMG file for 2 days now with several downloads and no reports of any issues. I've downloaded it on macOS 10.12, 10.14, 10.15 and 11 without any problems.

I've updated my internal Ant scripts to automate the build process on a Mac and so, from here on, all Archi Mac builds will be signed and notarized, including Alpha and Beta builds. I've also put together a script to build for Mac M1 using Eclipse 4.20 and Azul Java for aarch64 for a future version of Archi (perhaps to tie in with the macOS Monterey release in autumn).

So, it's with great pleasure that I close this issue. 🎉

Many thanks to everyone who helped!

Edit - older versions of Archi have also been signed and notarized, as has the latest beta version. There's also a nice DMG installer graphic!