Deprecation of HTTP Basic Auth and OAuth 1.0a

[opk12]

Thank you for this app. Just a heads-up if you didn't know, that OSM is slowly deprecating HTTP Basic Auth and OAuth 1.0a in favor of OAuth 2 (see announcement and tracking issue).

Related: #104

[Krakonos]

Thanks for the notice. I'm looking into this with a higher priority. There is OAuth2 library in Qt. However, reading osm docs it appears I need to generate client_id and secret. However, I can't keep the secret secret.

I looked briefly into josm and could not find a secret in the source code, not sure if they just inject it at buildtime though.

Do you have any visibility into the preferred way to do it?

[opk12]

Ah, this was briefly discussed today in the IRC chat #osm-dev (OFTC webchat, Matrix) and BTW there's also #josm if you ever need (OFTC, Matrix).

OAuth 2: How to best deal with the secret from step 1 at wiki.osm.org/wiki/OAuth in an open source app? If it's published to the GitHub repo, it won't be a secret anymore?

In that case you should mark the app as non-confidential when you register it
Best practice is also to use PKCE when authorising
see https://www.oauth.com/oauth2-servers/pkce/

I'm going to add a brief mention in that wiki article; feel free to link other OSM docs where you think it should be mentioned.

[Krakonos]

Thanks, that's helpful. I got it working. However, I'm not sure how to check if it's doing what it's supposed to do.

1. The code seems to work without the secret entirely.
2. When removing code_challenge and code_verifier, nothing really breaks.

I would expect the OSM server should require at least one of these things to be present?

Anyway, it looks like I can successfully use the OAuth2 flow with OSM API, so I'll integrate it with the rest of the system. The only big downside I see now is I seem to require a separate client id for every server (main/test api/3rd party servers).

[tsmock]

1. It should, assuming that PKCE is used. Technically, I don't think you need to use PKCE with OAuth 2.0, but you should. It is highly likely to be a mandatory part of the OAuth 2.1 specification (it is part of the current draft).
2. I would keep both code_challenge and code_verifier if you are using PKCE -- if removing them doesn't currently break something, then it is entirely possible that this is a bug in doorkeeper (the OAuth 2 library the OSM website is using) that will be fixed at some point. Or they aren't enforcing some of the recommended security practices yet.

When I was adding support for OAuth 2 in JOSM, I implemented a bunch of "optional" stuff that is likely to become mandatory with OAuth 2.1 and didn't implement some things that were optional in OAuth 2.0 but were removed/deprecated in the OAuth 2.1 draft.

If you are writing your own OAuth lib instead of using someone else's, I would highly recommend implementing PKCE. But you should probably use someone else's OAuth library.

I don't know how you are intending to implement the OAuth redirect, but in JOSM I'm using the remote control endpoint /oauth_authorization.

The only big downside I see now is I seem to require a separate client id for every server

Yep. That is a downside. I'm currently only planning on hardcoding the main api and test api oauth client ids in JOSM. There is an RFC for dynamic client registration, but it isn't currently implemented in the library the OSM website is using. There is also an RFC for getting OAuth endpoints, which also isn't implemented in the library that the OSM website is using.

EDIT: The OSM website now implements the RFC for getting OAuth endpoints, see https://www.openstreetmap.org/.well-known/oauth-authorization-server .

[mmd-osm]

Basic Auth and OAuth 1.0a have been turned off on July 1st now. Merkaartor upload to osm is currently not working anymore.

[nataraj-hates-MS-for-stealing-github]

I got it working

Have it been commit to any branch or repo? Can you share that code?
Merkaartot is useless right now, and I'd like to continue using it.

[Krakonos]

Work in progress on my personal branch. I have some local stuff not there yet. I've got the auth working + the UI, but need to put that together.

https://github.com/Krakonos/merkaartor/commits/oauth2

[Krakonos]

Well, I pushed the recent version. The refactor is mostly OK, download works. But it seems since last time I tested the flow, something changed and my Qt oauth component sends redirect url as http://localhost:1337/ instead of http://127.0.0.1:1337/. Unfortunately, I can not set localhost on the osm.org website, due to non-IP redirect addresses needing https.

I'll continue tomorrow, but you can test if it works for you.

One caveat is it will NOT work against other APIs than main osm.org due to hardcoded client_id. I will at least add client id for the testing API tomorrow.

[Krakonos]

Ok, so the IP is fixed, well, hacked. However, now I'm stuck getting the actual bearer token, and the QT handler eats the error (it likely is in the body, which it ignores).

     8.588 (WW) [ default ] Token request error:  1 "Error transferring https://www.openstreetmap.org/oauth2/token - server replied: "
     8.588 (WW) [ qt.networkauth.oauth2 ] Token request failed: "Error transferring https://www.openstreetmap.org/oauth2/token - server replied: "

[Krakonos]

Just pushed to the branch (https://github.com/Krakonos/merkaartor/commits/oauth2 ) with basic functionality.

What works:

• Adding OAuth2 based OSM server in preferences. You need to invoke "login" manually for the first time and every time the session token expires (the wiki claims the tokens do not expire). The token is stored and loaded on next use.
• Downloading and uploading data.
• login to both openstreetmap.org and master.apis.dev.openstreetmap.org

What does not work:

• Uploading preferences to OSM profile storage. I'm not sure if this feature is used, but should be easy to port.
• Unauthenticated error messages are a bit weird, since the error is in the request body. Seems to be a change in the OSM infrastructure, since I see the same for other errors (too big bbox) and those used to be handled well. I'll work on the error reporting next.
• Authentication failure is silent. I'll work on this as part of error reporting.
• 3rd party servers (they should still be able to use username/password, if there is interest, I'll add configuration for the client_id).
• PKCE flow. Not a big deal now since it's not required, but would be nice to have to avoid some man in the middle attacks.

If you've got time, I'd appreciate some rough testing. So far it works for me (tm).

[dreirund]

I now cannot upload data anymore.

When I upload, I get the error message

There was an error uploading this request (403)
Server message is 'Error transferring https://www.openstreetmap.org/api/0.6/changeset/create - server replied: Forbidden'

So Merkaartor seems no longer usable right now, making this an important/ deal breaking issue.

(I use latest release 0.19.0 from my distribution. If you have already fixes for this, maybe do a new release?)

[Krakonos]

If you can, please test the branch marked above. I'm busy over the week, but I think I'll merge & release it over the weekend.

[dreirund]

I'm busy over the week, but I think I'll merge & release it over the weekend.

OK, great!, so I will just wait until my distribution picks up.

[nataraj-hates-MS-for-stealing-github]

If you can, please test the branch marked above. I'm busy over the week, but I think I'll merge & release it over the weekend.

Sorry for delay...

I managed building the code. But not from the first attempt. What have I done:

• I am using Debain, so I run apt build-dep for Debian's version of merkaartor
• Created die __build in source dir, and run cmake .. from it
• Installed qt6-networkauth-dev, qt6-5compat-dev and libprotobuf-dev that was still missing (I wonder why does it need protobuf)
• when I managed to run cmake .. I ran make and get following error:

CMakeFiles/merkaartor_autogen.dir/build.make:77: *** target pattern contains no '%'.  Stop.
make[1]: *** [CMakeFiles/Makefile2:281: CMakeFiles/merkaartor_autogen.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

Same story I get when I build it master branch.

I looked at the make file with error, found protobuf mentioned there, and tried to configure with cmake -DPROTOBUF=OFF ..

This helped. So I built it and run it.

I tried to login via oauth2 method, but failed:

I am using KDE on this computer, Then I pressed "Login button" it opened knqueror browser (I never changed browser defaults here), then konqueror automatically reopend that page in firefox (I do not know why, it changed to that behavior after one update, I've never configured it). This page has URL "https://www.openstreetmap.org/api/oauth2/authorize?client_id=wv3ui28EyHjH0c4C1Wuz6_I-o47ithPAOt7Qt1ov9Ps&code_challenge=k-8NhjKbEjfP_hPtscyFkRO_ZmUghWjySZhDkyupp5E&code_challenge_method=S256&redirect_uri=http://127.0.0.1:1337/&response_type=code&scope=read_prefs%20write_prefs%20write_api%20read_gpx%20write_gpx%20write_notes&state=BNF39NRK" and it said, if I translate it back from Russian:

File not found

Could not find a file/directory/API operation with the same name on the OpenStreetMap server (HTTP 404)

Feel free to contact the OpenStreetMap community if you find a broken link/error. Write down the exact URL of your request.

In the console Merkaartor wrote following lines:

    41.814 (DD) [ default ] Start login...
    41.814 (WW) [ default ] Unknown OSM API host  "www.openstreetmap.org" , assuming alias to openstreetmap.org
    41.814 (DD) [ default ] Base URL:  QUrl("http://www.openstreetmap.org/api/0.6")
    41.814 (DD) [ default ] Authorization URL:  QUrl("http://www.openstreetmap.org/api/oauth2/authorize")
    41.814 (DD) [ default ] Access Token URL:  QUrl("http://www.openstreetmap.org/api/oauth2/token")
    41.815 (DD) [ default ] Stage:  1 with params QMultiMap(("client_id", QVariant(QString, "wv3ui28EyHjH0c4C1Wuz6_I-o47ithPAOt7Qt1ov9Ps"))("redirect_uri", QVariant(QString, "http://127.0.0.1:1337/"))("response_type", QVariant(QString, "code"))("scope", QVariant(QString, "read_prefs write_prefs write_api read_gpx write_gpx write_notes"))("state", QVariant(QString, "BNF39NRK")))
    41.815 (DD) [ default ] Requesting Authorization.
    41.815 (DD) [ default ] Stage:  1 with params QMultiMap(("client_id", QVariant(QString, "wv3ui28EyHjH0c4C1Wuz6_I-o47ithPAOt7Qt1ov9Ps"))("code_challenge", QVariant(QString, "k-8NhjKbEjfP_hPtscyFkRO_ZmUghWjySZhDkyupp5E"))("code_challenge_method", QVariant(QString, "S256"))("redirect_uri", QVariant(QString, "http://127.0.0.1:1337/"))("response_type", QVariant(QString, "code"))("scope", QVariant(QString, "read_prefs write_prefs write_api read_gpx write_gpx write_notes"))("state", QVariant(QString, "BNF39NRK")))
    48.115 (WW) [ default ] OAuth2 reply handler timed out.
    48.115 (DD) [ default ] Failed!

I do not know how does this OAuth2 communicate with the browser, but I whould suspect, that this issue with redirecting might break everythig.

As far as I know Qt has some limited html-rendering engine that can be used to handle that OAuth2 stuff. Guess better not try to rely on OS browser facility, as it is not reliable, and use browser of our own, if it is ever possible.

[nataraj-hates-MS-for-stealing-github]

Tried to do the same thing on laplop with LXQt:

• Login button even did not try to run the browser, showed "Failed" response in the same dialog window
• In console I saw following messages:

   256.167 (DD) [ default ] Start login...
   256.167 (WW) [ default ] Unknown OSM API host  "" , assuming alias to openstreetmap.org
   256.167 (DD) [ default ] Base URL:  QUrl("")
   256.167 (DD) [ default ] Authorization URL:  QUrl("oauth2/authorize")
   256.167 (DD) [ default ] Access Token URL:  QUrl("oauth2/token")
   256.167 (DD) [ default ] Stage:  1 with params QMultiMap(("client_id", QVariant(QString, "wv3ui28EyHjH0c4C1Wuz6_I-o47ithPAOt7Qt1ov9Ps"))("redirect_uri", QVariant(QString, "http://127.0.0.1:1337/"))("response_type", QVariant(QString, "code"))("scope", QVariant(QString, "read_prefs write_prefs write_api read_gpx write_gpx write_notes"))("state", QVariant(QString, "jRASHX8Y")))
   256.167 (DD) [ default ] Requesting Authorization.
   256.167 (DD) [ default ] Stage:  1 with params QMultiMap(("client_id", QVariant(QString, "wv3ui28EyHjH0c4C1Wuz6_I-o47ithPAOt7Qt1ov9Ps"))("code_challenge", QVariant(QString, "29hhwtj0rbuj3dvgqVc2B_kvDCOHJl4uLbC5a8mtU3E"))("code_challenge_method", QVariant(QString, "S256"))("redirect_uri", QVariant(QString, "http://127.0.0.1:1337/"))("response_type", QVariant(QString, "code"))("scope", QVariant(QString, "read_prefs write_prefs write_api read_gpx write_gpx write_notes"))("state", QVariant(QString, "jRASHX8Y")))
xdg-open: file 'oauth2/authorize?client_id=wv3ui28EyHjH0c4C1Wuz6_I-o47ithPAOt7Qt1ov9Ps&code_challenge=29hhwtj0rbuj3dvgqVc2B_kvDCOHJl4uLbC5a8mtU3E&code_challenge_method=S256&redirect_uri=http://127.0.0.1:1337/&response_type=code&scope=read_prefs%20write_prefs%20write_api%20read_gpx%20write_gpx%20write_notes&state=jRASHX8Y' does not exist
   261.999 (WW) [ default ] OAuth2 reply handler timed out.
   261.999 (DD) [ default ] Failed!

My idea is still the same: external browser is not reliable...

[Krakonos]

I think I'm able to reproduce your issue. The URL entered has to be https://www.openstreetmap.org/, which is different to what it used to be. This needs to be adressed and perhaps rewritten automatically to facilitate migration.

Notice that http changes to https, and the /api/0.6 goes away. After these modifications, it seems to work with redirects.

I could embed a browser window instead, but that window is not necessarily trusted, and some features might be missing - as one example, users tend to have password managers that are not usable in this scenario.

Also, there might be a second flow necessary that doesn't use redirect, which could be problematic, but a code shown in browser instead. In this instance, we can also show the login URL user needs to navigate to. This can be then done on a different computer even, so I think that is a feasible backup solution.

Edit: The second issue seems you have empty URL field, can you check the contents there?

[nataraj-hates-MS-for-stealing-github]

Adding https://www.openstreetmap.org/ to the URL solves that problem on both laptops.... Both successfully opens login dialogs

I would suggest to change that dialog to the radio button choice: First is fixed " https://www.openstreetmap.org/" second is a field for custom URL. 98% of users will need the first choice.

But whenever I properly enter login, password and confirm that I trust merkaartor app, I am redirected to http://127.0.0.1:1337/?code=XXXXXXXXXXXXX page, that reports that connection can not be established.

telnet does not want to connect that address/port too

May be if we use built in browser, there will be no need in built in server to response on 127.0.0.1:1337, so we just grab that code from the redirect http request without processing it? :-)

[nataraj-hates-MS-for-stealing-github]

users tend to have password managers that are not usable in this scenario.

Hm... This I did not think about...

but that window is not necessarily trusted,

As for me, I would not trust any external browser app. But that is my preferences :-)

but a code shown in browser instead.

This sounds as a good idea... Much better then running local web server :-)

[Krakonos]

Yeah, the port is a problem due to how the Qt library handles the thing. I will implement the manual flow as a backup and release the port flow as is and try to figure out how to improve it. There are other issues with it - mainly the osm.org site returns errors in response body, but the flow does not give me API to retrieve that body in the error handler. I'm not sure if this can be fixed by inheritance, but TBH at that point it might be easier to roll a custom "library", since the OAuth2 flow in Qt requires so much boilerplate around it, it doesn't really give us much...

[nataraj-hates-MS-for-stealing-github]

If you get stuck with it, please let me know. I do not have much time, and I have not beet working with Qt for more than 15 years, so it would not be easy for me, but this issue should be fixed...

[Krakonos]

Well, I pushed a few UI changes with a slightly better error reporting and better messaging.
I got stuck on the manual flow though, I don't think I have found "The Qt way" to do it and it's painful. I'll continue tomorrow.

[nataraj-hates-MS-for-stealing-github]

Should I test anything?

[Krakonos]

You can have a look at the UI, but functionality should be mostly unchanged (except for some error handling, which is still sub-par due to deficiencies in the qt library). I'll find some time today to work on the manual (OOB) flow.

[Krakonos]

OOB flow pushed. You can give it a go. I have it set up so it opens the URL automatically, the dialog needs some work. I used off the shelf dialog, but the URL is too long and not clickable. I will probably roll a custom one, should not be too difficult.

I'm not too happy with the overall solution though - but I looked into the Qt tracker and there seem to be some changes in the works, so maybe i'll be able to improve it soon.

[Krakonos]

Also, changed logging to be hidden by default. Use the following environment variable to show them:

QT_LOGGING_RULES="merk.OsmServer.debug=true" 

[Thra11]

I've built 4bb1222 and tested both OAuth2 redirect and OAuth2 OOB. At first, both failed as they were attempting to load a url that starts with

https://www.openstreetmap.org/api/oauth2/authorize?client_id=...

which returns a 404. It looks like it had remembered the url from an older version with the /api/0.6/ specifier. Once I realised and selected the correct url from the dropdown, login worked and I was able to upload a change.

[Krakonos]

Thanks, I added a simple migration logic. For testing this, just run the new commit. To retest, edit ~/.config/Merkaartor/Merkaartor.conf section [OsmServers] lines version=1 (set to 0 to force the migration path again).

[kkofler]

Are you going to make a release any time soon or should distros just package a snapshot to make their package work again?

[Krakonos]

I'm looking to merge this branch & tag a release in the following days. Any further improvements and fixes will be released separately.

[Krakonos]

Fixed in PR #299, pushed 0.20.0 tag today. Please report any bugs in oauth2 implementation as separate tickets.