Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecation of HTTP Basic Auth and OAuth 1.0a #867

Closed
grischard opened this issue Apr 3, 2023 · 118 comments
Closed

Deprecation of HTTP Basic Auth and OAuth 1.0a #867

grischard opened this issue Apr 3, 2023 · 118 comments
Labels
service:api issues related to the API

Comments

@grischard
Copy link
Collaborator

grischard commented Apr 3, 2023

This is to track the discussion of the future deprecation of HTTP Basic Auth and OAuth 1.0a.

See pnorman's diary and zerebubuth/openstreetmap-cgimap#286

See the deprecation announcement for dates and details:

This change will have three key dates:

  • March 1st, 2024: New OAuth 1.0a application registrations are disabled. Existing applications will not be impacted. HTTP Basic Auth will not be impacted.
  • May 1st, 2024: Sysadmins will start brownouts to find applications that are still using OAuth 1.0a or HTTP Basic Auth
  • June 1st, 2024: OAuth 1.0a and HTTP Basic Auth will be shut down.
@pnorman pnorman added the service:api issues related to the API label Apr 4, 2023
@pnorman
Copy link
Collaborator

pnorman commented Aug 23, 2023

Over the last week 5% of API calls were HTTP Basic Auth, 23% OAuth 1.0a, and 73% OAuth 2.0. There's some daily jobs that are being run that call OAuth 1.0a starting at 01:15 UTC, 02:00 UTC, and one that calls HTTP Basic at 02:30 UTC, but even excluding those it's 18% OAuth 1.0a.

5 months ago, it was 70% OAuth 2.0, some traffic is moving to OAuth 2.0 but slowly.

@mmd-osm
Copy link

mmd-osm commented Aug 23, 2023

I think it would be helpful to have some stats highlighting the top OAuth 1.0a users. This way, we could raise follow up issue in the respective app repo. This would also help us to better track the overall progress.

The number of valid tokens per app id might be a good starting point:

SELECT app.id AS app_id,
       app.name AS app_name,
       count(tk.id) AS token_count
FROM client_applications AS app
LEFT JOIN oauth_tokens AS tk ON app.id = tk.client_application_id
AND tk.invalidated_at IS NULL
GROUP BY app.id,
         app.name
ORDER BY token_count DESC
LIMIT 100;

Possibly there are better ways to gather relevant insights on per-app OAuth 1.0a usage.

@tomhughes
Copy link
Member

I doubt the token count is very useful because some apps save a token and use it persistently and some generate a new one on every use.

More useful might be the number of users with at least one token for an app though there is still the question of how recent it is.

@tomhughes
Copy link
Member

Here's a complete list of UAs that used OAuth 1 yesterday, with a count of requests:

      1 FitoTrack
      2 CityZen
      2 Zend_Http_Client
      5 AtlasNavi
      5 viking
      6 Go-http-client
     14 Authlib
     17 maps.me
     20 Node
     22 PHPoAuthLib
     30 FireYak
     32 Locus
     43 OAuth
     50 OsmAnd
     57 MapRoulette
     73 Apache-HttpClient
    249 Python-urllib
    255 Python-httplib2
    301 GuzzleHttp
    338 PECL-OAuth
    370 osm-wikidata
    431 NewOSMContributorsInBelgium,
    450 Organic%20Maps
   1573 MAPS.ME
   2776 Vespucci
   3875 okhttp
   6254 Dalvik
  10015 StreetComplete_ee
  13827 python-requests
  17775 -
  45333 NewOSMContributorsInRussia,
  70304 Mozilla
  74752 Java
  87722 StreetComplete
 129793 JOSM

I've reduced it down to just the UA name to exclude version differences etc.

@tomhughes
Copy link
Member

The number of generic frameworks in that list limits it's usefulness (:angry:) so here's some results from @mmd-osm's query but limited to tokens issued this year to try and exclude historic things, and everything over 1000 tokens as a start:

 app_id |                 app_name                 | token_count 
--------+------------------------------------------+-------------
   3998 | Geofabrik Downloads                      |      198430
    750 | uMap                                     |       88382
   1896 | Organic Maps                             |       43020
   3277 | HOT Export Tool                          |       22932
   2302 | New MapRoulette                          |       20081
  10793 | StreetComplete                           |       18758
   2985 | How did you contribute to OpenStreetMap? |       17070
   9790 | MR MKS                                   |       14896
   3817 | Osmcha                                   |       11493
   5742 | umap-de                                  |       10960
   8524 | Organic Maps                             |       10710
    946 | JOSM - Java OpenStreetMap Editor         |        9061
   5066 | OsmAnd                                   |        9050
   7526 | OSM-Boundaries                           |        7487
    821 | OsmHydrant                               |        5224
   1875 | uMap Framasoft                           |        5214
   8520 | OrganicMaps F-Droid                      |        4526
   9922 | tasking-manager-hyd.geo.apple.com        |        4478
   3175 | overpass turbo                           |        3583
   3551 | OSM Streak                               |        3489
   4188 | TeachOSM Tasking Manager                 |        3361
   5265 | OpenPlaceReviews                         |        3226
   2100 | KartaView                                |        3098
   7404 | MapComplete                              |        2888
  11156 | DEMR3                                    |        2773
  10210 | Vespucci 2022                            |        2621
  12240 | OSM Imagery                              |        2616
   1198 | umap.osm.ch                              |        1941
   1161 | Osmose                                   |        1586
   1917 | GNOME Maps                               |        1411
    952 | Vespucci                                 |        1376
   8490 | mr_mumbai                                |        1249
   3215 | Freemap Slovakia                         |        1200
   6020 | OSM Teams                                |        1070

I think there's plenty there we can target as a starting point...

@tsmock
Copy link

tsmock commented Aug 23, 2023

129793 JOSM

Do we have a better idea of when OAuth 1.0a/Basic Auth will be officially deprecated? Last I heard, it was still under consideration.

I've had a ticket open in JOSM Trac for hiding Basic Auth/OAuth 1.0a (see #22810) which I had scheduled to revisit in January. I don't (yet) want to remove OAuth 1.0/Basic Auth, since it still works, but I don't mind hiding it from new users if necessary.

@tomhughes
Copy link
Member

I'm inclined to say they're deprecated right now - that's literally the point of this ticket that we're trying to get people to move to OAuth 2 instead.

The question is when we stop supporting them.

I wasn't actually bothered by the JOSM entries because I assumed it was just people that had configured it some time ago and that they could migrate when necessary.

@pnorman
Copy link
Collaborator

pnorman commented Aug 23, 2023

Yes, this ticket starts the deprecation process. We have no date for when we will shut down OAuth 1.0a or HTTP Basic Auth, but earlier stated that we did not plan to shut them off this year. I would expect sometime mid-next year but that depends entirely on how efforts to move people to OAuth 2.0 go.

@pnorman
Copy link
Collaborator

pnorman commented Aug 23, 2023

MapRoulette has moved, see maproulette/maproulette3#1876

I reached out to the TeachOSM and Apple Tasking Managers.

@mmd-osm
Copy link

mmd-osm commented Aug 24, 2023

@mmd-osm
Copy link

mmd-osm commented Aug 24, 2023

I'm not aware of a repo for How did you contribute to OpenStreetMap? Let's try to ping @pa5cal instead.

@pa5cal
Copy link

pa5cal commented Aug 26, 2023

I'm not aware of a repo for How did you contribute to OpenStreetMap? Let's try to ping @pa5cal instead.

Thanks, @mmd-osm !
I will try to change my HDYC webpage to OAuth2 in the next few months.
The first tests today looked quite promising.

@pietervdvn
Copy link

For all (javascript) devs reading this https://www.npmjs.com/package/osm-auth has been updated to use OAuth2.0

@mmd-osm
Copy link

mmd-osm commented Feb 16, 2024

@simonpoole :

And I would point out that there are real show stoppers (from an OSM ecosystem pov) that we haven't even touched on yet, for example UMAP. Writing and testing the code is one thing by OSM-FR, but then people will have to upgrade their instances,

uMap 2.0.0 is out now, supporting OSM OAuth 2.0. People can start planning their uMap instance upgrade right away.

-> This has already started for umap.openstreetmap.de: https://en.osm.town/@gislars/111946204444707665

@docentYT
Copy link

docentYT commented Feb 17, 2024

OAuth 2.0 on OpenStreetMap is so far ready practically only for web tools or with an interface that the user will use.

With software using the osm api, but running on servers, it is already starting to be a problem. The person responsible for running the server needs to do a few things (see: metaodi/osmapi#144). Why can't you do something analogous to, for example, the Discord Developer Portal, where you create a new application and generate a token for it, with which you can authorise yourself? This is a much simpler and more convenient way.

I have not found any official information anywhere as to whether the token will not expire.

@mmd-osm
Copy link

mmd-osm commented Feb 18, 2024

@docentYT : generating such a token on the backend is not super complicated: https://github.com/openstreetmap/openstreetmap-website/blob/master/app/models/user.rb#L379-L387 shows how this can be accomplished.

However, Doorkeeper doesn't this expose this functionality on a UI. To make this happen, someone needs to write some code and come up with a good UI. Of course, website maintainers need to agree to the idea. So that's really a topic for https://github.com/openstreetmap/openstreetmap-website.

We also discussed some alternatives. Maybe you can spend 10-30 mins to read through this issue and the linked discussion(s) on https://c.osm.org and osm.org/diary.

@tomhughes
Copy link
Member

Yes we certainly could do it and I had considered long before anybody bought it up here - it's not hard and we essentially already do something similar internally for the iD and web site tokens.

The use case is small though - essentially single user command line tools - and my concern is that if we enable it then it will get used more widely and that programs which should do things properly will instead just ask people to generate a token on the web site and paste it in.

It's not even especially hard to have a command line tool generate a request token and output the authentication URL and then wait for the user to paste the returned token in before continuing as has been demonstrated by the fact that several people have written simple examples of doing that.

@AntonKhorev
Copy link

and then wait for the user to paste the returned token

Technically it's not a token, it's a code that has to be exchanged for a token. Some people don't want to do this exchange.

@DavidKarlas
Copy link

FYI, I migrated https://osm.karlas.si this morning

@mnalis
Copy link

mnalis commented Mar 6, 2024

https://pic4review.pavie.info should be supporting Oauth2 now

@docentYT
Copy link

docentYT commented Mar 10, 2024

I have created a web-based tool to obtain a token.
Diary
Tool

Also, I would like to announce that I have started adding support for oAuth2 in osm_easy_api.

@docentYT
Copy link

osm_easy_api as of now supports only OAuth 2.0.

@mmd-osm
Copy link

mmd-osm commented Apr 3, 2024

(the following info is relevant for the dev instance only, that's master.apis.dev.openstreetmap.org)

I plan to merge zerebubuth/openstreetmap-cgimap#354 once Organic Maps has merged their OAuth2 implementation in organicmaps/organicmaps#7333 ... this is likely to happen some time in April. has happened on April 4th.

As a result of this merge, the dev instance will no longer be accepting diff uploads using OAuth 1.0a (which is handled by CGImap). Other features, like reading user preferences on the dev instance would continue to work a bit longer, assuming they're being handled by Rails.

Important point: this has no impact on the schedule that @pnorman communicated previously for production.

Let me know, if you see any issues with this approach.

Edit:

@mnalis
Copy link

mnalis commented May 29, 2024

@tomhughes
Copy link
Member

As of openstreetmap/chef@8dd0893 these are now permanently disabled.

@kkofler
Copy link

kkofler commented Jul 22, 2024

The discontinuing of HTTP Basic Auth is a major disservice to both client developers and end users. It makes client development a lot more complicated. It also complicates client usage, because you are now forced to use a browser to log in. Also hits all the simple CLI clients, revert scripts, etc.

In the short term, the main effect is to break clients such as Merkaartor that have not yet been updated (see openstreetmap/merkaartor#286), which I would already consider enough of a reason to treat this change as a no go. In the long term, the workflow is going to be a pain to use, because, instead of inputting a password once in a configuration dialog (or on the CLI for CLI clients), one has to click on a link, open a browser (unless the client embeds a browser, but then you cannot make use of your browser's password saving feature), then copy&paste a token from that browser into the application. I do not see how this is progress in any way.

I can understand why the obsolete OAuth 1 protocols have been dropped, but not HTTP Basic Auth. There is nothing wrong with that protocol as long as TLS encryption is used (keeping the password safe). (That said, at least according to the URL in the configuration dialog, Merkaartor appears to be using unencrypted HTTP, which I would consider a bigger issue than the authentication method.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
service:api issues related to the API
Projects
None yet
Development

No branches or pull requests