Upgrading from 2.3.0 to 2.4.0 raises JSONDecodeError and psycopg.errors.UndefinedColumn

[iamjonmiller]

Describe the bug

I am attempting to update from 2.3.0 to 2.4.0. I have updated the package and begun to test my endpoints.

To start I am trying to exchange a username and password for a token. This is the most basic and important of my endpoints.

Upon submitting a username and pw in JSON format through the built in REST GUI I am greeted with:

JSONDecodeError at /authentication/token/ Expecting value: line 1 column 1 (char 0)

When I dig into this error it's much more complicated and odd.

psycopg.errors.UndefinedColumn: column oauth2_provider_application.hash_client_secret does not exist LINE 1: ...", "oauth2_provider_application"."client_secret", "oauth2_pr...

I interpret this to be saying:

The table oauth2_provider_application is missing the field hash_client_secret. I can confirm that this is true in looking at my database.

But I thought this was handled by the migration I ran to upgrade to 2.0.0? This has been running just fine on 2.0, 2.1, 2.2, and 2.3.

Reading through the changelog I don't see anything relating to this that has changed and I can't seem to find any upgrade guide, which I sort of remember, but might be getting confused with another package.

To Reproduce
My only settings are:
OAUTH2_PROVIDER = { 'PKCE_REQUIRED': False, 'ACCESS_TOKEN_EXPIRE_SECONDS': 2592000 # 30 days, }

I am aware PKCE defaults to True now and I am working on moving in that direction.

Version
2.4.0

• I have tested with the latest published release and it's still a problem.
• I have tested with the master branch and it's still a problem.

[n2ygk]

Did you do a `manage.py migrate`?

…

On Thu, Aug 29, 2024 at 5:20 PM Jonathan Miller ***@***.***> wrote: *Describe the bug* I am attempting to update from 2.3.0 to 2.4.0. I have updated the package and begun to test my endpoints. To start I am trying to exchange a username and password for a token. This is the most basic and important of my endpoints. Upon submitting a username and pw in JSON format through the built in REST GUI I am greeted with: JSONDecodeError at /authentication/token/ Expecting value: line 1 column 1 (char 0) When I dig into this error it's much more complicated and odd. psycopg.errors.UndefinedColumn: column oauth2_provider_application.hash_client_secret does not exist LINE 1: ...", "oauth2_provider_application"."client_secret", "oauth2_pr... I interpret this to be saying: The table oauth2_provider_application is missing the field hash_client_secret. I can confirm that this is true in looking at my database. But I thought this was handled by the migration I ran to upgrade to 2.0.0? This has been running just fine on 2.0, 2.1, 2.2, and 2.3. Reading through the changelog I don't see anything relating to this that has changed and I can't seem to find any upgrade guide, which I sort of remember, but might be getting confused with another package. *To Reproduce* My only settings are: OAUTH2_PROVIDER = { 'PKCE_REQUIRED': False, 'ACCESS_TOKEN_EXPIRE_SECONDS': 2592000 # 30 days, } I am aware PKCE defaults to True now and I am working on moving in that direction. *Version* 2.4.0 - I have tested with the latest published release and it's still a problem. - I have tested with the master branch and it's still a problem. — Reply to this email directly, view it on GitHub <#1471>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBHS522J6GLDTFURBUZ2I3ZT6GADAVCNFSM6AAAAABNLHWVOGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ4TKNJZG43TCOI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[iamjonmiller]

Did you do a manage.py migrate?

I did not, but I did manage.py makemigrations and it didn't find anything to migrate related to oauth (just two tiny field edits I made to other models).

[n2ygk]

showmigrations will show if any are pending

…

On Thu, Aug 29, 2024 at 5:39 PM Jonathan Miller ***@***.***> wrote: Did you do a manage.py migrate? I did not, but I did manage.py makemigrations and it didn't find anything to migrate related to oauth (just two tiny field edits I made to other models). — Reply to this email directly, view it on GitHub <#1471 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBHS5ZO6NLLNP3BYPKYBQ3ZT6IIFAVCNFSM6AAAAABNLHWVOGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJZGA3DSOBUGA> . You are receiving this because you commented.Message ID: ***@***.***>

[iamjonmiller]

showmigrations will show if any are pending

oauth2_provider
 [X] 0001_initial
 [X] 0002_auto_20190406_1805
 [X] 0003_auto_20201211_1314
 [X] 0004_auto_20200902_2022
 [X] 0005_auto_20211222_2352
 [X] 0006_alter_application_client_secret
 [X] 0007_application_post_logout_redirect_uris

All oauth2_provider migrations look to be done. That lines up with what I saw in the docs.

[n2ygk]

@iamjonmiller I'm not sure why but 0009_add_hash_client_secret migration appears to be missing from your list. This got added in #1311 which was part of the 2.4.0 release. I check the pypi package as well:

(venv) dot-tutorial$ pip uninstall django-oauth-toolkit
Found existing installation: django-oauth-toolkit 2.4.0
Uninstalling django-oauth-toolkit-2.4.0:
  Would remove:
    /Users/ac45/src/dot-tutorial/venv/lib/python3.12/site-packages/django_oauth_toolkit-2.4.0.dist-info/*
    /Users/ac45/src/dot-tutorial/venv/lib/python3.12/site-packages/oauth2_provider/*
Proceed (Y/n)? 
  Successfully uninstalled django-oauth-toolkit-2.4.0
(venv) dot-tutorial$ ls venv/lib/python3.12/site-packages/oauth2_provider/migrations/
ls: venv/lib/python3.12/site-packages/oauth2_provider/migrations/: No such file or directory
(venv) dot-tutorial$ pip install django-oauth-toolkit==2.4.0
Collecting django-oauth-toolkit==2.4.0
  Using cached django_oauth_toolkit-2.4.0-py3-none-any.whl.metadata (5.8 kB)
Requirement already satisfied: django!=4.0.0,>=3.2 in ./venv/lib/python3.12/site-packages (from django-oauth-toolkit==2.4.0) (5.0.6)
Requirement already satisfied: requests>=2.13.0 in ./venv/lib/python3.12/site-packages (from django-oauth-toolkit==2.4.0) (2.32.2)
Requirement already satisfied: oauthlib>=3.1.0 in ./venv/lib/python3.12/site-packages (from django-oauth-toolkit==2.4.0) (3.2.2)
Requirement already satisfied: jwcrypto>=0.8.0 in ./venv/lib/python3.12/site-packages (from django-oauth-toolkit==2.4.0) (1.5.6)
Requirement already satisfied: pytz>=2024.1 in ./venv/lib/python3.12/site-packages (from django-oauth-toolkit==2.4.0) (2024.1)
Requirement already satisfied: asgiref<4,>=3.7.0 in ./venv/lib/python3.12/site-packages (from django!=4.0.0,>=3.2->django-oauth-toolkit==2.4.0) (3.8.1)
Requirement already satisfied: sqlparse>=0.3.1 in ./venv/lib/python3.12/site-packages (from django!=4.0.0,>=3.2->django-oauth-toolkit==2.4.0) (0.5.0)
Requirement already satisfied: cryptography>=3.4 in ./venv/lib/python3.12/site-packages (from jwcrypto>=0.8.0->django-oauth-toolkit==2.4.0) (42.0.7)
Requirement already satisfied: typing-extensions>=4.5.0 in ./venv/lib/python3.12/site-packages (from jwcrypto>=0.8.0->django-oauth-toolkit==2.4.0) (4.11.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./venv/lib/python3.12/site-packages (from requests>=2.13.0->django-oauth-toolkit==2.4.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.12/site-packages (from requests>=2.13.0->django-oauth-toolkit==2.4.0) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./venv/lib/python3.12/site-packages (from requests>=2.13.0->django-oauth-toolkit==2.4.0) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.12/site-packages (from requests>=2.13.0->django-oauth-toolkit==2.4.0) (2024.2.2)
Requirement already satisfied: cffi>=1.12 in ./venv/lib/python3.12/site-packages (from cryptography>=3.4->jwcrypto>=0.8.0->django-oauth-toolkit==2.4.0) (1.16.0)
Requirement already satisfied: pycparser in ./venv/lib/python3.12/site-packages (from cffi>=1.12->cryptography>=3.4->jwcrypto>=0.8.0->django-oauth-toolkit==2.4.0) (2.22)
Using cached django_oauth_toolkit-2.4.0-py3-none-any.whl (73 kB)
Installing collected packages: django-oauth-toolkit
Successfully installed django-oauth-toolkit-2.4.0

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip
(venv) dot-tutorial$ ls venv/lib/python3.12/site-packages/oauth2_provider/migrations/
0001_initial.py                                0005_auto_20211222_2352.py                     0009_add_hash_client_secret.py
0002_auto_20190406_1805.py                     0006_alter_application_client_secret.py        0010_application_allowed_origins.py
0003_auto_20201211_1314.py                     0007_application_post_logout_redirect_uris.py  __init__.py
0004_auto_20200902_2022.py                     0008_alter_accesstoken_token.py                __pycache__/
(venv) dot-tutorial$ 

[iamjonmiller]

Yep, that is definitely missing. No idea why it's not showing up. I think we should probably do a better job of highlighting when a migration is required in a update, just as a warning.

[n2ygk]

Yep, that is definitely missing. No idea why it's not showing up. I think we should probably do a better job of highlighting when a migration is required in a update, just as a warning.

Did you find that migration in the installed package's oauth2_provider/migrations? If not I'm not sure why it's missing for you.

Agreed about breaking changes. Going forward anything that changes models will become a new major release to highlight the breaking change. Still, I don't understand why the migration is missing for you. It's definitely present in the wheel on pypi:

(venv) dot-tutorial$ unzip -l ~/Downloads/django_oauth_toolkit-2.4.0-py3-none-any.whl |grep 0009
      416  05-20-2024 13:53   oauth2_provider/migrations/0009_add_hash_client_secret.py

and when I removed my DB it shows up:

(venv) dot-tutorial$ rm db.sqlite3 
(venv) dot-tutorial$ ./manage.py showmigrations
admin
 [ ] 0001_initial
 [ ] 0002_logentry_remove_auto_add
 [ ] 0003_logentry_add_action_flag_choices
auth
 [ ] 0001_initial
 [ ] 0002_alter_permission_name_max_length
 [ ] 0003_alter_user_email_max_length
 [ ] 0004_alter_user_username_opts
 [ ] 0005_alter_user_last_login_null
 [ ] 0006_require_contenttypes_0002
 [ ] 0007_alter_validators_add_error_messages
 [ ] 0008_alter_user_username_max_length
 [ ] 0009_alter_user_last_name_max_length
 [ ] 0010_alter_group_name_max_length
 [ ] 0011_update_proxy_permissions
 [ ] 0012_alter_user_first_name_max_length
contenttypes
 [ ] 0001_initial
 [ ] 0002_remove_content_type_name
oauth2_provider
 [ ] 0001_initial
 [ ] 0002_auto_20190406_1805
 [ ] 0003_auto_20201211_1314
 [ ] 0004_auto_20200902_2022
 [ ] 0005_auto_20211222_2352
 [ ] 0006_alter_application_client_secret
 [ ] 0007_application_post_logout_redirect_uris
 [ ] 0008_alter_accesstoken_token
 [ ] 0009_add_hash_client_secret
 [ ] 0010_application_allowed_origins
sessions
 [ ] 0001_initial
(venv) dot-tutorial$ pip freeze | grep django-oauth
django-oauth-toolkit==2.4.0

Please investigate on your end and report what you find. Thanks.

[iamjonmiller]

I uninstalled and then reinstalled and now I am seeing incomplete migrations

oauth2_provider
 [X] 0001_initial
 [X] 0002_auto_20190406_1805
 [X] 0003_auto_20201211_1314
 [X] 0004_auto_20200902_2022
 [X] 0005_auto_20211222_2352
 [X] 0006_alter_application_client_secret
 [X] 0007_application_post_logout_redirect_uris
 [ ] 0008_alter_accesstoken_token
 [ ] 0009_add_hash_client_secret
 [ ] 0010_application_allowed_origins

[iamjonmiller]

Completed migrations and everything works fine. If you encounter this error try reinstalling the package!

Thanks @n2ygk