Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] SyntaxError: Invalid magic, got b')DV\x07' #462

Open
WinterPhoenix opened this issue Jun 27, 2024 · 5 comments · May be fixed by #464
Open

[BUG] SyntaxError: Invalid magic, got b')DV\x07' #462

WinterPhoenix opened this issue Jun 27, 2024 · 5 comments · May be fixed by #464
Labels
bug needs-review

Comments

@WinterPhoenix
Copy link

WinterPhoenix commented Jun 27, 2024 •
edited
Loading

Description
Steam Client Beta seems to be rolling out a new format again.

Steps to Reproduce the behavior

>>> from steam.utils.appcache import parse_appinfo
>>> header, apps = parse_appinfo(open(r"C:\Users\Winter\Desktop\WIP\appinfo.vdf", "rb"))

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Python311_64\Lib\site-packages\steam\utils\appcache.py", line 73, in parse_appinfo
    raise SyntaxError("Invalid magic, got %s" % repr(magic))
SyntaxError: Invalid magic, got b')DV\x07'

Example appinfo.vdf, like last time:
appinfo.zip

Versions Report

steam: 1.4.4

Dependencies:
                 vdf: 3.4
            protobuf: 4.21.11
            requests: 2.31.0
          cachetools: 5.2.0
              gevent: Not Installed
 gevent-eventemitter: Not Installed
       pycryptodomex: 3.16.0
              enum34: Not Installed
       win-inet-pton: Not Installed

Python runtime:
          executable: D:\Python311_64\python.exe
             version: 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)]
            platform: win32

System info:
              system: Windows
             machine: AMD64
             release: 10
             version: 10.0.19045
@sonic2kk
Copy link

Looks like the magic changed again for appinfo.vdf, happened a little while ago as well: #418.

This was referenced Jun 28, 2024
Closed
Closed
@Matoking
Copy link

Had a look at the new appinfo.vdf format. Not a case of just the magic number being changed:

#463 (comment)

Matoking added a commit to Matoking/vdf that referenced this issue Jun 28, 2024
@Matoking
Add decode support for new binary VDF
636b477 
The new Steam beta introduced a new `appinfo.vdf` version. This
appinfo.vdf V29 introduces a new binary VDF format which does not
include field keys in binary VDF segments as-is. Instead, each key is
represented by a 32-bit integer which needs to be mapped to an actual
string using a table are stored at the end of the `appinfo.vdf` file.

This also means the binary VDF segments are no longer self-contained.

Also see SteamDatabase/SteamAppinfo#56b1fec7f5ce6be961c3e44cf9baf117e363ad91

Refs ValvePython/steam#462
Matoking added a commit to Matoking/vdf that referenced this issue Jun 28, 2024
@Matoking
Add decode support for new binary VDF
38799ef 
The new Steam beta introduced a new `appinfo.vdf` version. This
appinfo.vdf V29 introduces a new binary VDF format which does not
include field keys in binary VDF segments as-is. Instead, each key is
represented by a 32-bit integer which needs to be mapped to an actual
string using a table are stored at the end of the `appinfo.vdf` file.

This also means the binary VDF segments in this case are no longer
self-contained.

Also see SteamDatabase/SteamAppinfo#56b1fec7f5ce6be961c3e44cf9baf117e363ad91

Refs ValvePython/steam#462
@WinterPhoenix WinterPhoenix mentioned this issue Jun 29, 2024
Closed
Matoking added a commit to Matoking/vdf that referenced this issue Jun 30, 2024
@Matoking
Add deserialization support for new binary VDF
c5f23fc 
The new Steam beta introduced a new `appinfo.vdf` version. This
appinfo.vdf V29 introduces a new binary VDF format which does not
include field keys in binary VDF segments as-is. Instead, each key is
represented by a 32-bit integer which needs to be mapped to an actual
string using a table stored at the end of the `appinfo.vdf` file.

This also means the binary VDF segments in this case are no longer
self-contained. The developer can parse and provide the table
themselves or instead use the `ValvePython/steam` library which
contains a function to parse `appinfo.vdf` files.

Also see SteamDatabase/SteamAppInfo#56b1fec7f5ce6be961c3e44cf9baf117e363ad91

Refs ValvePython/steam#462
Matoking added a commit to Matoking/vdf that referenced this issue Jun 30, 2024
@Matoking
Add deserialization support for new binary VDF
b08266b 
The new Steam beta introduced a new `appinfo.vdf` version. This
appinfo.vdf V29 introduces a new binary VDF format which does not
include field keys in binary VDF segments as-is. Instead, each key is
represented by a 32-bit integer which needs to be mapped to an actual
string using a table stored at the end of the `appinfo.vdf` file.

This also means the binary VDF segments in this case are no longer
self-contained. The developer can parse and provide the table
themselves or instead use the `ValvePython/steam` library which
contains a function to parse `appinfo.vdf` files.

Also see SteamDatabase/SteamAppInfo@56b1fec

Refs ValvePython/steam#462
Matoking pushed a commit to Matoking/steam that referenced this issue Jun 30, 2024
@sonic2kk @Matoking
Add deserialization support for appinfo.vdf V29
47468b5 
appinfo.vdf V29 was introduced in Steam beta. This new version
introduces a space-saving optimization: instead of encoding each key
name in the binary VDF segment directly, an int64 identifier is instead
used for each key, with a table at the end of the 'appinfo.vdf' file
providing the mapping to actual key names.

Also see
SteamDatabase/SteamAppInfo@56b1fec

Fixes ValvePython#462

Co-authored-by: Eamonn Rea <[email protected]>
Matoking added a commit to Matoking/vdf that referenced this issue Jun 30, 2024
@Matoking
Add deserialize support for binary VDF w/ table
981cad2 
The new Steam beta introduced a new `appinfo.vdf` version. This
appinfo.vdf V29 introduces a new binary VDF format which does not
include field keys in binary VDF segments as-is. Instead, each key is
represented by a 32-bit integer which needs to be mapped to an actual
string using a table stored at the end of the `appinfo.vdf` file.

This also means the binary VDF segments in this case are no longer
self-contained. The developer can parse and provide the table
themselves or instead use the `ValvePython/steam` library which
contains a function to parse `appinfo.vdf` files.

Also see SteamDatabase/SteamAppInfo@56b1fec

Refs ValvePython/steam#462
This was referenced Jun 30, 2024
Open
Open
Matoking pushed a commit to Matoking/steam that referenced this issue Jun 30, 2024
@sonic2kk @Matoking
Add deserialization support for appinfo.vdf V29
6a32f3d 
appinfo.vdf V29 was introduced in Steam beta. This new version
introduces a space-saving optimization: instead of encoding each key
name in the binary VDF segment directly, an int64 identifier is instead
used for each key, with a table at the end of the 'appinfo.vdf' file
providing the mapping to actual key names.

Also see
SteamDatabase/SteamAppInfo@56b1fec

Fixes ValvePython#462

Co-authored-by: Eamonn Rea <[email protected]>
@WinterPhoenix
Copy link
Author

@rossengeorgiev Please review the relevant PRs. This should really be fixed sooner rather than later.

@WinterPhoenix
Copy link
Author

WinterPhoenix commented Jul 25, 2024 •
edited
Loading

I made a fork that incorporates the PR that fixes this, as well as several other pending PRs in this repo. I don't really particularly want to be the new maintainer for this, but if I'm still using it I may as well. PRs on the fork welcome.

https://github.com/solsticegamestudios/steam

@Matoking
Copy link

One option would be to vendor the relevant code and/or libraries, though I don't know if that is possible for projects using GPL and other similar licenses like Protontricks. Debian also doesn't like vendored code to my knowledge.

I've instead opted to push a version to Flathub beta branch with the fix. For pip users I've instructed to install the libraries from GitHub directly. Other than that, the current options are limited as vdf and steam libraries are packaged by upstream distros.

@rossengeorgiev has had some sporadic activity on GitHub, but these two ValvePython projects haven't had any activity in a while. I have to emphasize that they're not obligated to work on these libraries - open source development is often a thankless job and maintainers are prone to burnout, after all - though it would be appreciated if they could state whether these projects are in maintenance mode or abandoned.

Matoking pushed a commit to Matoking/steam that referenced this issue Aug 3, 2024
@sonic2kk @Matoking
Add deserialization support for appinfo.vdf V29
052a6ae 
appinfo.vdf V29 was introduced in Steam beta. This new version
introduces a space-saving optimization: instead of encoding each key
name in the binary VDF segment directly, an int64 identifier is instead
used for each key, with a table at the end of the 'appinfo.vdf' file
providing the mapping to actual key names.

Also see
SteamDatabase/SteamAppInfo@56b1fec

Fixes ValvePython#462

Co-authored-by: Eamonn Rea <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug needs-review
Projects
None yet
Milestone
No milestone
3 participants
@Matoking @WinterPhoenix @sonic2kk