Ambiguity between complex number and string value in JSON-represented AST

[yusungsim]

Context

I'm using ast2json library to extract Python AST as JSON file, and later parse it in from other program.

Problem discription

What I found is that when a complex number in python is printed out as JSON string, which makes it ambiguous to distingush from string values.

For example, consider following Python code that uses ast2json.

import ast
import json
from ast2json import ast2json

code1 = '0.2j' # complex number in python
code2 = '"0.2j"' # string `0.2j`

ast1 = ast.parse(code1)
ast2 = ast.parse(code2)

print('---> parse result of {}', code1)
#print(ast.dump(ast1))
print('type of const:', type(ast1.body[0].value.value)) # complex

print('---> parse result of {}', code2)
#print(ast.dump(ast2))
print('type of const:', type(ast2.body[0].value.value)) # str

json1 = ast2json(ast1)
json2 = ast2json(ast2)

print('---> json result of {}', code1)
#print(json.dumps(json1))
print(json1["body"][0]["value"])

print('---> json result of {}', code2)
#print(json.dumps(json2))
print(json2["body"][0]["value"])

The result

$ python3 ast2json-complex-error.py
---> parse result of {} 0.2j
type of const: <class 'complex'>
---> parse result of {} "0.2j"
type of const: <class 'str'>
---> json result of {} 0.2j
{'_type': 'Constant', 'col_offset': 0, 'end_col_offset': 4, 'end_lineno': 1, 'kind': None, 'lineno': 1, 'n': '0.2j', 's': '0.2j', 'value': '0.2j'}
---> json result of {} "0.2j"
{'_type': 'Constant', 'col_offset': 0, 'end_col_offset': 6, 'end_lineno': 1, 'kind': None, 'lineno': 1, 'n': '0.2j', 's': '0.2j', 'value': '0.2j'}

code1 is a expression for complex number constant in Python, whereas code2 is a expression for string constant in Python. Using Python standard library's ast module, it can be checked out that two AST are different in type. code1 has complex number typed value 0.2j in AST, but code2 has string typed value '0.2j' in AST.

However, when using ast2jon to create JSON object, the resulting JSON has no method to express different types of two values. In the result, both constant AST has string value '0.2j'.

This makes impossible to distinguish two AST when dumped into JSON and used in external context. For example, when parsing constant expressions, parser has no method to distinguish two JSON objects and decide whether to return complex number or string.

Suggested solution

While it is not documented in Python specification or ast module document, one can utilize kind field of constant node to indicate type of Python value.

For example,

{'_type': 'Constant', 'col_offset': 0, 'end_col_offset': 4, 'end_lineno': 1, 'kind': 'complex', 'lineno': 1, 'n': '0.2j', 's': '0.2j', 'value': '0.2j'} would express the constant node of AST of the code 0.2j,

while {'_type': 'Constant', 'col_offset': 0, 'end_col_offset': 4, 'end_lineno': 1, 'kind': 'str', 'lineno': 1, 'n': '0.2j', 's': '0.2j', 'value': '0.2j'} would express the '0.2j' one.