Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strange behavior with tokens of import nodes #27

Open
jrfaller opened this issue Feb 1, 2019 · 2 comments
Open

Strange behavior with tokens of import nodes #27

jrfaller opened this issue Feb 1, 2019 · 2 comments

Comments

@jrfaller
Copy link

jrfaller commented Feb 1, 2019

Hi!

I tried the following code to parse import statements :

source = 'import asttokens as at'
atok = asttokens.ASTTokens(source, parse=True)

I get the following AST:

Module
  Import
    alias

Which is OK. My purpose is to gather the position of the identifiers asttokens and at in the code. Therefore I was trying to get the tokens composing the alias node, which sounds logic because asttokens as at is and alias definition. However when I ask for the tokens composing the alias node, using list(atok.get_tokens(atok.tree.body[0].names[0])) I get the following result :

[Token(type=1, string=u'import', start=(1, 0), end=(1, 6), line=u'import asttokens as at', index=0, startpos=0, endpos=6)]

Which is was not what I was expecting, I was expecting to have the ['asttokens', 'as', 'at'] tokens.

Is there something I am missing?

Best regards.

@dsagal
Copy link
Member

dsagal commented Feb 1, 2019

You are right, the built-in ast module doesn't seem to annotate the "alias" node with any useful information, so it doesn't have any location info. One reasonable thing we could do is add a method to mark_tokens.py like:

def visit_alias(self, node, first_token, last_token):
   asname = self._code.find_token(first_token, token.NAME, node.asname)
   return (asname, asname)

This would map the "alias" node to the token with the alias name ("at" in your case). If you experiment with this and feel this is a useful change, a test and a pull request would be appreciated. Note also that there are somewhat harder situations, like from os import path as _path, wait as _wait.

@jrfaller
Copy link
Author

jrfaller commented Feb 5, 2019

Hi!

Thanks for the answer!

I will try to cope with this problem, if I manage to do something, I will send a PR!

Cheers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants