You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. Windows' default encoding when opening files with Python's open() method is apparently not UTF-8 - this causes reading data from quite a few files to fail. This is an easy fix though, we just need to specify UTF-8 as the encoding everywhere we have an open() call with no encoding specified.
2. Fugashi depends on a DLL, and the behavior for handling these changed in 3.8. This issue is documented in polm/fugashi#33 and can currently be worked around by installing with Anaconda and presumably other virtual environments, but it does break installs not using virtual environments. The short term solution is probably just to wrap the fugashi import in a try/catch block and disable it if we can't import it, but long term it would be best if we could just submit a PR to fugashi that fixed the issue. This will require doing some homework about the new Python DLL handling, though.
edit: First issue is solved.
The text was updated successfully, but these errors were encountered:
1. Windows' default encoding when opening files with Python'sopen()
method is apparently not UTF-8 - this causes reading data from quite a few files to fail. This is an easy fix though, we just need to specify UTF-8 as the encoding everywhere we have anopen()
call with no encoding specified.2. Fugashi depends on a DLL, and the behavior for handling these changed in 3.8. This issue is documented in polm/fugashi#33 and can currently be worked around by installing with Anaconda and presumably other virtual environments, but it does break installs not using virtual environments. The short term solution is probably just to wrap the fugashi import in a try/catch block and disable it if we can't import it, but long term it would be best if we could just submit a PR to fugashi that fixed the issue. This will require doing some homework about the new Python DLL handling, though.
edit: First issue is solved.
The text was updated successfully, but these errors were encountered: