-
-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[brew completion] -<something>
breaks on some commands
#515
Comments
Thanks for the report. This type of error message is specific to macOS awk, but I don't have macOS so cannot test it. MacOS awk produces error messages when it sees data that is not compatible with the current encoding. As far as I search GitHub for the data in the error message, it seems to come from the man page of
$ man brew | grep 'API for new formulae' | cat -v
$ man brew | grep 'API for new formulae' | /usr/bin/awk '{print NF;}' |
Thanks for the prompt follow-up @akinomyoga much appreciated. Also really appreciate the great work on this awesome project. ✨ As for your question, indeed I see some weird encoding in the first command output: $ man brew | grep 'API for new formulae' | cat -v
Check HomebrewM-^@M-^Ys API for new formulae or cask data every
$ man brew | grep 'API for new formulae' | /usr/bin/awk '{print NF;}'
10 |
Thanks for the result.
The weird encoding you see is actually what is intended by
I actually expect
$ echo $'\u2019' | cat -v
I thought macOS
$ ble/widget/display-shell-version
$ locale
$ man brew | grep 'API for new formulae' | LC_ALL=C /usr/bin/awk '{print NF;}' |
--
breaks on some commands--
breaks on some commands
--
breaks on some commands--
breaks on some commands
Thank you Q2$ echo $'\u2019' | cat -v
M-^@M-^Y Q3$ ble/widget/display-shell-version
-bash: `ble/bin/diff': not a valid identifier
-bash: ble/bin/diff: No such file or directory
GNU bash, version 5.2.37(1)-release (aarch64-apple-darwin23.4.0)
ble.sh, version 0.4.0-nightly+31f264a (noarch) [git 2.46.2, GNU Make 4.3, GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)]
locale: LANG=en_US.UTF-8
terminal: TERM=tmux-256color wcwidth=15.1-west/16.0-2+ri, tmux:0 (84;0;0)
options: $ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL= $ man brew | grep 'API for new formulae' | LC_ALL=C /usr/bin/awk '{print NF;}'
this one does not seem to produce any output, its not exiting either |
Thank you. Hmm, this is interesting. The above result seems to contain an empty line. Is that a part of the output of the command? (I.e., does the command output an empty line? or does it output nothing?) Anyway, as far as the output of
$ echo $'\u2019' | od -t x1
$ man brew | grep 'API for new formulae' | od -t x1
$ echo $'\u2019' | /usr/bin/awk '{print NF;}'
$ echo $'\u2019' | LC_ALL=C /usr/bin/awk '{print NF;}'
This suggests another problem with the test for the existence of the command
$ type -a diff
$ (set -x; ble/bin#freeze-utility-path diff; echo "exit status: $?"; type -a ble/bin/diff) |
Ah, actually I recently set $ MANPAGER=less
$ man brew | grep 'API for new formulae' | LC_ALL=C /usr/bin/awk '{print NF;}'
10 Q4$ echo $'\u2019' | od -t x1
0000000 e2 80 99 0a
0000004
$ MANPAGER=less
$ man brew | grep 'API for new formulae' | od -t x1
0000000 20 20 20 20 20 20 20 20 20 20 20 20 20 20 43 68
0000020 65 63 6b 20 48 6f 6d 65 62 72 65 77 e2 80 99 73
0000040 20 41 50 49 20 66 6f 72 20 6e 65 77 20 66 6f 72
0000060 6d 75 6c 61 65 20 6f 72 20 63 61 73 6b 20 64 61
0000100 74 61 20 65 76 65 72 79 0a
0000111
$ echo $'\u2019' | /usr/bin/awk '{print NF;}'
1
$ echo $'\u2019' | LC_ALL=C /usr/bin/awk '{print NF;}'
1 Q5$ type -a diff
diff is /usr/bin/diff
diff is /opt/homebrew/bin/diff
$ (set -x; ble/bin#freeze-utility-path diff; echo "exit status: $?"; type -a ble/bin/diff)
>>>> ble/bin#freeze-utility-path diff
>>>> local cmd path 'q='\''' 'Q='\''\'\'''\''' fail= flags=
>>>> for cmd in "$@"
>>>> [[ diff == -n ]]
>>>> [[ '' == *n* ]]
>>>> ble/bin#has ble/bin/.frozen:diff
>>>> builtin type -t -- ble/bin/.frozen:diff
>>>> ble/bin#get-path diff
>>>> local cmd=diff
>>>> ble/util/assign path 'builtin type -P -- "$cmd" 2>/dev/null'
>>>> local _ble_local_tmpfile
>>>> ble/util/assign/mktmp
>>>> _ble_local_tmpfile=/tmp/blesh/502/82534.util.assign.tmp.0
>>>> (( BASH_SUBSHELL ))
>>>> _ble_local_tmpfile=/tmp/blesh/502/82534.util.assign.tmp.0.83911
>>>> builtin eval -- 'builtin type -P -- "$cmd" 2>/dev/null'
>>>>> builtin type -P -- diff
>>>> local _ble_local_ret=0 _ble_local_arr=
>>>> mapfile -t _ble_local_arr
>>>> ble/util/assign/rmtmp
>>>> (( _ble_util_assign_level-- ))
>>>> (( BASH_SUBSHELL ))
>>>> printf 'caller %s\n' ble/util/assign/rmtmp ble/util/assign ble/bin#get-path ble/bin#freeze-utility-path
>>>> IFS='
'
>>>> builtin eval 'path="${_ble_local_arr[*]}"'
>>>>> path=/usr/bin/diff
>>>> return 0
>>>> [[ -n /usr/bin/diff ]]
>>>> [[ /usr/bin/diff == ./* ]]
>>>> [[ /usr/bin/diff == ../* ]]
>>>> builtin eval 'function ble/bin/diff { '\''/usr/bin/diff'\'' "$@"; }'
-bash: `ble/bin/diff': not a valid identifier
>>>> (( !fail ))
>>>> echo 'exit status: 0'
exit status: 0
>>>> type -a ble/bin/diff
-bash: type: ble/bin/diff: not found last one did exit with exit code 1 If it is of any value, here is another example of the behavior: $ lsof -i --/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
-- INSERT --
input record number 4, file
source line number 29
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 29
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 29
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 29
-- Okay, seems like it's not necessarly breaking on In this video you can see my typing sample.mov$ type -a lsof
lsof is /usr/sbin/lsof So it might be something that is not limited to |
--
breaks on some commands-<something>
breaks on some commands
Thank you for the answers and additional information.
OK, the new result seems consistent with the previous one, yet the original error message is not reproduced. Maybe there are other conditions for the error message to be output.
I suspected the possibility of an unexpected behavior of
Seems fine. The only non-7-bit characters are "80 99 73" (which is the UTF-8 representation of
These are also consistent with the previous results.
I suspected that the man page of
Thank you for the additional explanation. I'm sorry that I haven't explained the details to you, but I knew that the reported error is related to the extraction of the option description from the man page in the I'd like to find out anything common in the affected parts of the man pages in
$ MANPAGER=less
$ man lsof | grep ' AIX:' | cat -v
Hmm, OK. Do you set
$ declare -p POSIXLY_CORRECT |
I think I need to ask you to run a patched version of |
Appreciate the explantations very much. 🙏
Sounds good, thank you. Q6$ MANPAGER=less
$ man lsof | grep ' AIX:' | cat -v
# nothing
$ man lsof | grep ' AIX:' | od -t x1
# nothing
$ man lsof | grep 'AIX:' | cat -v
AIX:
$ man lsof | grep 'AIX:' | od -t x1
0000000 20 20 20 20 20 20 20 c2 a0 c2 a0 c2 a0 c2 a0 41
0000020 49 58 3a 0a
0000024 Q5aYes, $ declare -p POSIXLY_CORRECT
declare -- POSIXLY_CORRECT="y" |
|
OK, thank you! I'm now preparing the next version to check another place. |
Cool, much appreciated! |
|
Hm I can't get it to produce the debug file for some reason. Screen.Recording.2024-10-19.at.8.02.14.PM.mov |
Did you instead see the message |
Just did a search on the output against this, but could not find any match. Including the full output here for reference: $ lsof -aa/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
-- INSERT --
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
-aaa/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
-aaaa/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
-aaaa Also just checked the patch again to confirm I'm running the latest patch: $ grep "check2" $_ble_base/lib/core-complete.sh
function __blesh_debug_mandb_check2__ {
echo 'debug (check2): Error not found.' >/dev/tty
__blesh_debug_mandb_check2__ "${subcaches[@]}" |
Thank you.
|
Sounds good, ty. $ lsof -able/complete/mandb/generate-cache: step1 (caller ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook)
ble/complete/mandb/generate-cache: step3a (caller ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook)
awk: ble/function#advice/before:ble/bin/awk ble/function#try ble/function#advice/.proc ble/bin/awk ble/complete/mandb/.generate-cache-from-man ble/complete/mandb/generate-cache ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook / 1 5 12 1 16 52 3 9 19 35 13 21 18 15 16 20 1 38 3 10 70 1
awk: ble/function#advice/before:ble/bin/awk ble/function#try ble/function#advice/.proc ble/bin/awk ble/complete/mandb/.generate-cache-from-man ble/complete/mandb/generate-cache ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook / 1 5 12 1 341 52 3 9 19 35 13 21 18 15 16 20 1 38 3 10 70 1
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
ble/complete/mandb/generate-cache: step3b (caller ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook)
ble/complete/mandb/generate-cache: step4 (caller ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook)
-a |
Thank you for your quick responses. So, the error seems to be produced inside
|
Nice, here you go: |
$ /usr/bin/awk -W version || /usr/bin/awk --version |
$ /usr/bin/awk -W version
/usr/bin/awk: unknown option -W ignored
$ /usr/bin/awk --version
awk version 20200816 |
$ path=/usr/bin/awk
$ ble/util/assign version '"$path" -W version || "$path" --version' 2>/dev/null </dev/null
$ declare -p version |
$ path=/usr/bin/awk
$ ble/util/assign version '"$path" -W version || "$path" --version' 2>/dev/null </dev/null
$ declare -p version
declare -- version="" |
Strange...
$ { /usr/bin/awk -W version || /usr/bin/awk --version; } 2>/dev/null </dev/null
$ /usr/bin/awk --version 2>/dev/null
$ /usr/bin/awk --version >/dev/null |
$ { /usr/bin/awk -W version || /usr/bin/awk --version; } 2>/dev/null </dev/null
# nothing
$ /usr/bin/awk --version 2>/dev/null
awk version 20200816
$ /usr/bin/awk --version >/dev/null
# nothing |
Thank you for the results. Ah, OK.
$ /usr/bin/awk -W version < /dev/null; echo "$?"
$ /usr/bin/awk --version 2>/dev/null </dev/null
|
A8c
Exactly, kind of weird, its printing this error message, that seems to be more of a warning, as its not exiting with a non-0 exit code. $ /usr/bin/awk -W version < /dev/null; echo "$?"
/usr/bin/awk: unknown option -W ignored
0
$ /usr/bin/awk --version 2>/dev/null </dev/null
awk version 20200816 A7d |
Thanks! I'll add a fix for this. However, this is probably an issue independent of the original error message (though I initially suspected this). Thank you for this new version. Hmm, the error is not recorded somehow. I have a question.
|
Sounds good. Yea, let me share the full output: $ lsof -aable/complete/mandb/generate-cache: step3a (caller ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook)
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
ble/complete/mandb/generate-cache: step3b (caller ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook) |
$ echo '\ \ \ \ AIX:' | ble/complete/mandb/.preconv
$ echo '\ \ \ \ AIX:' | ble/complete/mandb/.preconv | groff -T utf8 -m man | od -t x1
$ echo '\ \ \ \ AIX:' | groff -T utf8 -m man | od -t x1
$ echo '\ \ \ \ AIX:' | groff -T utf8 -m man | /usr/bin/awk '{print NF;}'
$ printf '\x20\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0' | /usr/bin/awk '{print NF;}' edit: Also, this: $ type ble/complete/mandb/convert-mandoc
$ echo '\ \ \ \ AIX:' | ble/complete/mandb/convert-mandoc | od -t x1
$ printf '\x20\xc2\xa0\xc2\xa0' | /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}' | od -t x1 |
|
$ echo '\ \ \ \ AIX:' | ble/complete/mandb/.preconv
# Exit Code: 0
# stdout:
.lf 1 - \ \ \ \ AIX:
# stderr:
$ echo '\ \ \ \ AIX:' | ble/complete/mandb/.preconv | groff -T utf8 -m man | od -t x1
# Exit Code: 0
# stdout:
0000000 20 20 20 20 41 49 58 3a 0a 0a 0a 0a 0a 0a 0a 0a 0000020 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 1u�[?25h�[?2026l 1u�[?25h�[?2026l�[?2026h�[?25l�[?2004h�[ 36836.debug.mandb-lsof.txt Applications Brewfile.lock.json Desktop Documents Downloads Dropbox Library Local Sites Movies Music Pictures PlayOnMac's virtual drives Postman Public a a.log a.py a.sh a.ts abc ai_overlay_tmp arm-target athame bentoml bin blesh-debug-mandb-brew.tar.gz blesh-debug-mandb-lsof.tar.gz captain-hardcore dead.letter deno.lock env.sh fonts go iCloud Drive (Archive) init.lua intelephense mbox miniforge3 nltk_data node_modules opt out.txt package-lock.json pnpm pnpm-lock.yaml test_out.afm test_out.ttf tmux-client-54992.log tmux-client-90135.log 0000100 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0000112
# stderr:
$ echo '\ \ \ \ AIX:' | groff -T utf8 -m man | od -t x1
# Exit Code: 0
# stdout:
0000000 20 20 20 20 41 49 58 3a 0a 0a 0a 0a 0a 0a 0a 0a 0000020 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 1u�[?25h�[?2026l 1u�[?25h�[?2026l�[?2026h�[?25l�[?2004h�[ 36836.debug.mandb-lsof.txt Applications Brewfile.lock.json Desktop Documents Downloads Dropbox Library Local Sites Movies Music Pictures PlayOnMac's virtual drives Postman Public a a.log a.py a.sh a.ts abc ai_overlay_tmp arm-target athame bentoml bin blesh-debug-mandb-brew.tar.gz blesh-debug-mandb-lsof.tar.gz captain-hardcore dead.letter deno.lock env.sh fonts go iCloud Drive (Archive) init.lua intelephense mbox miniforge3 nltk_data node_modules opt out.txt package-lock.json pnpm pnpm-lock.yaml test_out.afm test_out.ttf tmux-client-54992.log tmux-client-90135.log 0000100 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0000112
# stderr:
$ echo '\ \ \ \ AIX:' | groff -T utf8 -m man | /usr/bin/awk '{print NF;}'
# Exit Code: 0
# stdout:
1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
# stderr:
$ printf '\x20\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0' | /usr/bin/awk '{print NF;}'
# Exit Code: 0
# stdout:
1
# stderr:
$ type ble/complete/mandb/convert-mandoc
# Exit Code: 0
# stdout:
ble/complete/mandb/convert-mandoc is a function ble/complete/mandb/convert-mandoc () { if [[ $_ble_util_locale_encoding == UTF-8 ]]; then ble/bin/groff -k -Tutf8 -man; else ble/bin/groff -Tascii -man; fi }
# stderr:
$ echo '\ \ \ \ AIX:' | ble/complete/mandb/convert-mandoc | od -t x1
# Exit Code: 0
# stdout:
0000000 20 20 20 20 41 49 58 3a 0a 0a 0a 0a 0a 0a 0a 0a 0000020 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 1u�[?25h�[?2026l 1u�[?25h�[?2026l�[?2026h�[?25l�[?2004h�[ 36836.debug.mandb-lsof.txt Applications Brewfile.lock.json Desktop Documents Downloads Dropbox Library Local Sites Movies Music Pictures PlayOnMac's virtual drives Postman Public a a.log a.py a.sh a.ts abc ai_overlay_tmp arm-target athame bentoml bin blesh-debug-mandb-brew.tar.gz blesh-debug-mandb-lsof.tar.gz captain-hardcore dead.letter deno.lock env.sh fonts go iCloud Drive (Archive) init.lua intelephense mbox miniforge3 nltk_data node_modules opt out.txt package-lock.json pnpm pnpm-lock.yaml test_out.afm test_out.ttf tmux-client-54992.log tmux-client-90135.log 0000100 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0000112
# stderr:
$ printf '\x20\xc2\xa0\xc2\xa0' | /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}' | od -t x1
# Exit Code: 0
# stdout:
# stderr:
/usr/bin/awk: towc: multibyte conversion failure on: '†¬†' input record number 1, file source line number 1 A7e
Got the above on first run, trying to replicate it again, but can't produce the behavior anymore, did the patch include a potential fix, because it seems like I no longer get anything slipping through :) |
Thank you so much for these results with the details of exit code/stdout/stderr!
Excellent, this must be the core part of the issue!
|
Sorry, felt asleep last night lol. Aha that's wonderful! Yea, I wrote this very basic utility to run the test commands: dug
function dug {
> out.txt
while IFS= read -r line; do
cmd="${line/#\$ /}"
stdout=$(eval "$cmd")
stderr=$(eval "$cmd" 2>&1 1>/dev/null)
exit_code=$?
{
echo '```bash'
echo $line
echo "# Exit Code: $exit_code"
echo "# stdout:"
echo $stdout
echo "# stderr:"
echo $stderr
echo '```'
echo ""
} >> out.txt
done < "$1"
cat out.txt
rm out.txt
} A9a$ bash blesh-gh515-test-awk.sh.txt
printf '\x20\xc2\xa0\xc2\xa0' | /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}': FAIL
printf '\xc2\xa0\xc2\xa0' | /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}': FAIL
printf '\xc2\xa0\u00A0' | /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}': FAIL
printf '\xc2\xa0x' | /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}': FAIL
printf '\xc2\xa0' | /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}': FAIL
printf '\u00A0' | /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}': OK
printf '\xc2\xa0' | /usr/bin/awk '{gsub(/[\x00-\x1F]/, "", $0); print $0}': FAIL
printf '\xc2\xa0' | /usr/bin/awk '{gsub(/[\x00-\x1F]/, ""); print $0}': FAIL
printf '\xc2\xa0' | /usr/bin/awk '{gsub(/[\x00-\x1F]/, "")}': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x1F]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x01-\x1F]/, "")': OK
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x1E]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x1D]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x1C]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x1B]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x1A]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x19]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x18]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x17]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x16]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x15]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x14]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x13]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x12]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x11]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x10]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x0F]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x0E]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x0D]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x0C]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x0B]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x0A]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x09]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x08]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x07]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x06]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x05]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x04]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x03]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x02]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x01]/, "")': FAIL
printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x00]/, "")': FAIL A9b$ printf '\x20\xc2\xa0\xc2\xa0' | LC_COLLATE=C /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}'
# Exit Code: 2
# stdout:
# stderr:
/usr/bin/awk: towc: multibyte conversion failure on: '†¬†' input record number 1, file source line number 1 $ printf '\x20\xc2\xa0\xc2\xa0' | LC_CTYPE=C LC_COLLATE=C /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}'
# Exit Code: 0
# stdout:
  
# stderr:
$ printf '\x20\xc2\xa0\xc2\xa0' | LC_ALL=C /usr/bin/awk '{line = $0; gsub(/[\x00-\x1F]/, "", line); print line}'
# Exit Code: 0
# stdout:
  
# stderr:
|
Thank you! The minimal reproducer of the error would be this: $ printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x00]/, "")' Since I think this is a bug of the UTF-8 support of macOS
Thank you, so
$ printf '\xc2\xa0' | /usr/bin/awk 'gsub(/\x00/, "")'
$ printf '\xc2\xa0' | LC_CTYPE=C /usr/bin/awk 'gsub(/[\x00-\x00]/, "")' |
Aha, yea that makes sense, I can confirm that I see a difference between the default awk installation under macOS and the latest awk I installed with brew: $ /usr/bin/awk --version
# Exit Code: 0
# stdout:
awk version 20200816
# stderr:
$ printf '\xc2\xa0' | /usr/bin/awk 'gsub(/[\x00-\x00]/, "")'
# Exit Code: 2
# stdout:
# stderr:
/usr/bin/awk: towc: multibyte conversion failure on: '†' input record number 1, file source line number 1 $ /opt/homebrew/bin/awk --version
# Exit Code: 0
# stdout:
awk version 20240728
# stderr:
$ printf '\xc2\xa0' | /opt/homebrew/bin/awk 'gsub(/[\x00-\x00]/, "")'
# Exit Code: 0
# stdout:
 
# stderr:
A9C$ printf '\xc2\xa0' | /usr/bin/awk 'gsub(/\x00/, "")'
# Exit Code: 0
# stdout:
# stderr:
$ printf '\xc2\xa0' | LC_CTYPE=C /usr/bin/awk 'gsub(/[\x00-\x00]/, "")'
# Exit Code: 0
# stdout:
# stderr:
Thank you so much for digging deep into this! |
The source code of macOS awk is provided here. It seems to be based on I tried to find a way to report a bug to Apple. There seem to be several ways, but they seem to require an Apple user account and detailed information on the macOS computer. Could you report the bug? I think the following result you provided is sufficient to demonstrate the bug in the report:
It should also be mentioned that
Thank you for these results. The above one means that |
Ah that's interesting. Sure, I just filled a bug report with macOS Feedback Assistant including a link to your response (FB15548158).
Awesome, ty. |
Thanks! |
ble version: 0.4.0-nightly+32f290d
Bash version: 5.2.37(1)-release (aarch64-apple-darwin23.4.0)
On some commands I noticed that typing
--
will break the current input and leak other output.eg
As soon as I've typed the
--
this happens:The text was updated successfully, but these errors were encountered: