Error Analyzer - Print error msg

[Roy214]

Resolves: #6679

[Roy214]

• Print if there is any error reported by sssd.

$ sudo sssctl analyze error list
[sudo] password for sssd-git: 

 [be[health.dohmh.nycnet]] [sbus_issue_request_done] (0x0040): sssd.dataprovider.getAccountInfo: Error [1432158212]: SSSD is offline
 For details refer https://sssd.io/troubleshooting/errors.html

 (Fri Jun 10 15:40:00 2016) [sssd[be[ipa.example.com]]] [krb5_child_timeout] (0x0040): Timeout for child [23514] reached. In case KDC is distant or network is slow you may consider increasing value of krb5_auth_timeout.
 For details refer https://sssd.io/troubleshooting/errors.html

[alexey-tikhonov]

[sudo] password for sssd-git:

I think sssctl analyze shouldn't require 'root'?

[justin-stephenson]

Hi, Thank you for the PR @Roy214.

Overall the approach looks good to me, but some changes are still needed.

1. Only methods needed by error.py should be moved to util.py. Methods likes list_requests, print_formatted, etc... are still only used by the request module so they can stay in request.py

2. For methods which are moved to util.py, please also remove them from request.py and make sure sssctl analyze request still works as expected.

3. Add modules/error.py and modules/util.py to src/tools/analyzer/Makefile.am

[Roy214]

$ sudo sssctl analyze error list

 [be[health.dohmh.nycnet]] [sbus_issue_request_done] (0x0040): sssd.dataprovider.getAccountInfo: Error [1432158212]: SSSD is offline
 For details refer https://sssd.io/troubleshooting/errors.html

 (Fri Jun 10 15:40:00 2016) [sssd[be[ipa.example.com]]] [krb5_child_timeout] (0x0040): Timeout for child [23514] reached. In case KDC is distant or network is slow you may consider increasing value of krb5_auth_timeout.
 For details refer https://sssd.io/troubleshooting/errors.html

 (2023-01-23 18:10:48): [be[gsslab1.idm.global]] [sasl_bind_send] (0x0020): [RID#33] ldap_sasl_interactive_bind_s failed (-2)[Local error]
 For details refer https://sssd.io/troubleshooting/errors.html

[Roy214]

Hi @alexey-tikhonov,

Without root or sudo it's failing.

[sssd-git@rhel8 analyzer]$ sssctl analyze error list
Traceback (most recent call last):
  File "/usr/libexec/sssd/sss_analyze", line 5, in <module>
    sss_analyze.run()
  File "/usr/lib/python3.6/site-packages/sssd/sss_analyze.py", line 107, in run
    analyzer.main()
  File "/usr/lib/python3.6/site-packages/sssd/sss_analyze.py", line 102, in main
    args.func(args)
  File "/usr/lib/python3.6/site-packages/sssd/modules/error.py", line 62, in print_error
    source.set_component(component, False)
  File "/usr/lib/python3.6/site-packages/sssd/source_files.py", line 73, in set_component
    raise IOError
OSError
Command '/usr/libexec/sssd/sss_analyze' failed with [1]
[sssd-git@rhel8 analyzer]$ sssctl analyze request list
Traceback (most recent call last):
  File "/usr/libexec/sssd/sss_analyze", line 5, in <module>
    sss_analyze.run()
  File "/usr/lib/python3.6/site-packages/sssd/sss_analyze.py", line 107, in run
    analyzer.main()
  File "/usr/lib/python3.6/site-packages/sssd/sss_analyze.py", line 102, in main
    args.func(args)
  File "/usr/lib/python3.6/site-packages/sssd/modules/request.py", line 268, in list_requests
    for line in self.matched_line(source, patterns):
  File "/usr/lib/python3.6/site-packages/sssd/modules/request.py", line 94, in matched_line
    for line in source:
  File "/usr/lib/python3.6/site-packages/sssd/source_files.py", line 31, in __iter__
    with open(files) as file:
PermissionError: [Errno 13] Permission denied: '/var/log/sssd/sssd_nss.log' <---
Command '/usr/libexec/sssd/sss_analyze' failed with [1]

[Roy214]

@justin-stephenson,

I am yet to do the second part.

"For methods which are moved to util.py, please also remove them from request.py and make sure sssctl analyze request still works as expected."

[alexey-tikhonov]

Is it really useful to add
For details refer https://sssd.io/troubleshooting/errors.html
message to every line?

What about adding it to "footer" or "header" once?

[Roy214]

Is it really useful to add For details refer https://sssd.io/troubleshooting/errors.html message to every line?

What about adding it to "footer" or "header" once?

Something like below:

$ sudo sssctl analyze error list
(Fri Apr 14 15:07:19 2023) system1 sssd[sssd]: Child [1277] ('SSSDdomain':'%BE_SSSDdomain') was terminated by own WATCHDOG. Consult corresponding logs to figure out the reason.
 

(Fri Apr 14 16:07:19 2023) [sssd[be[example.com]]] [sssd_async_socket_init_done] (0x0020): sdap_async_sys_connect request failed: [110]: Connection timed out.
 

(Sat Apr 15 01:07:19 2023) [sssd[be[example.com]] [sbus_issue_request_done] (0x0040): sssd.dataprovider.getAccountInfo: Error [1432158212]: SSSD is offline
 

For possible solutions please refer https://sssd.io/troubleshooting/errors.html 

[Roy214]

@alexey-tikhonov,

Updated the error page link at the bottom.

[sssd-git@rhel8 sssd]$ sudo sssctl analyze error list
(Fri Apr 14 15:07:19 2023) system1 sssd[sssd]: Child [1277] ('SSSDdomain':'%BE_SSSDdomain') was terminated by own WATCHDOG. Consult corresponding logs to figure out the reason.
 

(Fri Apr 14 16:07:19 2023) [sssd[be[example.com]]] [sssd_async_socket_init_done] (0x0020): sdap_async_sys_connect request failed: [110]: Connection timed out.
 

(Sat Apr 15 01:07:19 2023) [sssd[be[example.com]] [sbus_issue_request_done] (0x0040): sssd.dataprovider.getAccountInfo: Error [1432158212]: SSSD is offline
 

For possible solutions please refer https://sssd.io/troubleshooting/errors.html 

[Roy214]

After deleting the matched_line and load function from resquest.py

[sssd-git@rhel8 analyzer]$ sudo sssctl analyze request list
[sudo] password for sssd-git: 
sudo: /etc/sudo.env: No such file or directory
2023-04-26 21:46:04: [uid 0] CID #1: sudo
2023-04-26 21:46:11: [uid 1009] CID #2: id
..

[sssd-git@rhel8 analyzer]$ sudo sssctl analyze request show 1
sudo: /etc/sudo.env: No such file or directory
(2023-04-26 21:46:04): [nss] [accept_fd_handler] (0x0400): [CID#1] Client [cmd sudo][uid 0][0x55e8d6d54ca0][28] connected!
(2023-04-26 21:46:04): [nss] [sss_cmd_get_version] (0x0200): [CID#1] Received client version [1].
(2023-04-26 21:46:04): [nss] [sss_cmd_get_version] (0x0200): [CID#1] Offered version [1].
(2023-04-26 21:46:04): [nss] [nss_getby_id] (0x0400): [CID#1] Input ID: 0 (looking up 'POSIX data')
(2023-04-26 21:46:04): [nss] [cache_req_set_plugin] (0x2000): [CID#1] CR #0: Setting "User by ID" plugin
(2023-04-26 21:46:04): [nss] [cache_req_send] (0x0400): [CID#1] CR #0: REQ_TRACE: New request [CID #1] 'User by ID'
(2023-04-26 21:46:04): [nss] [sss_domain_get_state] (0x1000): [CID#1] Domain gsslab1.idm.global is Active
....

[sssd-git@rhel8 analyzer]$ sudo sssctl analyze --logdir=/var/log/sssd request list
sudo: /etc/sudo.env: No such file or directory
2023-04-26 21:46:04: [uid 0] CID #1: sudo
2023-04-26 21:46:11: [uid 1009] CID #2: id
2023-04-26 21:46:12: [uid 974001347] CID #3: (sd-pam)
2023-04-26 21:46:27: [uid 0] CID #4: sudo

[sssd-git@rhel8 analyzer]$ sudo sssctl analyze request show 20 --pam
sudo: /etc/sudo.env: No such file or directory
ID 20 not found in logs!

[sssd-git@rhel8 analyzer]$ sudo sssctl analyze error list
sudo: /etc/sudo.env: No such file or directory
(Fri Apr 14 15:07:19 2023) system1 sssd[sssd]: Child [1277] ('SSSDdomain':'%BE_SSSDdomain') was terminated by own WATCHDOG. Consult corresponding logs to figure out the reason.
 

(Fri Apr 14 16:07:19 2023) [sssd[be[example.com]]] [sssd_async_socket_init_done] (0x0020): sdap_async_sys_connect request failed: [110]: Connection timed out.
 

(Sat Apr 15 01:07:19 2023) [sssd[be[example.com]] [sbus_issue_request_done] (0x0040): sssd.dataprovider.getAccountInfo: Error [1432158212]: SSSD is offline
 

For possible solutions please refer https://sssd.io/troubleshooting/errors.html 

[Roy214]

@justin-stephenson,
I have tried to build the sssd but seems like make.am is not added here.

make[3]: Leaving directory '/home/sssd-git/sssd_new/sssd/x86_64/src/tools/analyzer'
make[2]: Leaving directory '/home/sssd-git/sssd_new/sssd/x86_64/src/tools/analyzer'
make[1]: Leaving directory '/home/sssd-git/sssd_new/sssd/x86_64'
Build install successful

Waiting for review. I am not able to apply the label.

[justin-stephenson]

@justin-stephenson, I have tried to build the sssd but seems like make.am is not added here.

make[3]: Leaving directory '/home/sssd-git/sssd_new/sssd/x86_64/src/tools/analyzer'
make[2]: Leaving directory '/home/sssd-git/sssd_new/sssd/x86_64/src/tools/analyzer'
make[1]: Leaving directory '/home/sssd-git/sssd_new/sssd/x86_64'
Build install successful

Waiting for review. I am not able to apply the label.

Hi,

I don't see any changes to src/tools/analyzer/Makefile.am in this PR, it should be visible in the Files changed tab of this PR.

Please also check the problems identified by @github-code-scanning - they are commented here also and should be addressed.

It would be great if you can also squash your commits into a single commit (See interactive rebase approach here https://www.cloudbees.com/blog/git-squash-how-to-condense-your-commit-history)

[Roy214]

Hello @justin-stephenson,

Please have a look once.

[justin-stephenson]

Hello @justin-stephenson,

Please have a look once.

I see 2 commits now in this Pull Request, it should be just 1.

[Roy214]

@thalman,

I have made the changes could you please review. I can not change the status.

[thalman]

Thank you for the patch Roy. Looks good to me.

[Roy214]

@thalman @justin-stephenson, Do we need to change the labels.

[thalman]

I removed the "change requested" label, now it is waiting for your approval @justin-stephenson

[Roy214]

@justin-stephenson, "waiting for your review"

[justin-stephenson]

I am hoping someone from the SSSD team can approve SSSD/sssd.io#67 so these two PRs will be merged at the same time.

[andreboscatto]

@alexey-tikhonov @justin-stephenson @sumit-bose we were discussing this (#4830), do you think this kind of work can be also done with sssctl analyze?

[alexey-tikhonov]

@alexey-tikhonov @justin-stephenson @sumit-bose we were discussing this (#4830), do you think this kind of work can be also done with sssctl analyze?

@andreboscatto, could you please clarify the question?
How is it - man pages with sssctl?

[sumit-bose]

@alexey-tikhonov @justin-stephenson @sumit-bose we were discussing this (#4830), do you think this kind of work can be also done with sssctl analyze?

@andreboscatto, could you please clarify the question? How is it - man pages with sssctl?

Hi,

instead of reading a man page to troubleshoot Smartcard authentication use this tool to get the related log message and maybe some hints what to do to avoid the error.

bye,
Sumit

[justin-stephenson]

@alexey-tikhonov @justin-stephenson @sumit-bose we were discussing this (#4830), do you think this kind of work can be also done with sssctl analyze?

@andreboscatto, could you please clarify the question? How is it - man pages with sssctl?

Hi,

instead of reading a man page to troubleshoot Smartcard authentication use this tool to get the related log message and maybe some hints what to do to avoid the error.

bye, Sumit

For sure this is possible, it is essentially the same as what @Roy214 is creating here but for SSSD general error messages, i.e. Search for certain log messages, and print a link to a sssd.io page with tips on how resolve it.

This is more reactive than proactive, geared towards troubleshooting over general setup and configuration however.

[Roy214]

@justin-stephenson @sumit-bose

@alexey-tikhonov @justin-stephenson @sumit-bose we were discussing this (#4830), do you think this kind of work can be also done with sssctl analyze?

@andreboscatto, could you please clarify the question? How is it - man pages with sssctl?

Hi,
instead of reading a man page to troubleshoot Smartcard authentication use this tool to get the related log message and maybe some hints what to do to avoid the error.
bye, Sumit

For sure this is possible, it is essentially the same as what @Roy214 is creating here but for SSSD general error messages, i.e. Search for certain log messages, and print a link to a sssd.io page with tips on how resolve it.

This is more reactive than proactive, geared towards troubleshooting over general setup and configuration however.

@justin-stephenson @alexey-tikhonov @andreboscatto @sumit-bose If you guys agree I can try to implement the same kind of stuff for #4830

[andreboscatto]

Why not?! If you have suitable time it would be great :)

[justin-stephenson]

Approved from my side, now that the sssd.io errors page is merged.

[pbrezina]

I just changed the commit subject and rebased.

[pbrezina]

Pushed PR: #6686

• master
  • ed3726c - sssctl: add error analyzer