Crash after many PVR timer updates #18402

chris-NR · 2020-09-10T15:32:54Z

Bug report

Describe the bug

kodi is crashing regularly (approx once per day) whilst unattended. I had a look at the logs (see below) and deduced that the problem may be related to PVR timer updates and so I set logging to debug (including the PVR component). The system crashed a few hours later. Other than the crash I have not observed any other issues.

Expected Behavior

The system should not crash

Actual Behavior

System crashes regularly.

To Reproduce

Unable to reproduce on demand but will happen within a day or so. I noticed 190 cycles of 795 pvr timer updates in the log.

Debuglog

The log file was 15M so I had to shorten it by removing 188 pvr timer cycles
debug log

Here is a summary of the 190 pvr timer cycles:
pvr timer cycles

Your Environment

Used Operating system:

lrusak · 2020-09-10T15:44:56Z

It is worth testing with Kodi master as Kodi 18 may not see any more updates.

ksooo · 2020-09-12T12:42:31Z

Throws in main thread whike trying to copy a std::string, which leads to ABORT. I doubt that this is related to PVR timer updates.

Thread 1 (Thread 0xb37fd9e0 (LWP 8674)):
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0xb4e43260 in __GI_abort () at abort.c:79
#2  0xb4d5b870 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#3  0xb4d59584 in ?? () from /usr/lib/libstdc++.so.6
#4  0xb4d595ec in std::terminate() () from /usr/lib/libstdc++.so.6
#5  0xb4d59938 in __cxa_throw () from /usr/lib/libstdc++.so.6
#6  0xb4d82800 in std::__throw_length_error(char const*) () from /usr/lib/libstdc++.so.6
#7  0xb4d9d398 in std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) () from /usr/lib/libstdc++.so.6
#8  0xb4d9e3b0 in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned int) () from /usr/lib/libstdc++.so.6
#9  0xb4d9eb70 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) () from /usr/lib/libstdc++.so.6
#10 0x006a3ef0 in CSysInfo::TranslateInfo(int) const ()
#11 0x006edc3c in CInfoLoader::GetInfo(int) ()
#12 0x0065b5fc in KODI::GUILIB::GUIINFO::CSystemGUIInfo::GetLabel(std::string&, CFileItem const*, int, KODI::GUILIB::GUIINFO::CGUIInfo const&, std::string*) const ()
#13 0x0065ead4 in KODI::GUILIB::GUIINFO::CGUIInfoProviders::GetLabel(std::string&, CFileItem const*, int, KODI::GUILIB::GUIINFO::CGUIInfo const&, std::string*) const ()
#14 0x00543da0 in CGUIInfoManager::GetLabel(int, int, std::string*) const ()
#15 0x0070c50c in CGUIWindowSystemInfo::SetControlLabel(int, char const*, int, int) ()
#16 0x0070c8b4 in CGUIWindowSystemInfo::FrameMove() ()
#17 0x00603f3c in CGUIWindowManager::FrameMove() ()
#18 0x00559f2c in CApplication::FrameMove(bool, bool) ()
#19 0x0052658c in CXBApplicationEx::Run(CAppParamParser const&) ()
#20 0x006a0340 in XBMC_Run ()
#21 0x002bb250 in main ()

chris-NR · 2020-09-25T10:57:19Z

UPDATE: recent crash history:
Date: Thu Sep 10 12:54:30 BST 2020
Date: Sat Sep 19 04:59:56 BST 2020
Date: Sat Sep 19 12:23:52 BST 2020
Date: Sat Sep 19 19:41:27 BST 2020
Date: Sun Sep 20 01:06:27 BST 2020
Date: Mon Sep 21 03:42:27 BST 2020
Date: Mon Sep 21 21:47:25 BST 2020
Date: Wed Sep 23 01:05:57 BST 2020
Date: Wed Sep 23 10:55:38 BST 2020
Date: Thu Sep 24 13:15:28 BST 2020
Date: Fri Sep 25 03:17:05 BST 2020

I have summarised the logs as follows:

SYSTEM INFO SECTION
Aborting Thread (1)
Last 10 lines of log file

Please see here

Let me know if there is any further info you need

djp952 · 2020-09-27T01:22:34Z

Have you considered checking with the MythTV PVR addon maintainer? You seem to have a great deal of timers, and quick look at the v5.10.16 code for that addon would indicate that the timer ids should never go negative. Note this in your original debug log (line 1995):

2020-09-10 05:03:14.234 T:2627773296   DEBUG: UpdateEntries: Updated timer 2334 on client 440135507
2020-09-10 05:03:14.234 T:2627773296   DEBUG: UpdateEntries: Updated timer 12464 on client 440135507
2020-09-10 05:03:14.234 T:2627773296   DEBUG: UpdateEntries: Updated timer -2033289534 on client 440135507
2020-09-10 05:03:14.235 T:2627773296   DEBUG: UpdateEntries: Updated timer -2033287198 on client 440135507
2020-09-10 05:03:14.235 T:2627773296   DEBUG: UpdateEntries: Updated timer -1965857906 on client 440135507
2020-09-10 05:03:14.236 T:2627773296   DEBUG: UpdateEntries: Updated timer -1965827677 on client 440135507
2020-09-10 05:03:14.236 T:2627773296   DEBUG: UpdateEntries: Updated timer -1800398970 on client 440135507

It crashes not long after the index goes negative both times. If I found the correct place in the addon code, the maximum value for an ID that isn't a recording is 0x7FFFFFFF, so they should always be a positive number in the Kodi log. Also note the drastic change in number range, up until they go negative they never seem to exceed 15000 or so. It could very well be that the amount of timers you have is exceeding some internal limitation in the addon and it starts sending bad strings to Kodi.

The repo I found for the addon v5.10.16 was here, not sure if it's the primary developer or not: https://github.com/janbar/pvr.mythtv

chris-NR · 2020-09-27T15:29:23Z

Thanks for looking into this. I have turned on additional debug info for the mythtv-pvr plugin and will pass this on to that team next time it crashes......

chris-NR · 2020-10-01T14:21:37Z

latest crash (mythtv-pvr specific logging, no kodi debug logging) here
am running now with kodi debug logging and mythtv-pvr logging but not crashing so reliably (typical!)

willat8 · 2020-10-01T14:31:39Z

I've had this issue for years, but it's pretty much disappeared when running LibreElec 9.2.5. Went over 21 days with no crashes whereas before that I'd be lucky to make it to 2.

chris-NR · 2020-10-01T18:36:14Z

Exact opposite in my case. I have also been running for years and don't recall seeing this issue until I moved from Raspian and an earlier version of kodi 18.x to LibreElec 9.2.5.

willat8 · 2020-10-01T22:01:16Z

Dont want to get too off-topic, but do you have your CEC adaptor set to power off your TV when the screensaver activates? Around when I stopped having regular crashes when using only the MythTV plugin on Kodi, I'd also swapped to using https://kodi.wiki/view/Add-on:Turn_Off to handle CEC power off.

chris-NR · 2020-10-01T22:26:58Z

No, the screensaver does not turn off the TV via CEC (although not a bad idea, I didn't know that was possible). As of now I have not had it crash with both mythtv-pvr extra logging AND kodi debug logging, still waiting.....

chris-NR · 2020-10-03T09:00:01Z

UPDATE:
Managed to get the system to crash with both kodi AND mythtv-pvr debug logging. I have also discovered that the system only appears to crash if I leave it on the main kodi TV page when switching off for the night. Steps to reproduce are:

Watch some TV
Stop watching TV
Return to the main TV page in kodi
Switch off the "monitor" - in my case a wide screen TV connected over HDMI
Wait for crash....

notes: 1 & 2 may not be needed, screensaver has not kicked in prior to turning off.

I now have a 526Mb log file which I have summarised (section ommited indicated by ---- SNIP) see here

ksooo · 2020-10-03T09:49:39Z

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0xb4e62260 in __GI_abort () at abort.c:79
#2  0xb4d7a870 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#3  0xb4d78584 in ?? () from /usr/lib/libstdc++.so.6
#4  0xb4d785ec in std::terminate() () from /usr/lib/libstdc++.so.6
#5  0xb4d78938 in __cxa_throw () from /usr/lib/libstdc++.so.6
#6  0xb4da1800 in std::__throw_length_error(char const*) () from /usr/lib/libstdc++.so.6
#7  0xb4dbc398 in std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) () from /usr/lib/libstdc++.so.6
#8  0xb4dbd3b0 in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned int) () from /usr/lib/libstdc++.so.6
#9  0xb4dbdc44 in std::string::assign(std::string const&) () from /usr/lib/libstdc++.so.6
#10 0x00481100 in PVR::CPVRGUIInfo::GetPVRLabel(CFileItem const*, KODI::GUILIB::GUIINFO::CGUIInfo const&, std::string&) const ()
...

Is the system running out of memeory?

ksooo · 2020-10-03T09:51:21Z

@fritsch @lrusak do you have an idea what might be the reason for std::string ctor failing with a length_error

fritsch · 2020-10-03T10:58:37Z

Is a char* passed that has no \0 terminator?

Edit: I also think of a memory leak. If that's the case we should see something in dmesg if OOM was involved.

mglae · 2020-10-03T13:37:07Z

@CoolDreamZ Executing touch /storage/.config/debug.enhanced will create a verbose backtrace on further crashes. They may be helpful for research.

chris-NR · 2020-10-03T17:30:38Z

@mglae will do, thanks

chris-NR · 2020-10-05T07:33:58Z

LATEST CRASH
journal
dmesg
crash log with enhanced backtrace

chris-NR · 2020-10-26T14:09:48Z

UPDATE Latest 10 crashes

Date: Fri Oct 16 03:27:12 BST 2020
Date: Fri Oct 16 20:38:25 BST 2020
Date: Sun Oct 18 18:53:38 BST 2020
Date: Mon Oct 19 09:10:20 BST 2020
Date: Tue Oct 20 04:13:50 BST 2020
Date: Wed Oct 21 10:14:03 BST 2020
Date: Thu Oct 22 06:02:13 BST 2020
Date: Sat Oct 24 05:47:44 BST 2020
Date: Sun Oct 25 05:19:19 GMT 2020
Date: Mon Oct 26 11:00:42 GMT 2020

crash occurs whether or not the mythtv backend server is online
I am 90% certain the crash only occurs if the kodi is left on the main TV page

I have summarised the logs as follows:

SYSTEM INFO SECTION
Aborting Thread (1)
Last 10 lines of log file

Please see here

[amusingly pastebin thinks my log files are potentially offensive!]

janbar · 2020-12-23T18:53:31Z

It could be caused by threads competition on a GUI label of the current screen page. One consumes the string and an other change it at the same time (probably because some timers have been updated).

ksooo · 2020-12-23T19:26:13Z

It could be caused by threads competition on a GUI label of the current screen page.

In the crash logs provided so far I cannot see any actions like that.

chris-NR · 2021-01-01T15:19:32Z

UPDATE:

The system has continued to crash every 1-2 days so I created a fresh installation of:

LibreElec 9.2.6
kodi 18.9
(mythtv.pvr 5.10.19)

and restored a backup of the previous 9.2.5 installation into it.

The system crashed a few hours later (on the main TV page)

here is the log

janbar · 2021-01-01T18:51:22Z

Hi, the log show the root cause:

#10 0x00481138 in PVR::CPVRGUIInfo::GetPVRLabel(CFileItem const*, KODI::GUILIB::GUIINFO::CGUIInfo const&, std::string&) const ()
#11 0x004919c4 in PVR::CPVRGUIInfo::GetLabel(std::string&, CFileItem const*, int, KODI::GUILIB::GUIINFO::CGUIInfo const&, std::string*) const ()

ksooo · 2021-01-01T19:14:31Z

Hi, the log show the root cause:

Aha, and what exactly goes wrong in this method and where (source code line, please). I unfortunately cannot find anything suspicious there. At the end, a string assignment goes wrong. This can have different reasons.

ksooo · 2021-01-01T19:19:56Z

http://www.cplusplus.com/reference/string/string/assign/ => "If the resulting string length would exceed the max_size, a length_error exception is thrown."

So, it could be a very large string causing the problem. Hmm.

ksooo · 2021-01-01T19:25:05Z

#9 0xb4d57c44 in std::string::assign(std::string const&) () from /usr/lib/libstdc++.so.6

And even more strange is that the log indicates, that the string copy function causes the error. So, the "too long" string must already exist in the source object as length of source and copy should be identical, this all does not make any sense to me.

fritsch · 2021-01-01T19:35:28Z

memcpy gets:
0xb4e10270 <+52>: mvn r0, #-2147483648 ; 0x80000000

while 0x80000000 here is intepreted as a negative number, while memcpy uses it unsigned.

I see that the original method hands in pointers to strings. Is there a chance that the string at this position vanishes and just the ptr (which was passed in by value) remains to this position and Copy just reads from garbage?

ksooo · 2021-01-01T19:57:59Z

The crashing method GetPVRLabel does not even use the data provided as std::string*

bool CPVRGUIInfo::GetLabel(std::string& value, const CFileItem* item, int contextWindow, const CGUIInfo& info, std::string* fallback) const
{
  return GetListItemAndPlayerLabel(item, info, value) ||
         GetPVRLabel(item, info, value) ||
         GetRadioRDSLabel(item, info, value);
}

fritsch · 2021-01-01T20:05:30Z

Okay, then it's the assignment that happens in GetPVRLabel. Here we actually copy stuff into value. Perhaps a very simple approach of adding some prints to exactly see which value is copied into it, might shed some light?

fritsch · 2021-01-01T20:12:57Z

I don't fully understand the code in GetPVRLabel, but I see here that some shared_ptr might potentially point to nullptr, but e.g. info.GetData1() is called never the less.

Most methods here return true, some even if no copy (assignment) has happened.

I don't know this code, therefore it might just use a fallback in that case.

ksooo · 2021-01-01T21:07:05Z

I see here that some shared_ptr might potentially point to nullptr, but e.g. info.GetData1() is called never the less.

This is perfectly okay. I know the code very well. The potential nullptr is passed only to methods designed to handle this case properly.

fritsch · 2021-01-02T08:38:51Z

Okay. As the problem seems to happen during a normal string copy operation (e.g. assignment), there might be just some \0 missing in one of the supplied data.

What we could try: limit the amount of chars we copy in that method, e.g. Create a temporary string with max length of 1000 and assign that one? If the bug is still triggered, we have some stack corruption.

I don't really see another trap to detect this bug more easily.

fritsch · 2021-01-02T08:42:14Z

E.g. assign inputstr.substr(0, 1000);
We could even try to log, if such a resulting string has the length of 1000 and that way find the error case? (Given, that really something in that string is wrong).

ksooo · 2021-01-02T09:43:28Z

The real question is how to provide a test build for the issue submitter. He's using libreelec. I'm out in this case. I long time ago stopped building LE myself.

fritsch · 2021-01-02T09:47:16Z

Posting a simple patch should be fine. LE maintainers should step up and provide that for their user I think.

chris-NR · 2021-01-19T10:11:43Z

@fritsch I have built LibreElec 9.2.6 from source in my fork here

I restored my original setup into it using LibreElec restore and it still crashes, new log here

If you are able to provide a patch I will test it

chris-NR · 2022-05-07T20:00:22Z

I have upgraded to Kodi 19.4 via LibreElec 10.0.2

2022-05-04 15:19:12.935 T:8486     INFO <general>: -----------------------------------------------------------------------
2022-05-04 15:19:12.935 T:8486     INFO <general>: Starting Kodi (19.4 (19.4.0) Git:19.4-Matrix). Platform: Linux ARM 32-bit
2022-05-04 15:19:12.935 T:8486     INFO <general>: Using Release Kodi x32
2022-05-04 15:19:12.935 T:8486     INFO <general>: Kodi compiled 2022-03-05 by GCC 10.2.0 for Linux ARM 32-bit version 5.10.95 (330335)
2022-05-04 15:19:12.935 T:8486     INFO <general>: Running on BCM2835 with LibreELEC (official): 10.0.2, kernel: Linux ARM 64-bit version 5.10.95
2022-05-04 15:19:12.935 T:8486     INFO <general>: FFmpeg version/source: 4.3.2-Kodi
2022-05-04 15:19:12.935 T:8486     INFO <general>: 4 CPU cores available
2022-05-04 15:19:12.935 T:8486     INFO <general>: ARM Features: Neon enabled

The crash is still happening when left unattended on the "TV" main menu page.

Thread 1 (Thread 0xf3859040 (LWP 8486)):
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#1  0xf5432798 in __GI_abort () at abort.c:79
#2  0xf5315c4c in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#3  0xf5313850 in ?? () from /usr/lib/libstdc++.so.6
#4  0xf53138d0 in std::terminate() () from /usr/lib/libstdc++.so.6
#5  0xf5313c64 in __cxa_throw () from /usr/lib/libstdc++.so.6
#6  0xf530f478 in std::__throw_length_error(char const*) () from /usr/lib/libstdc++.so.6
#7  0xf5357438 in std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) () from /usr/lib/libstdc++.so.6
#8  0xf535837c in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned int) () from /usr/lib/libstdc++.so.6
#9  0xf5358c98 in std::string::assign(std::string const&) () from /usr/lib/libstdc++.so.6
#10 0x007932fc in PVR::CPVRGUIInfo::GetPVRLabel(CFileItem const*, KODI::GUILIB::GUIINFO::CGUIInfo const&, std::string&) const ()
#11 0x00781c88 in PVR::CPVRGUIInfo::GetLabel(std::string&, CFileItem const*, int, KODI::GUILIB::GUIINFO::CGUIInfo const&, std::string*) const ()
#12 0x005ba00c in KODI::GUILIB::GUIINFO::CGUIInfoProviders::GetLabel(std::string&, CFileItem const*, int, KODI::GUILIB::GUIINFO::CGUIInfo const&, std::string*) const ()
#13 0x006fd5e8 in CGUIInfoManager::GetLabel(int, int, std::string*) const ()
#14 0x006fd818 in CGUIInfoManager::GetImage(int, int, std::string*) ()
#15 0x005be18c in KODI::GUILIB::GUIINFO::CGUIInfoLabel::GetLabel(int, bool, std::string*) const ()
#16 0x005eee04 in CGUIImage::UpdateInfo(CGUIListItem const*) ()
#17 0x005eb7f8 in CGUIImage::UpdateVisibility(CGUIListItem const*) ()
#18 0x005e12c4 in CGUIControlGroup::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#19 0x005d7e14 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#20 0x005e23fc in CGUIControlGroupList::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#21 0x005d7e14 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#22 0x005e23fc in CGUIControlGroupList::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#23 0x005d7e14 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#24 0x005e12e4 in CGUIControlGroup::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#25 0x005d7e14 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#26 0x005e12e4 in CGUIControlGroup::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#27 0x005d7e14 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#28 0x005e12e4 in CGUIControlGroup::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#29 0x005d7e14 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#30 0x005e12e4 in CGUIControlGroup::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#31 0x005d7e14 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#32 0x0060c62c in CGUIWindow::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#33 0x006177d4 in CGUIWindowManager::Process(unsigned int) ()
#34 0x006d3814 in CApplication::FrameMove(bool, bool) ()
#35 0x0071cabc in CXBApplicationEx::Run(CAppParamParser const&) ()
#36 0x0028f1a8 in main ()
############# END STACK TRACE ###############

The last few lines in the log are:

2022-05-07 01:29:00.474 T:8622    ERROR <general>: CPVRTimerInfoTag: No epg tag given for epg based timer type (16)!
2022-05-07 01:32:46.565 T:8622     INFO <general>: Skipped 28 duplicate messages..
2022-05-07 01:32:46.565 T:8622    ERROR <general>: CPVRTimerInfoTag: No epg tag given for epg based timer type (16)!
2022-05-07 01:37:46.763 T:8622     INFO <general>: Skipped 28 duplicate messages..
2022-05-07 01:37:46.763 T:8622    ERROR <general>: CPVRTimerInfoTag: No epg tag given for epg based timer type (16)!
2022-05-07 01:38:38.446 T:8622     INFO <general>: Skipped 28 duplicate messages..
2022-05-07 01:38:38.446 T:8622    ERROR <general>: CPVRTimerInfoTag: No epg tag given for epg based timer type (16)!
2022-05-07 01:39:01.359 T:8622     INFO <general>: Skipped 28 duplicate messages..
2022-05-07 01:39:01.359 T:8622    ERROR <general>: CPVRTimerInfoTag: No epg tag given for epg based timer type (16)!

############### END LOG FILE ################

janbar · 2022-05-09T19:58:42Z

The throw exception is typically caused by a vector reallocation. And the code still reference a pointer to the old memory segment.

github-actions · 2024-08-05T21:13:39Z

This issue is now marked stale because it has been open over a year without activity. Remove the stale label or add a comment to reset the stale state.

xbmc-gh-bot bot added the Triage: Needed (managed by bot!) issue that was just created and needs someone looking at it label Sep 10, 2020

lrusak assigned ksooo Sep 10, 2020

ksooo removed their assignment Sep 12, 2020

chris-NR mentioned this issue Oct 1, 2020

PVR timer updates may be causing kodi to crash regularly janbar/pvr.mythtv#158

Open

github-actions bot added the Stale Due to inactivity a future closure of this thread is planned. To cancel remove this label label Aug 5, 2024

Crash after many PVR timer updates #18402

Crash after many PVR timer updates #18402

Comments

chris-NR commented Sep 10, 2020 • edited Loading

Bug report

Describe the bug

Expected Behavior

Actual Behavior

To Reproduce

Debuglog

Your Environment

lrusak commented Sep 10, 2020

ksooo commented Sep 12, 2020 • edited Loading

chris-NR commented Sep 25, 2020 • edited Loading

djp952 commented Sep 27, 2020

chris-NR commented Sep 27, 2020

chris-NR commented Oct 1, 2020

willat8 commented Oct 1, 2020

chris-NR commented Oct 1, 2020

willat8 commented Oct 1, 2020

chris-NR commented Oct 1, 2020

chris-NR commented Oct 3, 2020

ksooo commented Oct 3, 2020

ksooo commented Oct 3, 2020

fritsch commented Oct 3, 2020 • edited Loading

mglae commented Oct 3, 2020

chris-NR commented Oct 3, 2020

chris-NR commented Oct 5, 2020

chris-NR commented Oct 26, 2020

janbar commented Dec 23, 2020

ksooo commented Dec 23, 2020

chris-NR commented Jan 1, 2021

janbar commented Jan 1, 2021 • edited Loading

ksooo commented Jan 1, 2021 • edited Loading

ksooo commented Jan 1, 2021

ksooo commented Jan 1, 2021 • edited Loading

fritsch commented Jan 1, 2021

ksooo commented Jan 1, 2021

fritsch commented Jan 1, 2021

fritsch commented Jan 1, 2021

ksooo commented Jan 1, 2021

fritsch commented Jan 2, 2021

fritsch commented Jan 2, 2021

ksooo commented Jan 2, 2021 • edited Loading

fritsch commented Jan 2, 2021

chris-NR commented Jan 19, 2021

chris-NR commented May 7, 2022

janbar commented May 9, 2022 • edited Loading

github-actions bot commented Aug 5, 2024

chris-NR commented Sep 10, 2020 •

edited

Loading

ksooo commented Sep 12, 2020 •

edited

Loading

chris-NR commented Sep 25, 2020 •

edited

Loading

fritsch commented Oct 3, 2020 •

edited

Loading

janbar commented Jan 1, 2021 •

edited

Loading

ksooo commented Jan 1, 2021 •

edited

Loading

ksooo commented Jan 1, 2021 •

edited

Loading

ksooo commented Jan 2, 2021 •

edited

Loading

janbar commented May 9, 2022 •

edited

Loading