Compare commits
2 Commits
d6bcc06832
...
b40dd0828a
| Author | SHA1 | Date | |
|---|---|---|---|
| b40dd0828a | |||
| c1127be692 |
23
infra_morale.log
Normal file
23
infra_morale.log
Normal file
@ -0,0 +1,23 @@
|
|||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
File diff suppressed because it is too large
Load Diff
@ -684,3 +684,283 @@ paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
|
|||||||
⚡ 1 warnings found:
|
⚡ 1 warnings found:
|
||||||
- 💥 [db2] WARNING: Replication lag is 73 seconds.
|
- 💥 [db2] WARNING: Replication lag is 73 seconds.
|
||||||
✅ Genesis Radio Healthcheck 2025-04-29 09:45:14: All systems normal.
|
✅ Genesis Radio Healthcheck 2025-04-29 09:45:14: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 10:00:13: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 10:15:13: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 10:30:15: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 10:45:29: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 11:00:11: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 11:15:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 11:30:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 11:45:16: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 12:00:12: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 12:15:17 ⚠️
|
||||||
|
⚡ 2 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 12:30:17: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 12:45:16: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 13:00:13: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 13:15:13: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 13:30:14: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 13:45:14: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 14:00:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 14:15:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 14:30:11: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 14:45:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 15:00:16: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 15:16:51 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 15:32:34 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 15:45:24: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 16:00:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 16:15:21: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 16:30:27: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 16:45:20: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 17:00:20: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 17:15:18 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 17:30:18 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 17:45:16: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 18:00:17: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 18:15:27 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 18:30:24: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 18:45:22 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 19:00:17: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 19:15:19: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 19:30:11: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 19:45:11 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- 💥 [db2] WARNING: Replication lag is 68 seconds.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 20:00:17: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 20:15:21: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 20:30:16: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 20:45:10: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 21:00:14 ⚠️
|
||||||
|
⚡ 56 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 21:15:13 ⚠️
|
||||||
|
⚡ 3 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 21:30:26 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 21:45:28: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 22:00:26 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 22:15:28: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 22:30:22 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 22:45:26: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 23:00:23: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 23:15:33: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 23:30:23: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 23:45:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 00:00:23: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 00:15:24 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 00:30:16: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 00:45:23: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 01:00:20: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 01:15:17 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 01:30:19: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 01:45:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 02:00:21: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 02:15:25: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 02:30:21 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 02:45:13 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- 💥 [db2] WARNING: Replication lag is 60 seconds.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 03:00:15: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 03:15:13 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- 💥 [db2] WARNING: Replication lag is 61 seconds.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 03:30:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 03:45:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 04:00:11: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 04:15:20: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 04:30:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 04:45:21: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 05:00:19: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 05:15:21 ⚠️
|
||||||
|
⚡ 24 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 05:30:19: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 05:45:17 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 06:00:17: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 06:15:28 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 06:30:13: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 06:45:19 ⚠️
|
||||||
|
⚡ 5 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 07:00:13 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 07:15:10: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 07:30:12 ⚠️
|
||||||
|
⚡ 2 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 07:45:13: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 08:00:11: All systems normal.
|
||||||
|
Exception (client): Error reading SSH protocol banner[Errno 104] Connection reset by peer
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2369, in _check_banner
|
||||||
|
buf = self.packetizer.readline(timeout)
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/packet.py", line 395, in readline
|
||||||
|
buf += self._read_timeout(timeout)
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/packet.py", line 663, in _read_timeout
|
||||||
|
x = self.__socket.recv(128)
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
ConnectionResetError: [Errno 104] Connection reset by peer
|
||||||
|
|
||||||
|
During handling of the above exception, another exception occurred:
|
||||||
|
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2185, in run
|
||||||
|
self._check_banner()
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2373, in _check_banner
|
||||||
|
raise SSHException(
|
||||||
|
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner[Errno 104] Connection reset by peer
|
||||||
|
|
||||||
|
Exception (client): Error reading SSH protocol banner
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2369, in _check_banner
|
||||||
|
buf = self.packetizer.readline(timeout)
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/packet.py", line 395, in readline
|
||||||
|
buf += self._read_timeout(timeout)
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/packet.py", line 665, in _read_timeout
|
||||||
|
raise EOFError()
|
||||||
|
EOFError
|
||||||
|
|
||||||
|
During handling of the above exception, another exception occurred:
|
||||||
|
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2185, in run
|
||||||
|
self._check_banner()
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2373, in _check_banner
|
||||||
|
raise SSHException(
|
||||||
|
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
|
||||||
|
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 08:15:12 ⚠️
|
||||||
|
⚡ 10 warnings found:
|
||||||
|
- ⚠️ [shredder] ERROR: Could not read log /var/log/nginx/error.log: Error reading SSH protocol banner[Errno 104] Connection reset by peer
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 08:30:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 08:45:18: All systems normal.
|
||||||
|
|||||||
77
postmortem/genesisradiozfsmigration.md
Normal file
77
postmortem/genesisradiozfsmigration.md
Normal file
@ -0,0 +1,77 @@
|
|||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
# Define the content of the post-mortem
|
||||||
|
post_mortem_content = f"""# 🔧 Post-Mortem: Genesis Radio Storage Migration
|
||||||
|
**Date:** April 30, 2025
|
||||||
|
**Prepared by:** Doc
|
||||||
|
**Systems Affected:** StationPlaylist (SPL), Voice Tracker, Genesis Radio media backend
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧠 Executive Summary
|
||||||
|
|
||||||
|
Genesis Radio’s backend was migrated from a legacy MinIO instance using local disk (ext4) to a new **ZFS-based, encrypted MinIO deployment on `shredderv2`**. This change was driven by a need for more stable performance, improved security, and a cleaner storage architecture with proper bucket separation.
|
||||||
|
|
||||||
|
This migration was completed **without touching production** until final validation, and all critical services remained online throughout the transition. We also revamped the rclone caching strategy to reduce freeze-ups and playback hiccups.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ What We Did
|
||||||
|
|
||||||
|
- Created **three new secure buckets**: `genesislibrary-secure`, `genesisassets-secure`, and `genesisshows-secure`
|
||||||
|
- Migrated data from backup server using `rclone sync`:
|
||||||
|
- `genesislibrary` came directly from backup
|
||||||
|
- `genesisassets` and `genesisshows` were pulled from the same bucket, with de-duping and cleanup to be completed post-migration
|
||||||
|
- Retained **original SPL drive letters** (`Q:\\`, `R:\\`) to avoid changes to the playout config
|
||||||
|
- Switched rclone mounts to point to the new secure buckets, with **aggressive VFS caching** using SSD-backed cache directories
|
||||||
|
- Took a clean **ZFS snapshot** (`@pre-s3-switch`) before switching over
|
||||||
|
- Confirmed no regression in SPL, VT Tracker, or streaming audio
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚙️ Technical Improvements
|
||||||
|
|
||||||
|
- **VFS caching overhaul**:
|
||||||
|
- Increased read-ahead (`1G`), lowered write-back wait
|
||||||
|
- Split cache between `X:\\librarycache` and `L:\\assetcache`
|
||||||
|
- No more rclone choking on large files or freezing during transitions
|
||||||
|
- **Encrypted S3 storage** with isolated buckets per functional role
|
||||||
|
- **TLS-secured** Console and MinIO endpoints with automated renewal
|
||||||
|
- Mounted buckets at startup via batch script (future systemd equivalents to be implemented)
|
||||||
|
- Snapshot-based rollback in ZFS enabled post-deployment resilience
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🩹 What Went Weird (and We Fixed It)
|
||||||
|
|
||||||
|
- SPL froze during initial `mc mirror` attempts — solution: switched to `rclone`, which performed exponentially faster
|
||||||
|
- Some hiccups during early cache tuning, including sparse file support issues — solved by switching to ZFS
|
||||||
|
- Missing media files in Mastodon were traced to uploads during sync; resolved with staged sync + retry before final switch
|
||||||
|
- Certbot automation wasn’t configured — resolved with a systemd timer that stops nginx, renews, and restarts nginx automatically
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧯 What We Learned
|
||||||
|
|
||||||
|
- MinIO is solid, but **rclone wins for bulk sync performance**
|
||||||
|
- VFS cache settings **make or break** media-heavy workloads like SPL
|
||||||
|
- ZFS is a game-changer: no sparse file errors, reliable snapshots, clean rollback
|
||||||
|
- Planning matters: pre-syncing from backup avoided downtime
|
||||||
|
- Not touching prod until ready keeps stress and screwups to a minimum
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📦 Next Steps
|
||||||
|
|
||||||
|
- [ ] Clean `genesisassets-secure` of misplaced show files
|
||||||
|
- [ ] Sync `azuracast` from live system (no backup copy yet)
|
||||||
|
- [ ] Build automated snapshot send-to-backup workflow (`zfs send | ssh backup zfs recv`)
|
||||||
|
- [ ] Stage full failover simulation (optional but fun)
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Save it as a Markdown file
|
||||||
|
file_path = "/mnt/data/genesis_radio_migration_postmortem.md"
|
||||||
|
with open(file_path, "w") as f:
|
||||||
|
f.write(post_mortem_content)
|
||||||
|
|
||||||
|
file_path
|
||||||
75
postmortem/mastoobjectstorage.md
Normal file
75
postmortem/mastoobjectstorage.md
Normal file
@ -0,0 +1,75 @@
|
|||||||
|
# 🧾 Postmortem: Mastodon Object Storage Migration to Secure S3 (MinIO)
|
||||||
|
**Date:** April 30, 2025
|
||||||
|
**Engineer:** Doc (Genesis Radio / Genesis Hosting)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎯 Objective
|
||||||
|
|
||||||
|
Migrate Mastodon's object storage from an older MinIO bucket (`linodeassets`) to a new **ZFS-backed, encrypted** MinIO instance (`mastodonassets-secure`) on `shredderv2`, while maintaining uptime and improving storage performance and security.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧱 Infrastructure Touched
|
||||||
|
|
||||||
|
- Mastodon (Docker-based, hosted on Linode)
|
||||||
|
- MinIO S3 Object Storage (`oldminio` → `secureminio`)
|
||||||
|
- Nginx (reverse proxy for Console + S3 endpoints)
|
||||||
|
- ZFS pool: `nexus/mastodonassets`
|
||||||
|
- Domains:
|
||||||
|
- `shredderv2.sshjunkie.com` (S3 API)
|
||||||
|
- `consolev2.sshjunkie.com` (MinIO Console UI)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚠️ Issues Encountered
|
||||||
|
|
||||||
|
1. **403 Access Denied on Mastodon startup**
|
||||||
|
- ✅ Root cause: `genesisadminv2` MinIO user had no attached policy
|
||||||
|
- 🔧 Fixed via Console UI after re-enabling access
|
||||||
|
|
||||||
|
2. **MinIO Console unreachable (`consolev2.sshjunkie.com`)**
|
||||||
|
- SSL cert for the domain was missing
|
||||||
|
- 🔧 Used `certbot certonly --standalone` to issue new cert, re-enabled full HTTPS proxy
|
||||||
|
|
||||||
|
3. **Sync race conditions**
|
||||||
|
- Some media files were uploaded to the old bucket during the long transfer
|
||||||
|
- 🔧 Mitigated by running an additional `rclone sync` pass before cutover
|
||||||
|
|
||||||
|
4. **Rclone performance bottlenecks**
|
||||||
|
- MinIO client (`mc mirror`) was too slow
|
||||||
|
- ✅ Switched to `rclone`, saw drastic speed improvement
|
||||||
|
|
||||||
|
5. **SPL (StationPlaylist) freezing during asset access**
|
||||||
|
- Root cause: cache choking on sparse file writes under ext4
|
||||||
|
- ✅ Fix: moved critical rclone mounts to ZFS-backed drives
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ Success Criteria Met
|
||||||
|
|
||||||
|
- 🔒 All Mastodon assets are now stored in `mastodonassets-secure` with encryption
|
||||||
|
- 🪣 MinIO Console functional on `https://consolev2.sshjunkie.com`
|
||||||
|
- 🎯 Mastodon is running with zero visible user impact
|
||||||
|
- 💾 Snapshot (`nexus/mastodonassets@pre-s3-switch`) taken post-migration for rollback
|
||||||
|
- 🔁 Future syncs can now be performed cleanly from backup server instead of live system
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧠 Lessons Learned
|
||||||
|
|
||||||
|
- Always validate MinIO user policies before go-live
|
||||||
|
- Avoid redirects in `server_name` blocks during cert issuance
|
||||||
|
- ZFS dramatically improves caching performance with rclone VFS
|
||||||
|
- Post-cutover syncs are crucial for active upload systems like Mastodon
|
||||||
|
- UI access to MinIO is a lifesaver for emergency fixes — keep it working
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔚 Follow-Up Actions
|
||||||
|
|
||||||
|
- [ ] Schedule `certbot renew --standalone` with systemd timer
|
||||||
|
- [ ] Rotate MinIO user keys and audit access policies
|
||||||
|
- [ ] Monitor `/var/log/syslog` for VFS or sparse file errors
|
||||||
|
- [ ] Document your rclone mount and caching strategy for SPL and Mastodon
|
||||||
|
|
||||||
Loading…
x
Reference in New Issue
Block a user