Auto-commit from giteapush.sh at 2025-04-30 08:47:49
This commit is contained in:
parent
d6bcc06832
commit
c1127be692
23
infra_morale.log
Normal file
23
infra_morale.log
Normal file
@ -0,0 +1,23 @@
|
|||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
|
/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
|
||||||
File diff suppressed because it is too large
Load Diff
@ -684,3 +684,283 @@ paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
|
|||||||
⚡ 1 warnings found:
|
⚡ 1 warnings found:
|
||||||
- 💥 [db2] WARNING: Replication lag is 73 seconds.
|
- 💥 [db2] WARNING: Replication lag is 73 seconds.
|
||||||
✅ Genesis Radio Healthcheck 2025-04-29 09:45:14: All systems normal.
|
✅ Genesis Radio Healthcheck 2025-04-29 09:45:14: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 10:00:13: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 10:15:13: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 10:30:15: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 10:45:29: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 11:00:11: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 11:15:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 11:30:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 11:45:16: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 12:00:12: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 12:15:17 ⚠️
|
||||||
|
⚡ 2 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 12:30:17: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 12:45:16: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 13:00:13: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 13:15:13: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 13:30:14: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 13:45:14: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 14:00:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 14:15:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 14:30:11: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 14:45:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 15:00:16: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 15:16:51 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 15:32:34 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 15:45:24: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 16:00:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 16:15:21: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 16:30:27: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 16:45:20: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 17:00:20: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 17:15:18 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 17:30:18 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 17:45:16: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 18:00:17: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 18:15:27 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 18:30:24: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 18:45:22 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 19:00:17: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 19:15:19: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 19:30:11: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 19:45:11 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- 💥 [db2] WARNING: Replication lag is 68 seconds.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 20:00:17: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 20:15:21: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 20:30:16: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 20:45:10: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 21:00:14 ⚠️
|
||||||
|
⚡ 56 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 21:15:13 ⚠️
|
||||||
|
⚡ 3 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 21:30:26 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 21:45:28: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 22:00:26 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 22:15:28: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 22:30:22 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 22:45:26: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 23:00:23: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 23:15:33: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 23:30:23: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-29 23:45:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 00:00:23: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 00:15:24 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 00:30:16: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 00:45:23: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 01:00:20: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 01:15:17 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 01:30:19: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 01:45:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 02:00:21: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 02:15:25: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 02:30:21 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 02:45:13 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- 💥 [db2] WARNING: Replication lag is 60 seconds.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 03:00:15: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 03:15:13 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- 💥 [db2] WARNING: Replication lag is 61 seconds.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 03:30:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 03:45:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 04:00:11: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 04:15:20: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 04:30:18: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 04:45:21: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 05:00:19: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 05:15:21 ⚠️
|
||||||
|
⚡ 24 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 05:30:19: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 05:45:17 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 06:00:17: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 06:15:28 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 06:30:13: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 06:45:19 ⚠️
|
||||||
|
⚡ 5 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 07:00:13 ⚠️
|
||||||
|
⚡ 1 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 07:15:10: All systems normal.
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 07:30:12 ⚠️
|
||||||
|
⚡ 2 warnings found:
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 07:45:13: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 08:00:11: All systems normal.
|
||||||
|
Exception (client): Error reading SSH protocol banner[Errno 104] Connection reset by peer
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2369, in _check_banner
|
||||||
|
buf = self.packetizer.readline(timeout)
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/packet.py", line 395, in readline
|
||||||
|
buf += self._read_timeout(timeout)
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/packet.py", line 663, in _read_timeout
|
||||||
|
x = self.__socket.recv(128)
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
ConnectionResetError: [Errno 104] Connection reset by peer
|
||||||
|
|
||||||
|
During handling of the above exception, another exception occurred:
|
||||||
|
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2185, in run
|
||||||
|
self._check_banner()
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2373, in _check_banner
|
||||||
|
raise SSHException(
|
||||||
|
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner[Errno 104] Connection reset by peer
|
||||||
|
|
||||||
|
Exception (client): Error reading SSH protocol banner
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2369, in _check_banner
|
||||||
|
buf = self.packetizer.readline(timeout)
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/packet.py", line 395, in readline
|
||||||
|
buf += self._read_timeout(timeout)
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/packet.py", line 665, in _read_timeout
|
||||||
|
raise EOFError()
|
||||||
|
EOFError
|
||||||
|
|
||||||
|
During handling of the above exception, another exception occurred:
|
||||||
|
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2185, in run
|
||||||
|
self._check_banner()
|
||||||
|
File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/transport.py", line 2373, in _check_banner
|
||||||
|
raise SSHException(
|
||||||
|
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
|
||||||
|
|
||||||
|
⚠️ Genesis Radio Warning Healthcheck 2025-04-30 08:15:12 ⚠️
|
||||||
|
⚡ 10 warnings found:
|
||||||
|
- ⚠️ [shredder] ERROR: Could not read log /var/log/nginx/error.log: Error reading SSH protocol banner[Errno 104] Connection reset by peer
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 08:30:12: All systems normal.
|
||||||
|
✅ Genesis Radio Healthcheck 2025-04-30 08:45:18: All systems normal.
|
||||||
|
|||||||
75
postmortem/mastoobjectstorage.md
Normal file
75
postmortem/mastoobjectstorage.md
Normal file
@ -0,0 +1,75 @@
|
|||||||
|
# 🧾 Postmortem: Mastodon Object Storage Migration to Secure S3 (MinIO)
|
||||||
|
**Date:** April 30, 2025
|
||||||
|
**Engineer:** Doc (Genesis Radio / Genesis Hosting)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎯 Objective
|
||||||
|
|
||||||
|
Migrate Mastodon's object storage from an older MinIO bucket (`linodeassets`) to a new **ZFS-backed, encrypted** MinIO instance (`mastodonassets-secure`) on `shredderv2`, while maintaining uptime and improving storage performance and security.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧱 Infrastructure Touched
|
||||||
|
|
||||||
|
- Mastodon (Docker-based, hosted on Linode)
|
||||||
|
- MinIO S3 Object Storage (`oldminio` → `secureminio`)
|
||||||
|
- Nginx (reverse proxy for Console + S3 endpoints)
|
||||||
|
- ZFS pool: `nexus/mastodonassets`
|
||||||
|
- Domains:
|
||||||
|
- `shredderv2.sshjunkie.com` (S3 API)
|
||||||
|
- `consolev2.sshjunkie.com` (MinIO Console UI)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚠️ Issues Encountered
|
||||||
|
|
||||||
|
1. **403 Access Denied on Mastodon startup**
|
||||||
|
- ✅ Root cause: `genesisadminv2` MinIO user had no attached policy
|
||||||
|
- 🔧 Fixed via Console UI after re-enabling access
|
||||||
|
|
||||||
|
2. **MinIO Console unreachable (`consolev2.sshjunkie.com`)**
|
||||||
|
- SSL cert for the domain was missing
|
||||||
|
- 🔧 Used `certbot certonly --standalone` to issue new cert, re-enabled full HTTPS proxy
|
||||||
|
|
||||||
|
3. **Sync race conditions**
|
||||||
|
- Some media files were uploaded to the old bucket during the long transfer
|
||||||
|
- 🔧 Mitigated by running an additional `rclone sync` pass before cutover
|
||||||
|
|
||||||
|
4. **Rclone performance bottlenecks**
|
||||||
|
- MinIO client (`mc mirror`) was too slow
|
||||||
|
- ✅ Switched to `rclone`, saw drastic speed improvement
|
||||||
|
|
||||||
|
5. **SPL (StationPlaylist) freezing during asset access**
|
||||||
|
- Root cause: cache choking on sparse file writes under ext4
|
||||||
|
- ✅ Fix: moved critical rclone mounts to ZFS-backed drives
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ Success Criteria Met
|
||||||
|
|
||||||
|
- 🔒 All Mastodon assets are now stored in `mastodonassets-secure` with encryption
|
||||||
|
- 🪣 MinIO Console functional on `https://consolev2.sshjunkie.com`
|
||||||
|
- 🎯 Mastodon is running with zero visible user impact
|
||||||
|
- 💾 Snapshot (`nexus/mastodonassets@pre-s3-switch`) taken post-migration for rollback
|
||||||
|
- 🔁 Future syncs can now be performed cleanly from backup server instead of live system
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧠 Lessons Learned
|
||||||
|
|
||||||
|
- Always validate MinIO user policies before go-live
|
||||||
|
- Avoid redirects in `server_name` blocks during cert issuance
|
||||||
|
- ZFS dramatically improves caching performance with rclone VFS
|
||||||
|
- Post-cutover syncs are crucial for active upload systems like Mastodon
|
||||||
|
- UI access to MinIO is a lifesaver for emergency fixes — keep it working
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔚 Follow-Up Actions
|
||||||
|
|
||||||
|
- [ ] Schedule `certbot renew --standalone` with systemd timer
|
||||||
|
- [ ] Rotate MinIO user keys and audit access policies
|
||||||
|
- [ ] Monitor `/var/log/syslog` for VFS or sparse file errors
|
||||||
|
- [ ] Document your rclone mount and caching strategy for SPL and Mastodon
|
||||||
|
|
||||||
Loading…
x
Reference in New Issue
Block a user