bin-there-done-that/miscellaneous/dbcheck.log

🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 19:15:17 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 103 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 19:30:17 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 135 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 19:45:17 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 24 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 20:00:19 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 20:15:33 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 109 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 20:30:21 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 46 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 20:45:29 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 21:00:18 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 21:15:21 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 38 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 21:30:36 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 21 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 21:45:32 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 22 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 22:00:30 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 30 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 22:15:31 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 14 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 22:30:24 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 55 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 22:45:37 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 23:00:21 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 21 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 23:15:21 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 19 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 23:30:16 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 68 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-26 23:45:16 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 77 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 00:00:18 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 18 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 00:15:16 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 14 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 00:30:16 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 82 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 00:45:17 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 135 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 01:00:14 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 14 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 01:15:15 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 127 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 01:30:14 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 100 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 01:45:14 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 35 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 02:00:14 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 13 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 02:15:14 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 11 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 02:30:17 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 02:45:16 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 27 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 03:00:16 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 03:15:19 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 137 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 03:30:15 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 134 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 03:45:32 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 51 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 04:00:19 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 19 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 04:15:26 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 04:30:28 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 67 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 04:45:24 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 05:00:31 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 29 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 05:15:30 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 66 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 05:30:30 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 37 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 05:45:23 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 21 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 06:00:26 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 26 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 06:15:37 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 20 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 06:30:35 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 13 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 06:45:32 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 49 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 07:00:31 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 11 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 07:18:00 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 29 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 07:30:33 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 11 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 07:45:33 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 16 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 08:00:39 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 08:15:39 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 08:30:31 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 62 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 08:45:37 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 13 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 09:00:21 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 21 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 09:15:21 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 09:30:21 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 09:45:21 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 20 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 10:00:25 🚨
[db1] ERROR: SMART status unknown on /dev/sda. Output: 
[db2] WARNING: Replication lag is 17 seconds.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 13:15:16 🚨
[db2] WARNING: Replication lag is 62 seconds.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 14:15:18 🚨
[db2] WARNING: Replication lag is 63 seconds.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 15:15:19 🚨
[db2] WARNING: Replication lag is 69 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 15:30:19 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db2] WARNING: Replication lag is 105 seconds.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 16:15:18 🚨
[db2] WARNING: Replication lag is 135 seconds.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 16:45:19 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
Traceback (most recent call last):
  File "/home/doc/genesis-tools/miscellaneous/dbcheck1.py", line 257, in <module>
    main()
  File "/home/doc/genesis-tools/miscellaneous/dbcheck1.py", line 218, in main
    res = check_remote_disk(node["host"], node["ssh_user"], disk, node["name"])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/doc/genesis-tools/miscellaneous/dbcheck1.py", line 85, in check_remote_disk
    out = ssh_command(host, user, cmd)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/doc/genesis-tools/miscellaneous/dbcheck1.py", line 76, in ssh_command
    ssh.connect(hostname=host, username=user, timeout=10)
  File "/home/doc/dbcheck/lib/python3.12/site-packages/paramiko/client.py", line 386, in connect
    sock.connect(addr)
TimeoutError: timed out
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 19:30:19 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db1] CRITICAL: Service postgresql@16-main.service not running!
[db1] ERROR: Could not check replication: connection to server at "cluster.db1.genesishostingtechnologies.com" (38.102.127.174), port 5432 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?

[db2] WARNING: Replication lag is 90 seconds.
Failed to send Mastodon DM: <!DOCTYPE html>
<html lang='en'>
<head>
<meta content='text/html; charset=UTF-8' http-equiv='Content-Type'>
<meta charset='utf-8'>
<title>This page is not correct
 - Mastodon Glitch Edition</title>
<meta content='width=device-width,initial-scale=1' name='viewport'>
<link rel="stylesheet" crossorigin="anonymous" href="/packs/css/skins/glitch/mastodon-light-ba448ef4.chunk.css" media="not all and (prefers-color-scheme: dark)" integrity="sha256-P4BQPI06vGtRMsuyy44qvSrh1psmbDTSDhcI2ZZsDcI=" /><link rel="stylesheet" crossorigin="anonymous" href="/packs/css/skins/glitch/default-b4389951.chunk.css" media="(prefers-color-scheme: dark)" integrity="sha256-Ve2Dw/Y6VxR48zbYZoB8JdFFHsH9QiYGIZs5DLuH/U4=" />
<script src="/packs/js/common-8028e752d0f2aa93a5d6.js" crossorigin="anonymous" integrity="sha256-Yu9sY5Ca6EUBu337KcDfy59RIzPAUoLndPw2vudg2pg="></script>
<script src="/packs/js/flavours/glitch/error-10438b0fe55aa68f54dd.chunk.js" crossorigin="anonymous" integrity="sha256-DL1chTim8F/pYKgN8OlM4xXp5561mQ8KKtSZsRT9bwA="></script>
</head>
<body class='error'>
<div class='dialog'>
<div class='dialog__illustration'>
<img alt='Mastodon Glitch Edition' src='/oops.png'>
</div>
<div class='dialog__message'>
<h1>We&#39;re sorry, but something went wrong on our end.
</h1>
</div>
</div>
</body>
</html>

OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 20:30:16 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 20:45:16 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 21:00:17 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 21:15:17 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 21:30:15 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
[db2] WARNING: Replication lag is 66 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 21:45:19 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
[db2] WARNING: Replication lag is 106 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 22:00:15 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 22:15:32 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
[db2] WARNING: Replication lag is 95 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 22:30:30 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
[db2] WARNING: Replication lag is 79 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 22:45:38 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 23:00:28 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 23:15:47 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 23:30:29 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-27 23:45:34 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 00:00:23 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 00:15:15 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 00:30:16 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 00:45:17 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
[db2] WARNING: Replication lag is 77 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 01:00:15 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 01:15:15 🚨
[shredder] WARNING: Pattern 'FATAL' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 01:30:16 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db2] WARNING: Replication lag is 72 seconds.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 02:30:18 🚨
[db2] WARNING: Replication lag is 89 seconds.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 03:00:16 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 03:15:18 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 03:30:17 🚨
[db2] WARNING: Replication lag is 135 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 03:45:17 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 04:00:16 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 04:45:21 🚨
[db2] WARNING: Replication lag is 64 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 05:00:27 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 08:00:20 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 08:15:20 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 09:45:24 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 10:00:18 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 10:30:16 🚨
[db2] WARNING: Replication lag is 75 seconds.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 12:00:31 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 12:45:24 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
[db2] WARNING: Replication lag is 94 seconds.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 13:00:33 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
OK: All checks passed.
🚨 Genesis Radio Multi-Node Healthcheck 2025-04-28 14:15:19 🚨
[mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
OK: All checks passed.
  File "/home/doc/genesis-tools/miscellaneous/dbcheck1.py", line 181
    f.write("<table border='1' cellpadding='5' style='border-collapse: collapse;'>
            ^
SyntaxError: unterminated string literal (detected at line 181)
Traceback (most recent call last):
  File "/home/doc/genesis-tools/miscellaneous/dbcheck1.py", line 187, in <module>
    main()
  File "/home/doc/genesis-tools/miscellaneous/dbcheck1.py", line 113, in main
    res = check_remote_disk(node["host"], node["ssh_user"], disk, node["name"])
          ^^^^^^^^^^^^^^^^^
NameError: name 'check_remote_disk' is not defined. Did you mean: 'check_remote_logs'?
⚠️ Genesis Radio Warning Healthcheck 2025-04-28 15:15:11 ⚠️
⚡ 1 warnings found:
- 💥 [db2] WARNING: Replication lag is 105 seconds.
✅ Genesis Radio Healthcheck 2025-04-28 15:30:13: All systems normal.
⚠️ Genesis Radio Warning Healthcheck 2025-04-28 15:45:13 ⚠️
⚡ 1 warnings found:
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
✅ Genesis Radio Healthcheck 2025-04-28 16:00:12: All systems normal.
⚠️ Genesis Radio Warning Healthcheck 2025-04-28 16:15:11 ⚠️
⚡ 3 warnings found:
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
✅ Genesis Radio Healthcheck 2025-04-28 16:30:11: All systems normal.
⚠️ Genesis Radio Warning Healthcheck 2025-04-28 16:45:11 ⚠️
⚡ 1 warnings found:
- 💥 [db2] WARNING: Replication lag is 88 seconds.
✅ Genesis Radio Healthcheck 2025-04-28 17:00:10: All systems normal.
✅ Genesis Radio Healthcheck 2025-04-28 17:15:13: All systems normal.
⚠️ Genesis Radio Warning Healthcheck 2025-04-28 17:30:14 ⚠️
⚡ 2 warnings found:
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
⚠️ Genesis Radio Warning Healthcheck 2025-04-28 17:45:15 ⚠️
⚡ 1 warnings found:
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
✅ Genesis Radio Healthcheck 2025-04-28 18:00:13: All systems normal.
✅ Genesis Radio Healthcheck 2025-04-28 18:15:12: All systems normal.
✅ Genesis Radio Healthcheck 2025-04-28 18:30:11: All systems normal.
✅ Genesis Radio Healthcheck 2025-04-28 18:45:16: All systems normal.
✅ Genesis Radio Healthcheck 2025-04-28 19:00:15: All systems normal.
⚠️ Genesis Radio Warning Healthcheck 2025-04-28 19:15:12 ⚠️
⚡ 1 warnings found:
- 💥 [db2] WARNING: Replication lag is 101 seconds.
✅ Genesis Radio Healthcheck 2025-04-28 19:30:12: All systems normal.
✅ Genesis Radio Healthcheck 2025-04-28 19:45:14: All systems normal.
✅ Genesis Radio Healthcheck 2025-04-28 20:00:14: All systems normal.
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
/home/doc/dbcheck/bin/python: can't open file '/home/doc/genesis-tools/miscellaneous/dbcheck1.py': [Errno 2] No such file or directory
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 04:00:12 ⚠️
⚡ 1 warnings found:
- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 04:15:11 ⚠️
⚡ 1 warnings found:
- 💥 [db2] WARNING: Replication lag is 76 seconds.
⚠️ Genesis Radio Warning Healthcheck 2025-04-29 04:30:12 ⚠️
⚡ 1 warnings found:
- 💥 [db2] WARNING: Replication lag is 95 seconds.
✅ Genesis Radio Healthcheck 2025-04-29 04:45:14: All systems normal.