Auto-commit from giteapush.sh at 2025-05-01 10:32:16

cheatsheets/rclone_cheat_sheet.md

@@ -0,0 +1,133 @@
+# 📘 Rclone Command Cheat Sheet
+
+## ⚙️ Configuration
+
+### Launch Configuration Wizard
+```bash
+rclone config
+```
+
+### Show Current Config
+```bash
+rclone config show
+```
+
+### List Remotes
+```bash
+rclone listremotes
+```
+
+## 📁 Basic File Operations
+
+### Copy Files
+```bash
+rclone copy source:path dest:path
+```
+
+### Sync Files
+```bash
+rclone sync source:path dest:path
+```
+
+### Move Files
+```bash
+rclone move source:path dest:path
+```
+
+### Delete Files or Dirs
+```bash
+rclone delete remote:path
+rclone purge remote:path   # Delete entire path
+```
+
+### Check Differences
+```bash
+rclone check source:path dest:path
+```
+
+## 🔍 Listing and Info
+
+### List Directory
+```bash
+rclone ls remote:path
+rclone lsd remote:path     # List only directories
+rclone lsl remote:path     # Long list with size and modification time
+```
+
+### Tree View
+```bash
+rclone tree remote:path
+```
+
+### File Size and Count
+```bash
+rclone size remote:path
+```
+
+## 📦 Mounting
+
+### Mount Remote (Linux/macOS)
+```bash
+rclone mount remote:path /mnt/mountpoint
+```
+
+### Mount with Aggressive Caching (Windows)
+```bash
+rclone mount remote:path X: \
+  --vfs-cache-mode full \
+  --cache-dir C:\path\to\cache \
+  --vfs-cache-max-size 100G \
+  --vfs-read-chunk-size 512M \
+  --vfs-read-ahead 1G
+```
+
+## 🔁 Sync with Filtering
+
+### Include / Exclude Files
+```bash
+rclone sync source:path dest:path --exclude "*.tmp"
+rclone sync source:path dest:path --include "*.jpg"
+```
+
+## 📄 Logging and Dry Runs
+
+### Verbose and Dry Run
+```bash
+rclone sync source:path dest:path -v --dry-run
+```
+
+### Log to File
+```bash
+rclone sync source:path dest:path --log-file=rclone.log -v
+```
+
+## 📡 Remote Control (RC)
+
+### Start RC Server
+```bash
+rclone rcd --rc-web-gui
+```
+
+### Use RC Command
+```bash
+rclone rc core/stats
+rclone rc vfs/stats
+```
+
+## 🛠️ Miscellaneous
+
+### Serve Over HTTP/WebDAV/SFTP
+```bash
+rclone serve http remote:path
+rclone serve webdav remote:path
+rclone serve sftp remote:path
+```
+
+### Crypt Operations
+```bash
+rclone config create secure crypt remote:path
+```
+
+---
+
+> ✅ **Tip**: Always use `--dry-run` when testing `sync`, `move`, or `delete` to prevent accidental data loss.

cheatsheets/server_hardening_disaster_recovery.md

@@ -0,0 +1,87 @@
+# 🛡️ Server Hardening & Disaster Recovery Cheat Sheet
+
+## 🔐 Server Hardening Checklist
+
+### 🔒 OS & User Security
+- ✅ Use **key-based SSH authentication** (`~/.ssh/authorized_keys`)
+- ✅ Disable root login:  
+  ```bash
+  sudo sed -i 's/^PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
+  sudo systemctl restart sshd
+  ```
+- ✅ Change default SSH port and rate-limit with Fail2Ban or UFW
+- ✅ Set strong password policies:
+  ```bash
+  sudo apt install libpam-pwquality
+  sudo nano /etc/security/pwquality.conf
+  ```
+- ✅ Lock down `/etc/sudoers`, remove unnecessary sudo privileges
+
+### 🔧 Kernel & System Hardening
+- ✅ Install and configure `ufw` or `iptables`:
+  ```bash
+  sudo ufw default deny incoming
+  sudo ufw allow ssh
+  sudo ufw enable
+  ```
+- ✅ Disable unused filesystems:
+  ```bash
+  echo "install cramfs /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
+  ```
+- ✅ Set kernel parameters:
+  ```bash
+  sudo nano /etc/sysctl.d/99-sysctl.conf
+  # Example: net.ipv4.ip_forward = 0
+  sudo sysctl -p
+  ```
+
+### 🧾 Logging & Monitoring
+- ✅ Enable and configure `auditd`:
+  ```bash
+  sudo apt install auditd audispd-plugins
+  sudo systemctl enable auditd
+  ```
+- ✅ Centralize logs using `rsyslog`, `logrotate`, or Fluentbit
+- ✅ Use `fail2ban`, `CrowdSec`, or `Wazuh` for intrusion detection
+
+## 💾 Disaster Recovery Checklist
+
+### 📦 Backups
+- ✅ Automate **daily database dumps** (e.g., `pg_dump`, `mysqldump`)
+- ✅ Use **ZFS snapshots** for versioned backups
+- ✅ Sync offsite via `rclone`, `rsync`, or cloud storage
+- ✅ Encrypt backups using `gpg` or `age`
+
+### 🔁 Testing & Recovery
+- ✅ **Verify backup integrity** regularly:
+  ```bash
+  gpg --verify backup.sql.gpg
+  pg_restore --list backup.dump
+  ```
+- ✅ Practice **bare-metal restores** in a test environment
+- ✅ Use **PITR** (Point-In-Time Recovery) for PostgreSQL
+
+### 🛑 Emergency Scripts
+- ✅ Create service restart scripts:
+  ```bash
+  systemctl restart mastodon
+  docker restart azuracast
+  ```
+- ✅ Pre-stage `rescue.sh` to rebuild key systems
+- ✅ Include Mastodon/Gitea/etc. reconfig tools
+
+### 🗂️ Documentation
+- ✅ Maintain a **runbook** with:
+  - Service recovery steps
+  - IPs, ports, login methods
+  - Admin contacts and escalation
+
+### 🧪 Chaos Testing
+- ✅ Simulate failure of:
+  - A disk or volume (use `zpool offline`)
+  - A network link (`iptables -A OUTPUT ...`)
+  - A database node (use Patroni/pg_auto_failover tools)
+
+---
+
+> ✅ **Pro Tip**: Integrate all hardening and backup tasks into your Ansible playbooks for consistency and redeployability.

cheatsheets/zfs_cheat_sheet.md

@@ -0,0 +1,153 @@
+# 📘 ZFS Command Cheat Sheet
+
+## 🛠️ Pool Management
+
+### Create a Pool
+```bash
+zpool create <poolname> <device>
+zpool create <poolname> mirror <dev1> <dev2>
+zpool create <poolname> raidz1 <dev1> <dev2> <dev3> ...
+```
+
+### List Pools
+```bash
+zpool list
+```
+
+### Destroy a Pool
+```bash
+zpool destroy <poolname>
+```
+
+### Add Devices to a Pool
+```bash
+zpool add <poolname> <device>
+```
+
+### Export / Import Pool
+```bash
+zpool export <poolname>
+zpool import <poolname>
+zpool import -d /dev/disk/by-id <poolname>
+```
+
+## 🔍 Pool Status and Health
+
+### Check Pool Status
+```bash
+zpool status
+zpool status -v
+```
+
+### Scrub a Pool
+```bash
+zpool scrub <poolname>
+```
+
+### Clear Errors
+```bash
+zpool clear <poolname>
+```
+
+## 🧱 Dataset Management
+
+### Create a Dataset
+```bash
+zfs create <poolname>/<dataset>
+```
+
+### List Datasets
+```bash
+zfs list
+zfs list -t all
+```
+
+### Destroy a Dataset
+```bash
+zfs destroy <poolname>/<dataset>
+```
+
+## 📦 Mounting and Properties
+
+### Set Mount Point
+```bash
+zfs set mountpoint=/your/path <poolname>/<dataset>
+```
+
+### Mount / Unmount
+```bash
+zfs mount <dataset>
+zfs unmount <dataset>
+```
+
+### Auto Mount
+```bash
+zfs set canmount=on|off|noauto <dataset>
+```
+
+## 📝 Snapshots & Clones
+
+### Create a Snapshot
+```bash
+zfs snapshot <poolname>/<dataset>@<snapshotname>
+```
+
+### List Snapshots
+```bash
+zfs list -t snapshot
+```
+
+### Roll Back to Snapshot
+```bash
+zfs rollback <poolname>/<dataset>@<snapshotname>
+```
+
+### Destroy a Snapshot
+```bash
+zfs destroy <poolname>/<dataset>@<snapshotname>
+```
+
+### Clone a Snapshot
+```bash
+zfs clone <poolname>/<dataset>@<snapshot> <poolname>/<new-dataset>
+```
+
+## 🔁 Sending & Receiving
+
+### Send Snapshot to File or Pipe
+```bash
+zfs send <snapshot> > file
+zfs send -R <snapshot> | zfs receive <pool>/<dataset>
+```
+
+### Receive Snapshot
+```bash
+zfs receive <pool>/<dataset>
+```
+
+## 🧮 Useful Info & Tuning
+
+### Check Available Space
+```bash
+zfs list
+```
+
+### Set Quota or Reservation
+```bash
+zfs set quota=10G <dataset>
+zfs set reservation=5G <dataset>
+```
+
+### Enable Compression
+```bash
+zfs set compression=lz4 <dataset>
+```
+
+### Enable Deduplication (use cautiously)
+```bash
+zfs set dedup=on <dataset>
+```
+
+---
+
+> ✅ **Tip**: Always test ZFS commands in a safe environment before using them on production systems!

infra_morale.log

@@ -1 +1,3 @@
 /bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
+/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied
+/bin/sh: 1: /home/doc/genesis-tools/infra_morale/fake_status_bot.rb: Permission denied

infra_morale/.env

@@ -1,2 +1,2 @@
 MASTODON_BASE_URL=https://chatwithus.live
-MASTODON_TOKEN=Txh2DlBI7hgly8e7zsu0Pee2ONJcFAxMpbyiXTvsZKw
+MASTODON_TOKEN=07w3Emdw-cv_TncysrNU8Ed_sHJhwtnvKmnLqKlHmKA

masto_restart.log

@@ -0,0 +1,41 @@
+[2025-05-01 07:52:55] === Mastodon Restart Initiated ===
+[*] Posting 2-minute warning to Mastodon...
+[✓] 2-minute warning posted.
+[*] Posting 1-minute warning to Mastodon...
+[✓] 1-minute warning posted.
+[*] Connecting to root@chatwithus.live to restart Mastodon services...
+Restarting mastodon-web...
+Restarting mastodon-sidekiq...
+Restarting mastodon-streaming...
+All services restarted.
+[*] Waiting for Mastodon to come back online...
+[+] Mastodon is back online.
+[*] Posting final status to Mastodon...
+[✓] Final status posted.
+[*] Sending Telegram alert...
+[✓] Telegram alert sent.
+[✓] All tasks complete. Logged out of root@chatwithus.live.
+[2025-05-01 07:52:55] === Mastodon Restart Complete ===
+
+[2025-05-01 08:03:21] === Mastodon Restart Initiated ===
+[*] Posting 2-minute warning to Mastodon...
+[✓] 2-minute warning posted.
+[2025-05-01 08:03:47] === Mastodon Restart Initiated ===
+[*] Posting 2-minute warning to Mastodon...
+[✓] 2-minute warning posted.
+[*] Posting 1-minute warning to Mastodon...
+[✓] 1-minute warning posted.
+[*] Connecting to root@chatwithus.live to restart Mastodon services...
+Restarting mastodon-web...
+Restarting mastodon-sidekiq...
+Restarting mastodon-streaming...
+All services restarted.
+[*] Waiting for Mastodon to come back online...
+[+] Mastodon is back online.
+[*] Posting final status to Mastodon...
+[✓] Final status posted.
+[*] Sending Telegram alert...
+[✓] Telegram alert sent.
+[✓] All tasks complete. Logged out of root@chatwithus.live.
+[2025-05-01 08:03:47] === Mastodon Restart Complete ===
+

miscellaneous/bash/mastodon_restart.sh

@@ -0,0 +1,87 @@
+#!/bin/bash
+
+# === CONFIG ===
+SERVER="root@chatwithus.live"
+MASTODON_INSTANCE="https://chatwithus.live"
+ACCESS_TOKEN="07w3Emdw-cv_TncysrNU8Ed_sHJhwtnvKmnLqKlHmKA"
+
+TOOT_VISIBILITY="public"
+
+WARNING_TOOT_2M="🚨 Heads up! We’ll be restarting ChatWithUs.Live in about 2 minutes to perform routine maintenance and keep things running smoothly. Please wrap up anything important and hang tight — we’ll be right back."
+WARNING_TOOT_1M="⚠️ Just one more minute until we restart the server. If you’re in the middle of something, now’s the time to save and log out. Thanks for your patience while we keep the gears turning!"
+
+FINAL_TOOT="✅ ChatWithUs.Live services restarted from Krang via OPS script."
+
+TELEGRAM_BOT_TOKEN="8178867489:AAH0VjN7VnZSCIWasSz_y97iBLLjPJA751k"
+TELEGRAM_CHAT_ID="1559582356"
+TELEGRAM_TEXT="✅ Mastodon has been restarted by Krang. All services are back online."
+
+LOG_FILE="/home/doc/genesis-tools/masto_restart.log"
+TIMESTAMP=$(date "+%Y-%m-%d %H:%M:%S")
+
+{
+echo "[$TIMESTAMP] === Mastodon Restart Initiated ==="
+
+# === Post 2-Minute Warning Toot ===
+echo "[*] Posting 2-minute warning to Mastodon..."
+curl -s -X POST "$MASTODON_INSTANCE/api/v1/statuses" \
+  -H "Authorization: Bearer $ACCESS_TOKEN" \
+  -d "status=$WARNING_TOOT_2M" \
+  -d "visibility=$TOOT_VISIBILITY" > /dev/null && echo "[✓] 2-minute warning posted."
+
+# === Wait 1 minute ===
+sleep 60
+
+# === Post 1-Minute Warning Toot ===
+echo "[*] Posting 1-minute warning to Mastodon..."
+curl -s -X POST "$MASTODON_INSTANCE/api/v1/statuses" \
+  -H "Authorization: Bearer $ACCESS_TOKEN" \
+  -d "status=$WARNING_TOOT_1M" \
+  -d "visibility=$TOOT_VISIBILITY" > /dev/null && echo "[✓] 1-minute warning posted."
+
+# === Wait 1 more minute ===
+sleep 60
+
+# === Restart Mastodon Services ===
+echo "[*] Connecting to $SERVER to restart Mastodon services..."
+
+ssh "$SERVER" bash << 'EOF'
+echo "Restarting mastodon-web..."
+systemctl restart mastodon-web
+
+echo "Restarting mastodon-sidekiq..."
+systemctl restart mastodon-sidekiq
+
+echo "Restarting mastodon-streaming..."
+systemctl restart mastodon-streaming
+
+echo "All services restarted."
+EOF
+
+# === Wait Until Mastodon API is Responsive ===
+echo "[*] Waiting for Mastodon to come back online..."
+until curl -sf "$MASTODON_INSTANCE/api/v1/instance" > /dev/null; do
+    echo "   ... still starting up, retrying in 5s"
+    sleep 5
+done
+
+echo "[+] Mastodon is back online."
+
+# === Post Final Toot ===
+echo "[*] Posting final status to Mastodon..."
+curl -s -X POST "$MASTODON_INSTANCE/api/v1/statuses" \
+  -H "Authorization: Bearer $ACCESS_TOKEN" \
+  -d "status=$FINAL_TOOT" \
+  -d "visibility=$TOOT_VISIBILITY" > /dev/null && echo "[✓] Final status posted."
+
+# === Telegram Notification ===
+echo "[*] Sending Telegram alert..."
+curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
+  -d chat_id="$TELEGRAM_CHAT_ID" \
+  -d text="$TELEGRAM_TEXT" > /dev/null && echo "[✓] Telegram alert sent."
+
+echo "[✓] All tasks complete. Logged out of $SERVER."
+echo "[$TIMESTAMP] === Mastodon Restart Complete ==="
+echo ""
+
+} >> "$LOG_FILE" 2>&1

miscellaneous/dbcheck.log

@@ -1138,3 +1138,85 @@ Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
 Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
 Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
 Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+⚠️ Genesis Radio Warning Healthcheck 2025-05-01 07:00:15 ⚠️
+⚡ 3 warnings found:
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+⚠️ Genesis Radio Warning Healthcheck 2025-05-01 07:15:15 ⚠️
+⚡ 5 warnings found:
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+- 💥 [db2] WARNING: Replication lag is 135 seconds.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+✅ Genesis Radio Healthcheck 2025-05-01 07:30:14: All systems normal.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+✅ Genesis Radio Healthcheck 2025-05-01 07:45:17: All systems normal.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+⚠️ Genesis Radio Warning Healthcheck 2025-05-01 08:00:14 ⚠️
+⚡ 2 warnings found:
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+⚠️ Genesis Radio Warning Healthcheck 2025-05-01 08:15:15 ⚠️
+⚡ 4 warnings found:
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+- 💥 [db2] WARNING: Replication lag is 72 seconds.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+✅ Genesis Radio Healthcheck 2025-05-01 08:30:14: All systems normal.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+✅ Genesis Radio Healthcheck 2025-05-01 08:45:13: All systems normal.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+✅ Genesis Radio Healthcheck 2025-05-01 09:00:14: All systems normal.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+✅ Genesis Radio Healthcheck 2025-05-01 09:15:14: All systems normal.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+⚠️ Genesis Radio Warning Healthcheck 2025-05-01 09:30:13 ⚠️
+⚡ 1 warnings found:
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+⚠️ Genesis Radio Warning Healthcheck 2025-05-01 09:45:16 ⚠️
+⚡ 1 warnings found:
+- ⚠️ [mastodon] WARNING: Pattern 'ERROR' in /var/log/syslog
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+✅ Genesis Radio Healthcheck 2025-05-01 10:00:13: All systems normal.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+✅ Genesis Radio Healthcheck 2025-05-01 10:15:17: All systems normal.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}
+✅ Genesis Radio Healthcheck 2025-05-01 10:30:16: All systems normal.
+Failed to send Mastodon DM (attempt 1): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 2): {"error":"The access token is invalid"}
+Failed to send Mastodon DM (attempt 3): {"error":"The access token is invalid"}