Auto-commit from giteapush.sh at 2025-05-09 16:35:15

documents/genesishosting/access/account-creation.md

@@ -0,0 +1,20 @@
+# Account Creation Policy
+
+## Customer Accounts
+
+- Created automatically via WHMCS upon signup
+- Email verification is required before service activation
+- Strong passwords (minimum 10 characters) are enforced
+- 2FA is recommended and required for admin-facing services
+
+## Staff/Admin Accounts
+
+- Created manually by Super Admin only
+- Must use SSH keys for server access
+- Access logs are enabled and monitored
+- Each staff account must be linked to an internal email
+
+## Account Naming Convention
+
+- Customers: `client_{username}`
+- Admins: `admin.{firstname}`

documents/genesishosting/access/account-deletion.md

@@ -0,0 +1,13 @@
+# Account Deletion Policy
+
+## Customer Accounts
+
+- Users may request account deletion via WHMCS support ticket
+- Data is retained for 30 days post-termination (unless legally required)
+- Backups including user data are purged after 30 days
+
+## Internal Accounts
+
+- Deactivated immediately upon staff departure or role change
+- SSH keys, DirectAdmin access, and database credentials revoked
+- Logs associated with the account are retained for audit purposes

documents/genesishosting/access/least-priv.md

@@ -0,0 +1,20 @@
+# Least Privilege Policy
+
+Genesis Hosting enforces least privilege access for all systems.
+
+## Principles
+
+- Users are given the minimum level of access necessary to perform their work
+- Admin tools are isolated by function (e.g., billing vs. system access)
+- Escalation of privileges must be requested, documented, and time-bound
+
+## Tools in Use
+
+- WHMCS permissions are restricted by group
+- SSH access is limited using `AllowUsers` and firewalled IPs
+- TeamTalk server admins are rotated and audited monthly
+
+## Review Cycle
+
+- Access roles are reviewed quarterly
+- Logs of access changes are stored and rotated every 90 days

documents/genesishosting/access/user-roles.md

@@ -0,0 +1,18 @@
+# User Roles
+
+Genesis Hosting Technologies uses Role-Based Access Control (RBAC) to ensure that users only have access to what they need.
+
+## Role Definitions
+
+| Role           | Description                                              | Examples                         |
+|----------------|----------------------------------------------------------|----------------------------------|
+| Customer       | End users with access to services they’ve purchased      | DirectAdmin clients, Streamers   |
+| Support Staff  | Limited admin functions for resolving client issues      | Helpdesk, WHMCS support agents   |
+| Administrator  | Full access to provision, maintain, and modify services  | Infrastructure admins            |
+| Super Admin    | Root-level access to all systems                         | Owner/Lead Engineer              |
+
+## Guidelines
+
+- Roles are assigned during onboarding.
+- Access levels are reviewed quarterly.
+- No one should hold higher access than required for their duties.

documents/genesishosting/backups/backup-disaster-recovery.md

@@ -0,0 +1,26 @@
+# Disaster Recovery Plan
+
+Genesis Hosting is prepared to recover core systems from catastrophic failure.
+
+## Recovery Objectives
+
+- **RPO (Recovery Point Objective)**: 24 hours
+- **RTO (Recovery Time Objective)**: 4 hours for customer services
+
+## Full Recovery Flow
+
+1. Triage the affected systems
+2. Identify last successful backup or snapshot
+3. Restore individual services:
+   - DNS
+   - WHMCS
+   - DirectAdmin
+   - AzuraCast
+   - TeamTalk
+4. Run post-restore validation scripts
+5. Notify customers of incident and resolution
+
+## DR Testing
+
+- Simulated quarterly
+- Logs retained in `/var/log/genesisdr.log`

documents/genesishosting/backups/backup-integrity.md

@@ -0,0 +1,23 @@
+# Backup Integrity
+
+We verify all backups regularly to ensure they are complete, uncorrupted, and restorable.
+
+## Weekly Tasks
+
+- ZFS scrubs for all pools
+- Hash checks (SHA-256) for tarballs and dumps
+- rsync `--checksum` verification for remote mirrors
+
+## Alerts
+
+- Email/Mastodon alert if:
+  - ZFS reports checksum errors
+  - Scheduled backup is missing
+  - Remote sync fails or lags > 24h
+
+## Tools Used
+
+- `zfs scrub`
+- `sha256sum` + custom validation script
+- rclone sync logs
+- Telegram bot and Genesis Shield notifications

documents/genesishosting/backups/backup-policy.md

@@ -0,0 +1,29 @@
+# Backup Policy
+
+Genesis Hosting Technologies maintains regular backups to ensure customer data and internal infrastructure are recoverable in the event of failure, corruption, or disaster.
+
+## Backup Schedule
+
+| System         | Frequency | Retention | Method           |
+|----------------|-----------|-----------|------------------|
+| DirectAdmin    | Daily     | 7 Days    | rsync + tarball  |
+| WHMCS          | Daily     | 14 Days   | Encrypted dump   |
+| AzuraCast      | Daily     | 7 Days    | Docker volume snapshot + config export |
+| TeamTalk       | Daily     | 7 Days    | XML + config archive |
+| Full VMs       | Weekly    | 4 Weeks   | ZFS snapshots or Proxmox backups       |
+| Offsite Backups| Weekly    | 4 Weeks   | Rsync to remote ZFS or object storage  |
+
+## Retention Policy
+
+- Daily: 7 days
+- Weekly: 4 weeks
+- Monthly: Optional, for specific business data
+
+## Encryption
+
+- Backups are encrypted at rest (AES-256)
+- Transfers to remote locations use SSH or TLS
+
+## Notes
+
+- No backup occurs on client plans marked "opt-out"

documents/genesishosting/backups/dr/assets-mastodon-bucket.md

@@ -0,0 +1,45 @@
+## 2025-05-02 22:24:25 – MinIO Bucket Access Configuration for Mastodon
+
+**Bucket**: `assets-mastodon`
+**Server**: `shredderv2`
+**User**: `genesisuser`
+**Permissions**: Read / Write / Delete
+**Policy Name**: `assets-mastodon-rw-policy`
+
+### Commands Executed:
+
+```bash
+mc alias set localminio http://localhost:9000 genesisadmin MutationXv3!
+
+cat > assets_mastodon_rw_policy.json <<EOF
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Action": [
+        "s3:GetBucketLocation",
+        "s3:ListBucket"
+      ],
+      "Effect": "Allow",
+      "Resource": "arn:aws:s3:::assets-mastodon"
+    },
+    {
+      "Action": [
+        "s3:PutObject",
+        "s3:GetObject",
+        "s3:DeleteObject"
+      ],
+      "Effect": "Allow",
+      "Resource": "arn:aws:s3:::assets-mastodon/*"
+    }
+  ]
+}
+EOF
+
+mc admin policy add localminio assets-mastodon-rw-policy assets_mastodon_rw_policy.json
+mc admin policy set localminio assets-mastodon-rw-policy user=genesisuser
+```
+
+### Outcome:
+
+User `genesisuser` now has full authenticated access to `assets-mastodon` on `shredderv2`'s MinIO.

documents/genesishosting/backups/dr/assets_azuracast.md

@@ -0,0 +1,93 @@
+## 2025-05-02 22:24:25 – MinIO Bucket Access Configuration for Mastodon
+
+**Bucket**: `assets-mastodon`
+**Server**: `shredderv2`
+**User**: `genesisuser`
+**Permissions**: Read / Write / Delete
+**Policy Name**: `assets-mastodon-rw-policy`
+
+### Commands Executed:
+
+```bash
+mc alias set localminio http://localhost:9000 genesisadmin MutationXv3!
+
+cat > assets_mastodon_rw_policy.json <<EOF
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Action": [
+        "s3:GetBucketLocation",
+        "s3:ListBucket"
+      ],
+      "Effect": "Allow",
+      "Resource": "arn:aws:s3:::assets-mastodon"
+    },
+    {
+      "Action": [
+        "s3:PutObject",
+        "s3:GetObject",
+        "s3:DeleteObject"
+      ],
+      "Effect": "Allow",
+      "Resource": "arn:aws:s3:::assets-mastodon/*"
+    }
+  ]
+}
+EOF
+
+mc admin policy add localminio assets-mastodon-rw-policy assets_mastodon_rw_policy.json
+mc admin policy set localminio assets-mastodon-rw-policy user=genesisuser
+```
+
+### Outcome:
+
+User `genesisuser` now has full authenticated access to `assets-mastodon` on `shredderv2`'s MinIO.
+
+---
+
+## 2025-05-02 22:43:00 – MinIO Transfer Log: AzuraCast Assets
+
+**Source**: `thevault:/nexus/miniodata/assets_azuracast`
+**Destination**: `shredderv2 MinIO` bucket `assets-azuracast`
+
+### Transfer Method:
+
+```bash
+rclone sync thevault:/nexus/miniodata/assets_azuracast localminio:assets-azuracast \
+  --progress \
+  --transfers=8 \
+  --checkers=8 \
+  --s3-chunk-size=64M \
+  --s3-upload-concurrency=4 \
+  --s3-acl=private \
+  --s3-storage-class=STANDARD
+```
+
+### Outcome:
+
+Data from AzuraCast backup (`assets_azuracast`) successfully synchronized to MinIO bucket `assets-azuracast` on `shredderv2`.
+
+---
+
+## 2025-05-02 23:05:00 – MinIO Transfer Log: Mastodon Assets
+
+**Source**: `thevault:/nexus/miniodata/assets_mastodon`
+**Destination**: `shredderv2 MinIO` bucket `assets-mastodon`
+
+### Transfer Method:
+
+```bash
+rclone sync thevault:/nexus/miniodata/assets_mastodon localminio:assets-mastodon \
+  --progress \
+  --transfers=8 \
+  --checkers=8 \
+  --s3-chunk-size=64M \
+  --s3-upload-concurrency=4 \
+  --s3-acl=private \
+  --s3-storage-class=STANDARD
+```
+
+### Outcome:
+
+Assets from `assets_mastodon` replicated to `assets-mastodon` bucket on `shredderv2`. No impact to production (`shredderv1`) during sync.

documents/genesishosting/backups/restore-instructions.md

@@ -0,0 +1,32 @@
+# Restore Instructions
+
+The following steps outline how to restore data for each supported service.
+
+## DirectAdmin
+
+1. Access DA panel as admin
+2. Go to Admin Backup/Transfer
+3. Select user and backup date
+4. Click "Restore"
+
+## WHMCS
+
+1. SSH into WHMCS server
+2. Restore from encrypted MySQL dump
+3. Restart `php-fpm` and `nginx`
+
+## AzuraCast
+
+1. Stop all Docker containers
+2. Replace `station_data` and `config` volumes
+3. Restart stack via `docker-compose up -d`
+
+## TeamTalk
+
+1. Replace configuration file (`tt5srv.xml`)
+2. Restart TeamTalk server
+
+## VM-Level Restore (ZFS)
+
+1. `zfs rollback poolname/dataset@snapshotname`
+2. Verify service health and logs

documents/genesishosting/clients/abuse.md

@@ -0,0 +1,27 @@
+# Abuse Handling Policy
+
+We take reports of abuse seriously and aim to resolve them quickly.
+
+## How to Report Abuse
+
+Send an email to abuse@genesishostingtechnologies.com with:
+
+- Description of the abuse
+- IP or domain involved
+- Any relevant logs or screenshots
+
+## Internal Response Process
+
+1. Triage within 12 hours
+2. Investigate logs and usage
+3. Contact the client with 24h to respond
+4. Temporary suspension may be issued to prevent further harm
+
+## DMCA Takedowns
+
+- We comply with valid DMCA requests
+- The client will be notified and given 48h to address or refute
+
+## Escalation
+
+Repeat offenders may be permanently banned.

documents/genesishosting/clients/account-suspension.md

@@ -0,0 +1,22 @@
+# Account Suspension Policy
+
+Accounts may be suspended for violations of our Acceptable Use Policy, overdue invoices, or abuse complaints.
+
+## Common Reasons
+
+- Non-payment (after 5-day grace period)
+- Resource abuse or denial-of-service behavior
+- Hosting prohibited content
+- Violating community guidelines on TeamTalk
+
+## Suspension Procedure
+
+- Warning issued via WHMCS ticket and email
+- If no resolution within 24–48h, service is suspended
+- Admin note added to client profile for audit tracking
+
+## Reinstatement
+
+- Suspension is lifted upon payment or resolution
+- $5 reactivation fee may apply (for non-payment suspensions)
+- Services are not reinstated if terminated due to serious AUP violation

documents/genesishosting/clients/aup.md

@@ -0,0 +1,27 @@
+# Acceptable Use Policy (AUP)
+
+This policy outlines the acceptable use of services provided by Genesis Hosting Technologies.
+
+## Prohibited Activities
+
+Clients may not use our services to:
+
+- Host or distribute malware, phishing sites, or spyware
+- Send unsolicited email (spam), whether direct or relayed
+- Host copyrighted content without permission (DMCA applies)
+- Promote hate speech, harassment, or targeted abuse
+- Overuse system resources in a way that affects others
+
+## Special Notes
+
+- Streaming via AzuraCast must comply with DMCA and public broadcast standards
+- TeamTalk users must not harass, dox, or spam other users
+- VPNs, proxies, and anonymizing services are not allowed without prior approval
+
+## Enforcement
+
+Violations will result in one or more of the following:
+
+- Warning via email or WHMCS ticket
+- Service suspension
+- Permanent termination without refund (in egregious cases)

documents/genesishosting/clients/refunds-cancellations.md

@@ -0,0 +1,24 @@
+# Refunds & Cancellations
+
+Genesis Hosting Technologies offers a clear refund and cancellation policy.
+
+## Cancellation
+
+- Clients may cancel via WHMCS at any time
+- Cancellation before next billing date avoids future charges
+- No prorated refunds for unused time unless due to service failure
+
+## Refunds
+
+- Full refund within 7 days of initial purchase (DirectAdmin, AzuraCast, TeamTalk)
+- Domain registrations, SSL certificates, and add-ons are non-refundable
+- No refunds issued for abuse-related suspensions or policy violations
+
+## Exceptions
+
+- If we fail to deliver a service or suffer extended downtime (>24h), credit may be issued
+- All refund requests are reviewed manually by support
+
+## How to Request
+
+Submit a WHMCS ticket with reason for refund

documents/genesishosting/company/company-code-of-conduct.md

@@ -0,0 +1,20 @@
+# Code of Conduct
+
+We maintain a respectful, safe, and inclusive environment for both staff and clients.
+
+## Expectations
+
+- Treat all clients and team members with professionalism and courtesy
+- Communicate clearly and constructively — even during escalations
+- Uphold privacy, security, and transparency at every level
+- Follow internal and customer-facing policies at all times
+
+## Zero Tolerance
+
+We do not tolerate:
+
+- Harassment or abuse (verbal, written, or otherwise)
+- Discrimination based on identity, ability, or belief
+- Intentional sabotage of infrastructure or service integrity
+
+Violations may result in immediate termination of access or service.

documents/genesishosting/company/company-mission-statement.md

@@ -0,0 +1,12 @@
+# Mission Statement
+
+At Genesis Hosting Technologies, our mission is to provide secure, reliable, and transparent hosting services with a personal touch.
+
+We believe that even the smallest teams deserve enterprise-grade infrastructure — without enterprise-grade headaches.
+
+Our goal is to deliver:
+
+- Fast, stable hosting environments
+- Fair pricing with no upsell games
+- Transparent policies and proactive support
+- A commitment to data ownership and user privacy

documents/genesishosting/company/company-tos.md

@@ -0,0 +1,25 @@
+# Terms of Service (TOS)
+
+By using services from Genesis Hosting Technologies, you agree to the following terms:
+
+## Service Provision
+
+- Services are delivered as-is, with best-effort uptime and technical support
+- Users must abide by our Acceptable Use Policy (AUP)
+- Access may be suspended for abuse, non-payment, or security issues
+
+## Billing & Renewals
+
+- All services are billed monthly or annually
+- Automatic renewal is enabled by default
+- Invoices are due within 5 days of issue unless otherwise agreed
+
+## Termination
+
+- You may cancel at any time via WHMCS
+- We reserve the right to suspend or terminate accounts that violate our policies
+
+## Liability
+
+- We are not liable for data loss, service interruptions, or indirect damages
+- Backups are provided as a best-effort courtesy unless contractually guaranteed

documents/genesishosting/company/dmca.md

@@ -0,0 +1,25 @@
+# DMCA Policy
+
+Genesis Hosting Technologies complies with the Digital Millennium Copyright Act (DMCA).
+
+## Filing a Takedown Notice
+
+Email dmca@genesishostingtechnologies.com with:
+
+- Your contact information
+- Description of the copyrighted work
+- URL or IP address of the infringing content
+- A statement of good faith belief
+- A statement of accuracy and authority
+
+## What Happens Next
+
+- We review and respond within 48 hours
+- The client is notified and given a chance to respond
+- If no valid counter-notice is filed, content may be removed or suspended
+
+## Filing a Counter Notice
+
+Clients who believe their content was wrongly removed may submit a counter notice with similar contact and justification information.
+
+We will not tolerate repeated infringement and may terminate accounts accordingly.

documents/genesishosting/company/privacy-policy.md

@@ -0,0 +1,26 @@
+# Privacy Policy
+
+We respect your privacy and protect your data.
+
+## What We Collect
+
+- Account information: name, email, billing address
+- Service usage data: IPs, access logs, system metrics
+- Communications: support tickets and emails
+
+## How We Use It
+
+- Service provisioning and support
+- Abuse prevention and system integrity
+- Internal analytics (not shared or sold)
+
+## Data Sharing
+
+- We do not sell user data
+- We may share limited data with trusted providers (e.g., payment processors)
+- Law enforcement requests must include valid legal process
+
+## Data Retention
+
+- User data is retained as long as the account is active
+- Backups are purged per the Backup Policy

documents/genesishosting/disrec/zfsdestroycasestudy.md

@@ -0,0 +1,64 @@
+# 📛 Case Study: Why RAID Is Not a Backup
+
+## Overview
+
+On May 4, 2025, we experienced a production data loss incident involving the `nexus` dataset on `shredderv1`, a Linux RAID5 server. Despite no hardware failure, critical files were lost due to an unintended command affecting live data.
+
+This incident serves as a clear, real-world illustration of the maxim:
+
+> **RAID protects against hardware failure — not human error, data corruption, or bad automation.**
+
+---
+
+## 🔍 What Happened
+
+- `shredderv1` uses RAID5 for media storage.
+- The dataset `nexus/miniodata` (housing `genesisassets`, `genesislibrary`, etc.) was accidentally destroyed.
+- **No disks failed.** The failure was logical, not physical.
+
+---
+
+## 🔥 Impact
+
+- StationPlaylist (SPL) lost access to the Genesis media library.
+- MinIO bucket data was instantly inaccessible.
+- Temporary outage and scrambling to reconfigure mounts, media, and streaming.
+
+---
+
+## ✅ Recovery
+
+Thanks to our disaster recovery stack:
+
+- Nightly **rsync backups** were synced to **The Vault** (backup server).
+- **ZFS snapshots** existed on The Vault for the affected datasets.
+- We restored the latest snapshot **from The Vault back to Shredder**, effectively reversing the loss.
+- No data corruption occurred; sync validation showed dataset integrity.
+
+---
+
+## 🎓 Takeaway
+
+This is a live demonstration of why:
+
+- **RAID is not a backup**
+- **Snapshots without off-host replication** are not enough
+- **Real backups must be off-server and regularly tested**
+
+---
+
+## 🔐 Current Protection Measures
+
+- Production data (`genesisassets`, `genesislibrary`) now replicated nightly to The Vault via `rsync`.
+- ZFS snapshots are validated daily via a **dry-run restore validator**.
+- Telegram alerts notify success/failure of backup verification jobs.
+- Future goal: full ZFS storage on all production servers for native snapshot support.
+
+---
+
+## 🧠 Lessons Learned
+
+- Always assume you'll delete the wrong thing eventually.
+- Snapshots are amazing — **if** they're somewhere else.
+- Automated restore testing should be part of every backup pipeline.
+

documents/genesishosting/dns-check

@@ -0,0 +1,50 @@
+# 🌐 DNS Access Issues – Troubleshooting Guide
+
+If you're having trouble reaching **Genesis Radio** or the stream won't load, the issue may be with your DNS provider (the service that turns domain names into IP addresses).
+
+This happens more often than you'd think — and it's easy to fix.
+
+---
+
+## ✅ Quick Fix: Change Your DNS
+
+We recommend switching to one of these trusted, fast, privacy-respecting DNS providers:
+
+| Provider     | DNS Servers                 |
+|--------------|-----------------------------|
+| **Google**   | `8.8.8.8` and `8.8.4.4`      |
+| **Cloudflare** | `1.1.1.1` and `1.0.0.1`    |
+| **Quad9**    | `9.9.9.9`                   |
+
+---
+
+## 💻 How to Change Your DNS
+
+### Windows 10/11
+1. Open **Settings → Network & Internet**
+2. Click **Change adapter options**
+3. Right-click your active connection → **Properties**
+4. Select **Internet Protocol Version 4 (TCP/IPv4)** → Click **Properties**
+5. Choose **"Use the following DNS server addresses"**
+6. Enter:  
+   - Preferred: `1.1.1.1`  
+   - Alternate: `8.8.8.8`
+7. Save and reconnect
+
+---
+
+### macOS
+1. Go to **System Preferences → Network**
+2. Select your active network → Click **Advanced**
+3. Go to the **DNS** tab
+4. Click `+` and add:
+   - `1.1.1.1`
+   - `8.8.8.8`
+5. Apply changes and reconnect
+
+---
+
+### Linux (CLI)
+For a quick test:
+```bash
+sudo resolvectl dns eth0 1.1.1.1 8.8.8.8

documents/genesishosting/infra/genesis-shield.md

@@ -0,0 +1,24 @@
+# Genesis Shield – Security & Threat Monitoring
+
+Genesis Shield is our custom-built alert and ban system, integrated across our infrastructure.
+
+## Features
+
+- Aggregates Fail2Ban logs across all VMs
+- Bans pushed in real-time via Mastodon DM and Telegram
+- Scripts track:
+  - Repeated SSH failures
+  - API abuse
+  - Web panel brute force attempts
+
+## Interfaces
+
+- Terminal dashboard for live bans/unbans
+- Role-based control (root/admin only)
+- Daily threat summary via Mastodon bot
+
+## Roadmap
+
+- WHMCS integration for abuse tickets
+- Live threat map by country/IP
+- REST API for admin toolkit

documents/genesishosting/infra/infra-maintenance-windows.md

@@ -0,0 +1,25 @@
+# Maintenance Window Policy
+
+To maintain consistency and reduce customer impact, we adhere to a strict maintenance schedule.
+
+## Standard Window
+
+- **Every Sunday, 7 PM – 9 PM Eastern**
+- Non-emergency changes must occur during this window
+
+## What’s Allowed
+
+- OS & kernel updates
+- Docker/image upgrades
+- ZFS snapshots & cleanup
+- Rolling restarts of containers
+
+## Emergencies
+
+- Critical security patches can bypass the window
+- All emergency changes must be logged and reviewed
+
+## Notifications
+
+- Posted on Mastodon at least 1 hour before the window begins
+- Clients notified via WHMCS if it will affect their service

documents/genesishosting/infra/infra-monitoring-setup.md

@@ -0,0 +1,25 @@
+# Monitoring Setup
+
+We use a layered monitoring approach to ensure full visibility and rapid response.
+
+## Stack
+
+- **Prometheus** for metrics collection
+- **Grafana** for visualization dashboards
+- **Fail2Ban** for intrusion attempts
+- **Genesis Shield** for aggregated alerts (Telegram + Mastodon)
+
+## What We Monitor
+
+| System         | Metric Examples                           |
+|----------------|--------------------------------------------|
+| PostgreSQL     | Replication lag, disk usage, active queries |
+| Web Servers    | HTTP response time, TLS errors             |
+| MinIO / Assets | Cache hit ratio, sync status               |
+| Docker Hosts   | Container uptime, memory pressure          |
+
+## Alerting
+
+- Telegram: Real-time infra alerts
+- Mastodon bot: Daily summaries and status posts
+- Fallback email alerts for critical failures

documents/genesishosting/infra/server-naming-convention.md

@@ -0,0 +1,19 @@
+# Server Naming Convention
+
+To reduce confusion and improve clarity, we follow a clear and themed naming structure.
+
+## Naming Style
+
+Examples:
+
+- `krang.internal` – Master backend server
+- `replica.db3.sshjunkie.com` – Staging PostgreSQL replica
+- `shredderv2` – ZFS backup server
+- `anthony` – Ansible control node
+- `nexus` – Main ZFS pool server for assets
+
+## Guidelines
+
+- Avoid generic names (`server1`, `host123`)
+- Use themed names (e.g., TMNT characters for core infrastructure)
+- Include environment tags where needed (`-test`, `-prod`)

documents/genesishosting/infra/zfs-strategy.md

@@ -0,0 +1,23 @@
+# ZFS Strategy
+
+ZFS is used across Genesis Hosting Technologies for performance, integrity, and snapshot-based backup operations.
+
+## Pool Layout
+
+- RAIDZ1 or mirrored vdevs depending on use case
+- Dataset naming: `genesisassets-secure`, `genesisshows-secure`, etc.
+- Dedicated pools for:
+  - Mastodon media
+  - Client backups
+  - Internal scripts and logs
+
+## Snapshots
+
+- Hourly: last 24 hours
+- Daily: last 7 days
+- Weekly: last 4 weeks
+
+## Send/Receive
+
+- Used for offsite replication to Servarica and backup nodes
+- Verified using checksums and `zfs receive -F`

documents/genesishosting/master_compliance_checklist.md

@@ -0,0 +1,63 @@
+# ✅ Master Compliance Checklist  
+*(Status: ☐ = Not Started | 🟨 = In Progress | ✅ = Complete)*
+
+## 🧑‍💼 Access & User Management
+- [ ] Role-Based Access Control (RBAC) in place (Customer, Admin, etc.)
+- [ ] Account creation follows secure onboarding workflows
+- [ ] Admin access restricted to SSH keys only
+- [ ] Inactive accounts locked or removed quarterly
+- [ ] Least privilege enforced across all services
+
+## 💾 Backups & Disaster Recovery
+- [ ] Daily backups enabled for all key services (DirectAdmin, WHMCS, AzuraCast, TeamTalk)
+- [ ] Weekly offsite backups with verification
+- [ ] ZFS snapshots scheduled (hourly/daily/weekly)
+- [ ] Backup integrity validated with checksums or scrubs
+- [ ] Quarterly disaster recovery drill completed
+- [ ] Restore instructions documented and tested
+
+## 🔐 Security
+- [ ] 2FA enabled on all admin interfaces (WHMCS, SSH, DirectAdmin)
+- [ ] SSH password auth disabled, key-only enforced
+- [ ] Weekly patching or updates scheduled (Sunday 7–9 PM)
+- [ ] Centralized logging active and stored 30–90 days
+- [ ] Fail2Ban + Genesis Shield integrated and alerting
+- [ ] TLS 1.2+ enforced for all public services
+- [ ] AES-256 encryption at rest on backups and sensitive volumes
+
+## 🖥️ Provisioning & Automation
+- [ ] WHMCS integrated with DirectAdmin, AzuraCast, TeamTalk
+- [ ] All provisioning scripts tested and logged
+- [ ] Post-deploy verification checklist followed
+- [ ] DNS + SSL automation in place (Let's Encrypt)
+- [ ] Monitoring added after provisioning (Prometheus/Grafana)
+
+## 📋 Client Policies
+- [ ] Acceptable Use Policy posted and enforced
+- [ ] Abuse response process defined and working
+- [ ] DMCA policy publicly available and followed
+- [ ] Suspension and refund rules defined in WHMCS
+- [ ] Privacy Policy and Terms of Service available on client portal
+
+## 🌐 Services Configuration
+- [ ] DirectAdmin quotas enforced (disk, bandwidth, email)
+- [ ] AzuraCast listener/storage/bitrate limits respected
+- [ ] TeamTalk server abuse protection and user limits enforced
+- [ ] Domain registration/renewal workflows tested
+- [ ] SSL auto-renew working correctly (Let's Encrypt + certbot)
+
+## ⚙️ Infrastructure
+- [ ] ZFS pools configured for redundancy (RAIDZ1, mirrors)
+- [ ] rclone mount points with caching working and monitored
+- [ ] Genesis Shield actively alerting via Telegram/Mastodon
+- [ ] All VMs named per convention (e.g., `krang`, `shredderv2`)
+- [ ] Sunday maintenance window consistently followed
+- [ ] Ansible playbooks used for provisioning/config consistency
+
+## 🛠️ Tools & Scripts
+- [ ] All scripts version-controlled and documented
+- [ ] Backups and restore tools tested and working
+- [ ] Mastodon alert bot operating with secure tokens
+- [ ] Rclone VFS stats monitored regularly
+- [ ] Admin tools accessible only by authorized users
+"""

documents/genesishosting/pmgenesisiorealignment.md

@@ -0,0 +1,83 @@
+# Postmortem: Genesis I/O Realignment
+
+**Date:** May 8, 2025  
+**Author:** Doc  
+**Systems Involved:** minioraid5, shredder, chatwithus.live, zcluster.technodrome1/2, thevault  
+**Scope:** Local-first mirroring, permission normalization, MinIO transition
+
+---
+
+## 🎯 Objective
+
+To realign the Genesis file flow architecture by:
+
+- Making local block storage the **primary source** of truth for AzuraCast and Genesis buckets
+- Transitioning FTP uploads to target local storage instead of MinIO directly
+- Establishing **two-way mirroring** between local paths and MinIO buckets
+- Correcting inherited permission issues across `/mnt/raid5` using `find + chmod`
+- Preserving MinIO buckets as **backup mirrors**, not primary data stores
+
+---
+
+## 🔧 Work Performed
+
+### ✅ Infrastructure changes:
+- Deployed block storage volume to Linode Mastodon instance
+- Mirrored MinIO buckets (`genesisassets`, `genesislibrary`, `azuracast`) to local paths
+- Configured cron-based `mc mirror` jobs:
+  - Local ➜ MinIO: every 5 minutes with `--overwrite --remove`
+  - MinIO ➜ Local: nightly pull, no `--remove`
+
+### ✅ FTP Pipeline Adjustments:
+- Users now upload to `/mnt/spl/ftp/uploads` (local)
+- Permissions set so only admins access full `/mnt/spl/ftp`
+- FTP directory structure created for SPL automation
+
+### ✅ System Tuning:
+- Set `vm.swappiness=10` on all nodes
+- Apache disabled where not in use
+- Daily health checks via `pull_health_everywhere.sh`
+- Krang Telegram alerts deployed for cleanup and system state
+
+---
+
+## 🧠 Observations
+
+- **High load** on `minioraid5` during `mc mirror` and `chmod` overlap
+  - Load ~6.5 due to concurrent I/O pressure
+  - `chmod` stuck in `D` state (I/O wait) while `mc` dominated disk queues
+  - Resolved after `mc` completion — `chmod` resumed and completed
+
+- **MinIO buckets were temporarily inaccessible** due to permissions accidentally inherited by FTP group
+  - Resolved by recursively resetting permissions on `/mnt/raid5`
+
+- **Krang telemetry** verified:
+  - Mastodon swap usage rising under asset load
+  - All nodes had Apache disabled or dormant
+  - Health alerts triggered on high swap or load
+
+---
+
+## ✅ Outcome
+
+- Full Genesis and AzuraCast data now reside locally with resilient S3 mirrors
+- Mastodon running on block storage, no longer dependent on MinIO latency
+- FTP integration with SPL directory trees complete
+- Cleanup script successfully deployed across all nodes via Krang
+- Daily health reports operational with alerts for high swap/load
+
+---
+
+## 🔁 Recommendations
+
+- Consider adding snapshot-based ZFS backups for `/mnt/raid5`
+- Build `verify_mirror.sh` to detect drift between MinIO and local storage
+- Auto-trigger `chmod` only after `mc mirror` finishes
+- Monitor long-running background jobs with Krang watchdogs
+
+---
+
+**Signed,**  
+Doc  
+Genesis Hosting Technologies
+

documents/genesishosting/provisioning/checklist.md

@@ -0,0 +1,23 @@
+# Provisioning Checklist
+
+This checklist is followed every time a new service is deployed.
+
+## Pre-Provisioning
+
+- [ ] Verify order and payment in WHMCS
+- [ ] Confirm product mapping is correct
+- [ ] Check available server resources
+
+## Provisioning
+
+- [ ] Trigger appropriate script/module
+- [ ] Log provisioning result
+- [ ] Assign DNS entries if applicable
+- [ ] Generate Let’s Encrypt SSL if public-facing
+
+## Post-Provisioning
+
+- [ ] Send welcome email via WHMCS
+- [ ] Confirm monitoring alert is active
+- [ ] Test login credentials and endpoints
+- [ ] Label service with client ID in Grafana/Prometheus

documents/genesishosting/provisioning/post-deploy-verification.md

@@ -0,0 +1,22 @@
+# Post-Deployment Verification
+
+All services go through a post-deploy QA check to ensure they're live and stable.
+
+## Verification Tasks
+
+- [ ] Service reachable from public IP or internal route
+- [ ] DNS resolves correctly (for domains/subdomains)
+- [ ] SSL certificate is active and trusted
+- [ ] Admin login works as expected
+- [ ] Usage quotas correctly applied (disk, users, bandwidth)
+
+## Monitoring
+
+- [ ] Add to Prometheus for service-specific metrics
+- [ ] Set alert thresholds (e.g., disk > 80%)
+- [ ] Confirm Telegram/Mastodon alert webhook is functional
+
+## Documentation
+
+- [ ] Log final status in WHMCS admin notes
+- [ ] Store internal service details in `genesis-inventory.yaml`

documents/genesishosting/provisioning/whmcs-integration.md

@@ -0,0 +1,23 @@
+# WHMCS Integration
+
+WHMCS handles client billing, service provisioning, and support workflows.
+
+## Services Integrated
+
+| Service      | Method                          |
+|--------------|---------------------------------|
+| DirectAdmin  | Built-in WHMCS module           |
+| AzuraCast    | Custom provisioning script       |
+| TeamTalk     | API + XML user patching scripts |
+
+## Auto-Provisioning Steps
+
+1. Client signs up and completes payment
+2. WHMCS triggers product-specific hook
+3. Script/module provisions the service
+4. Welcome email is sent with credentials
+
+## Logging & Troubleshooting
+
+- Logs stored at `/var/log/whmcs-hooks.log`
+- Errors generate internal ticket automatically if provisioning fails

documents/genesishosting/security/incident-response.md

@@ -0,0 +1,25 @@
+# Incident Response Policy
+
+This document defines how we detect, respond to, and report security incidents.
+
+## Response Workflow
+
+1. Detection via monitoring, alert, or client report
+2. Triage severity and affected systems
+3. Contain and isolate threat (e.g., suspend access)
+4. Notify stakeholders if client-impacting
+5. Perform root cause analysis
+6. Patch, re-secure, and document the event
+
+## Timelines
+
+- Initial triage: within 2 hours
+- Client notification (if impacted): within 24 hours
+- Final report delivered internally within 72 hours
+
+## Tools Used
+
+- Fail2Ban
+- Genesis Shield alerting
+- Zabbix/Prometheus incident flags
+- Manual log reviews (forensic-level)

documents/genesishosting/security/logging-monitoring.md

@@ -0,0 +1,24 @@
+# Logging & Monitoring Policy
+
+We collect and monitor system activity to detect threats, enforce accountability, and assist in incident resolution.
+
+## Log Types
+
+- SSH login attempts
+- WHMCS access logs
+- AzuraCast and TeamTalk server logs
+- PostgreSQL query and connection logs
+- Fail2Ban logs (ban/unban events)
+
+## Monitoring Tools
+
+- Prometheus for metrics
+- Grafana dashboards for visual alerts
+- Genesis Shield (Telegram + Mastodon alerting)
+- Manual log review every 7 days
+
+## Retention
+
+- General logs: 30 days
+- Security-related logs: 90 days minimum
+- Logs archived to encrypted ZFS volume

documents/genesishosting/security/security-encryption-standards.md

@@ -0,0 +1,23 @@
+# Encryption Standards
+
+Encryption is applied to all data in transit and at rest across Genesis Hosting Technologies infrastructure.
+
+## In Transit
+
+- HTTPS via TLS 1.3 (minimum TLS 1.2 for legacy fallback)
+- SFTP for all file transfers
+- SSH for all administrative access
+- rclone with TLS for object storage replication
+
+## At Rest
+
+- ZFS encryption on backup pools
+- PostgreSQL encryption at the database or filesystem level
+- WHMCS and DirectAdmin credentials hashed and salted
+- Backups encrypted with AES-256 before remote transfer
+
+## Key Management
+
+- SSH keys rotated every 6 months
+- Let's Encrypt certs auto-renew every 90 days
+- Master encryption keys stored offline and version-controlled

documents/genesishosting/security/security-policy.md

@@ -0,0 +1,23 @@
+# Security Policy
+
+Genesis Hosting Technologies enforces strict security practices across all infrastructure and services to protect client data and maintain service integrity.
+
+## Core Principles
+
+- Least privilege for all users and services
+- Regular audits and patching
+- Encrypted communication and storage
+- Real-time monitoring and alerting
+
+## Enforcement Areas
+
+- 2FA required for all admin portals
+- SSH access limited to key-based logins
+- Centralized log collection and review
+- All critical assets monitored via Genesis Shield
+
+## Review Cycle
+
+- Policies reviewed quarterly
+- Logs retained for 30–90 days depending on system
+- Incidents reviewed post-mortem with improvements logged

documents/genesishosting/services/azuracast-policy.md

@@ -0,0 +1,32 @@
+# AzuraCast Streaming Policy
+
+## Features
+
+- Custom stream URLs (via relay or direct)
+- Icecast or SHOUTcast available
+- AutoDJ + scheduled playlists
+- Web-based file upload + schedule
+
+## Plans & Limits
+
+| Plan     | Storage | Listeners | Bitrate |
+|----------|---------|-----------|---------|
+| StreamLite | 2 GB  | 25        | 128 kbps|
+| StreamPro  | 10 GB | 100       | 192 kbps|
+| StreamMax  | 50 GB | 250       | 320 kbps|
+
+## Fair Usage Policy
+
+- No nonstop streaming of static loops to inflate uptime
+- Long-form live shows should rotate metadata periodically
+- Content must not violate copyright laws
+
+## Backups
+
+- Daily backups of config + playlists
+- Client media backup is optional (paid add-on)
+
+## Support
+
+- Stream diagnostics available in client panel
+- WHMCS ticket support for outages or playlist issues

documents/genesishosting/services/directadmin-policy.md

@@ -0,0 +1,27 @@
+# DirectAdmin Hosting Policy
+
+## Features
+
+- FTP, webmail, MySQL, file manager, and site statistics
+- Optional Let's Encrypt SSL enabled by default
+- Nightly site + database backups (7-day retention)
+
+## Plans & Limits
+
+| Plan       | Disk | Bandwidth | Domains | Email Accounts |
+|------------|------|-----------|---------|----------------|
+| Starter    | 5 GB | 100 GB    | 1       | 5              |
+| Standard   | 20 GB| 500 GB    | 5       | 25             |
+| Unlimited  | 100 GB| ∞        | ∞       | ∞              |
+
+## Abuse Prevention
+
+- Email rate limits applied to prevent outbound spam
+- CPU usage and inode caps enforced
+- Suspicious files scanned automatically
+
+## Support
+
+- Available via WHMCS ticket system
+- Response within 12 business hours
+

documents/genesishosting/services/domain-management-policy.md

@@ -0,0 +1,22 @@
+# Domain Management Policy
+
+## Registration
+
+- Domains registered through our WHMCS interface are managed via third-party registrar API
+- Registration typically completes within 5 minutes
+- WHOIS privacy included by default (where available)
+
+## Renewals
+
+- Auto-renew is enabled by default
+- Reminders sent 30, 7, and 1 day before expiration
+
+## Transfers
+
+- Domains can be transferred in or out with EPP code
+- Support required if domain is locked or expired
+
+## DNS
+
+- Free DNS hosting included
+- Custom DNS records managed through DirectAdmin or WHMCS panel

documents/genesishosting/services/ssl-certs.md

@@ -0,0 +1,23 @@
+# SSL Certificate Policy
+
+## Free Certificates
+
+- Let’s Encrypt certificates issued automatically
+- Applies to DirectAdmin, AzuraCast, and custom subdomains
+- Auto-renews every 60 days with 30-day buffer
+
+## Premium SSL
+
+- Custom SSL certs (e.g., EV/OV) available for purchase
+- Requires manual install via WHMCS ticket
+
+## Certificate Management
+
+- Certbot used for automation
+- Custom certs must be supplied in `.crt` + `.key` format
+- Broken SSL installs may be reverted to Let’s Encrypt fallback
+
+## Support
+
+- Certificate issues resolved within 24h of report
+- DNS challenges supported for wildcard certs

documents/genesishosting/services/teamtalk-policy.md

@@ -0,0 +1,26 @@
+# TeamTalk Hosting Policy
+
+## Features
+
+- Private and public servers
+- Voice chat, file sharing, push-to-talk
+- Admin access with room/channel management
+
+## Plans & Limits
+
+| Plan         | Users | Bitrate Limit | Admin Access |
+|--------------|-------|---------------|--------------|
+| Basic Chat   | 10    | 64 kbps       | Yes          |
+| Pro Voice    | 50    | 128 kbps      | Yes          |
+| Broadcast+   | 100   | 256 kbps      | Yes          |
+
+## Rules
+
+- No harassment, spamming, or automated bots without permission
+- Abuse may result in temp suspension or permanent ban
+- Admins are responsible for moderating their own servers
+
+## Configuration
+
+- Clients may request config changes via WHMCS ticket
+- Backups of XML configs stored nightly

postmortem/genesisradiozfsmigration.md

@@ -1,68 +0,0 @@
-# 🔧 Post-Mortem: Genesis Radio Storage Migration
-**Date:** April 30, 2025  
-**Prepared by:** Doc  
-**Systems Affected:** StationPlaylist (SPL), Voice Tracker, Genesis Radio media backend  
-
----
-
-## 🧠 Executive Summary
-
-Genesis Radio’s backend was migrated from a legacy MinIO instance using local disk (ext4) to a new **ZFS-based, encrypted MinIO deployment on `shredderv2`**. This change was driven by a need for more stable performance, improved security, and a cleaner storage architecture with proper bucket separation.
-
-This migration was completed **without touching production** until final validation, and all critical services remained online throughout the transition. We also revamped the rclone caching strategy to reduce freeze-ups and playback hiccups.
-
----
-
-## ✅ What We Did
-
-- Created **three new secure buckets**: `genesislibrary-secure`, `genesisassets-secure`, and `genesisshows-secure`
-- Migrated data from backup server using `rclone sync`:
-  - `genesislibrary` came directly from backup
-  - `genesisassets` and `genesisshows` were pulled from the same bucket, with de-duping and cleanup to be completed post-migration
-- Retained **original SPL drive letters** (`Q:\\`, `R:\\`) to avoid changes to the playout config
-- Switched rclone mounts to point to the new secure buckets, with **aggressive VFS caching** using SSD-backed cache directories
-- Took a clean **ZFS snapshot** (`@pre-s3-switch`) before switching over
-- Confirmed no regression in SPL, VT Tracker, or streaming audio
-
----
-
-## ⚙️ Technical Improvements
-
-- **VFS caching overhaul**:
-  - Increased read-ahead (`1G`), lowered write-back wait
-  - Split cache between `X:\\librarycache` and `L:\\assetcache`
-  - No more rclone choking on large files or freezing during transitions
-- **Encrypted S3 storage** with isolated buckets per functional role
-- **TLS-secured** Console and MinIO endpoints with automated renewal
-- Mounted buckets at startup via batch script (future systemd equivalents to be implemented)
-- Snapshot-based rollback in ZFS enabled post-deployment resilience
-
----
-
-## 🩹 What Went Weird (and We Fixed It)
-
-- SPL froze during initial `mc mirror` attempts — solution: switched to `rclone`, which performed exponentially faster
-- Some hiccups during early cache tuning, including sparse file support issues — solved by switching to ZFS
-- Missing media files in Mastodon were traced to uploads during sync; resolved with staged sync + retry before final switch
-- Certbot automation wasn’t configured — resolved with a systemd timer that stops nginx, renews, and restarts nginx automatically
-
----
-
-## 🧯 What We Learned
-
-- MinIO is solid, but **rclone wins for bulk sync performance**
-- VFS cache settings **make or break** media-heavy workloads like SPL
-- ZFS is a game-changer: no sparse file errors, reliable snapshots, clean rollback
-- Planning matters: pre-syncing from backup avoided downtime
-- Not touching prod until ready keeps stress and screwups to a minimum
-
----
-
-## 📦 Next Steps
-
-- [ ] Clean `genesisassets-secure` of misplaced show files
-- [ ] Sync `azuracast` from live system (no backup copy yet)
-- [ ] Build automated snapshot send-to-backup workflow (`zfs send | ssh backup zfs recv`)
-- [ ] Stage full failover simulation (optional but fun)
-
-

postmortem/mastoobjectstorage.md

@@ -1,75 +0,0 @@
-# 🧾 Postmortem: Mastodon Object Storage Migration to Secure S3 (MinIO)
-**Date:** April 30, 2025  
-**Engineer:** Doc (Genesis Radio / Genesis Hosting)
-
----
-
-## 🎯 Objective
-
-Migrate Mastodon's object storage from an older MinIO bucket (`linodeassets`) to a new **ZFS-backed, encrypted** MinIO instance (`mastodonassets-secure`) on `shredderv2`, while maintaining uptime and improving storage performance and security.
-
----
-
-## 🧱 Infrastructure Touched
-
-- Mastodon (Docker-based, hosted on Linode)
-- MinIO S3 Object Storage (`oldminio` → `secureminio`)
-- Nginx (reverse proxy for Console + S3 endpoints)
-- ZFS pool: `nexus/mastodonassets`
-- Domains:
-  - `shredderv2.sshjunkie.com` (S3 API)
-  - `consolev2.sshjunkie.com` (MinIO Console UI)
-
----
-
-## ⚠️ Issues Encountered
-
-1. **403 Access Denied on Mastodon startup**
-   - ✅ Root cause: `genesisadminv2` MinIO user had no attached policy
-   - 🔧 Fixed via Console UI after re-enabling access
-
-2. **MinIO Console unreachable (`consolev2.sshjunkie.com`)**
-   - SSL cert for the domain was missing
-   - 🔧 Used `certbot certonly --standalone` to issue new cert, re-enabled full HTTPS proxy
-
-3. **Sync race conditions**
-   - Some media files were uploaded to the old bucket during the long transfer
-   - 🔧 Mitigated by running an additional `rclone sync` pass before cutover
-
-4. **Rclone performance bottlenecks**
-   - MinIO client (`mc mirror`) was too slow
-   - ✅ Switched to `rclone`, saw drastic speed improvement
-
-5. **SPL (StationPlaylist) freezing during asset access**
-   - Root cause: cache choking on sparse file writes under ext4
-   - ✅ Fix: moved critical rclone mounts to ZFS-backed drives
-
----
-
-## ✅ Success Criteria Met
-
-- 🔒 All Mastodon assets are now stored in `mastodonassets-secure` with encryption
-- 🪣 MinIO Console functional on `https://consolev2.sshjunkie.com`
-- 🎯 Mastodon is running with zero visible user impact
-- 💾 Snapshot (`nexus/mastodonassets@pre-s3-switch`) taken post-migration for rollback
-- 🔁 Future syncs can now be performed cleanly from backup server instead of live system
-
----
-
-## 🧠 Lessons Learned
-
-- Always validate MinIO user policies before go-live
-- Avoid redirects in `server_name` blocks during cert issuance
-- ZFS dramatically improves caching performance with rclone VFS
-- Post-cutover syncs are crucial for active upload systems like Mastodon
-- UI access to MinIO is a lifesaver for emergency fixes — keep it working
-
----
-
-## 🔚 Follow-Up Actions
-
-- [ ] Schedule `certbot renew --standalone` with systemd timer
-- [ ] Rotate MinIO user keys and audit access policies
-- [ ] Monitor `/var/log/syslog` for VFS or sparse file errors
-- [ ] Document your rclone mount and caching strategy for SPL and Mastodon
-

postmortem/pmgenesisiorealignment.md

@@ -1,100 +0,0 @@
-# Postmortem: Genesis I/O Realignment
-
-**Date:** May 8, 2025
-**Author:** Doc
-**Systems Involved:** minioraid5, shredder, chatwithus.live, zcluster.technodrome1/2, thevault
-**Scope:** Local-first mirroring, permission normalization, MinIO transition
-
----
-
-## 🎯 Objective
-
-To realign the Genesis file flow architecture by:
-
-* Making local block storage the **primary source** of truth for AzuraCast and Genesis buckets
-* Transitioning FTP uploads to target local storage instead of MinIO directly
-* Establishing **two-way mirroring** between local paths and MinIO buckets
-* Correcting inherited permission issues across `/mnt/raid5` using `find + chmod`
-* Preserving MinIO buckets as **backup mirrors**, not primary data stores
-
----
-
-## 🔧 Work Performed
-
-### ✅ Infrastructure changes:
-
-* Deployed block storage volume to Linode Mastodon instance
-* Mirrored MinIO buckets (genesisassets, genesislibrary, azuracast) to local paths
-* Configured cron-based `mc mirror` jobs:
-
-  * Local ➜ MinIO: every 5 minutes with `--overwrite --remove`
-  * MinIO ➜ Local: nightly pull, no `--remove`
-* Prepared 5TB local drive for AzuraCast asset mirroring (pending full sync)
-
-### ✅ FTP Pipeline Adjustments:
-
-* Users now upload to `/mnt/spl/ftp/uploads` (local)
-* Permissions set so only admins access full `/mnt/spl/ftp`
-* FTP directory structure created for SPL automation
-
-### ✅ System Tuning:
-
-* Set `vm.swappiness=10` on all nodes
-* Apache disabled where not in use
-* Daily health checks via `pull_health_everywhere.sh`
-* Krang Telegram alerts deployed for cleanup and system state
-
----
-
-## 🧠 Observations
-
-* **High load** on `minioraid5` during `mc mirror` and `chmod` overlap
-
-  * Load \~6.5 due to concurrent I/O pressure
-  * `chmod` stuck in `D` state (I/O wait) while `mc` dominated disk queues
-  * Resolved after `mc` completion — `chmod` resumed and completed
-
-* **MinIO buckets were temporarily inaccessible** due to permissions accidentally inherited by FTP group
-
-  * Resolved by recursively resetting permissions on `/mnt/raid5`
-
-* **Krang telemetry** verified:
-
-  * Mastodon swap usage rising under asset load
-  * All nodes had Apache disabled or dormant
-  * Health alerts triggered on high swap or load
-
----
-
-## ✅ Outcome
-
-* Full Genesis and AzuraCast data now reside locally with resilient S3 mirrors
-* Mastodon running on block storage, no longer dependent on MinIO latency
-* FTP integration with SPL directory trees complete
-* Cleanup script successfully deployed across all nodes via Krang
-* Daily health reports operational with alerts for high swap/load
-
----
-
-## 🔁 Recommendations
-
-* Proceed with AzuraCast mirror only after:
-
-  * Mastodon asset storage transition is confirmed stable
-  * All `/mnt/raid5` permission fixes are complete
-
-* Consider adding snapshot-based ZFS backups for `/mnt/raid5`
-
-* Build `verify_mirror.sh` to detect drift between MinIO and local storage
-
-* Auto-trigger `chmod` only after `mc mirror` finishes
-
-* Monitor long-running background jobs with Krang watchdogs
-
-* Finalize and launch AzuraCast 5TB mirror sync
-
----
-
-**Signed,**
-Doc
-Genesis Hosting Technologies