From 7ea6dda6147ea96d6dfc64cfd9ece37a1c36c27e Mon Sep 17 00:00:00 2001 From: DocTator Date: Fri, 9 May 2025 16:35:15 -0400 Subject: [PATCH] Auto-commit from giteapush.sh at 2025-05-09 16:35:15 --- .../cheatsheets}/rclone_cheat_sheet.md | 0 .../server_hardening_disaster_recovery.md | 0 .../cheatsheets}/zfs_cheat_sheet.md | 0 .../genesishosting/access/account-creation.md | 20 ++++ .../genesishosting/access/account-deletion.md | 13 +++ documents/genesishosting/access/least-priv.md | 20 ++++ documents/genesishosting/access/user-roles.md | 18 ++++ .../backups/backup-disaster-recovery.md | 26 +++++ .../backups/backup-integrity.md | 23 ++++ .../genesishosting/backups/backup-policy.md | 29 +++++ .../backups/dr/assets-mastodon-bucket.md | 45 ++++++++ .../backups/dr/assets_azuracast.md | 93 ++++++++++++++++ .../backups/restore-instructions.md | 32 ++++++ documents/genesishosting/clients/abuse.md | 27 +++++ .../clients/account-suspension.md | 22 ++++ documents/genesishosting/clients/aup.md | 27 +++++ .../clients/refunds-cancellations.md | 24 +++++ .../company/company-code-of-conduct.md | 20 ++++ .../company/company-mission-statement.md | 12 +++ .../genesishosting/company/company-tos.md | 25 +++++ documents/genesishosting/company/dmca.md | 25 +++++ .../genesishosting/company/privacy-policy.md | 26 +++++ .../disrec/zfsdestroycasestudy.md | 64 +++++++++++ documents/genesishosting/dns-check | 50 +++++++++ .../genesishosting/infra/genesis-shield.md | 24 +++++ .../infra/infra-maintenance-windows.md | 25 +++++ .../infra/infra-monitoring-setup.md | 25 +++++ .../infra/server-naming-convention.md | 19 ++++ .../genesishosting/infra/zfs-strategy.md | 23 ++++ .../master_compliance_checklist.md | 63 +++++++++++ .../genesishosting/pmgenesisiorealignment.md | 83 +++++++++++++++ .../genesishosting/provisioning/checklist.md | 23 ++++ .../provisioning/post-deploy-verification.md | 22 ++++ .../provisioning/whmcs-integration.md | 23 ++++ .../security/incident-response.md | 25 +++++ .../security/logging-monitoring.md | 24 +++++ .../security/security-encryption-standards.md | 23 ++++ .../security/security-policy.md | 23 ++++ .../services/azuracast-policy.md | 32 ++++++ .../services/directadmin-policy.md | 27 +++++ .../services/domain-management-policy.md | 22 ++++ .../genesishosting/services/ssl-certs.md | 23 ++++ .../services/teamtalk-policy.md | 26 +++++ .../incident_response.md | 0 {markdown => documents/procedures}/GROWL.md | 0 {markdown => documents/procedures}/OPS.md | 0 {markdown => documents/procedures}/decom.md | 0 .../procedures}/infrastructure.md | 0 {markdown => documents/procedures}/map.md | 0 .../mastodon}/mastodon-content-policy.md | 0 .../mastodon}/mastodon-maintenance-policy.md | 0 .../mastodon}/mastodon-moderation-policy.md | 0 .../mastodon}/mastodon-uptime-policy.md | 0 .../mastodon}/mastodon-user-policy.md | 0 .../procedures}/planned_db_cluster_ZFS.md | 0 {markdown => documents/procedures}/runv1.md | 0 postmortem/genesisradiozfsmigration.md | 68 ------------ postmortem/mastoobjectstorage.md | 75 ------------- postmortem/pmgenesisiorealignment.md | 100 ------------------ 59 files changed, 1196 insertions(+), 243 deletions(-) rename {cheatsheets => documents/cheatsheets}/rclone_cheat_sheet.md (100%) rename {cheatsheets => documents/cheatsheets}/server_hardening_disaster_recovery.md (100%) rename {cheatsheets => documents/cheatsheets}/zfs_cheat_sheet.md (100%) create mode 100644 documents/genesishosting/access/account-creation.md create mode 100644 documents/genesishosting/access/account-deletion.md create mode 100644 documents/genesishosting/access/least-priv.md create mode 100644 documents/genesishosting/access/user-roles.md create mode 100644 documents/genesishosting/backups/backup-disaster-recovery.md create mode 100644 documents/genesishosting/backups/backup-integrity.md create mode 100644 documents/genesishosting/backups/backup-policy.md create mode 100644 documents/genesishosting/backups/dr/assets-mastodon-bucket.md create mode 100644 documents/genesishosting/backups/dr/assets_azuracast.md create mode 100644 documents/genesishosting/backups/restore-instructions.md create mode 100644 documents/genesishosting/clients/abuse.md create mode 100644 documents/genesishosting/clients/account-suspension.md create mode 100644 documents/genesishosting/clients/aup.md create mode 100644 documents/genesishosting/clients/refunds-cancellations.md create mode 100644 documents/genesishosting/company/company-code-of-conduct.md create mode 100644 documents/genesishosting/company/company-mission-statement.md create mode 100644 documents/genesishosting/company/company-tos.md create mode 100644 documents/genesishosting/company/dmca.md create mode 100644 documents/genesishosting/company/privacy-policy.md create mode 100644 documents/genesishosting/disrec/zfsdestroycasestudy.md create mode 100644 documents/genesishosting/dns-check create mode 100644 documents/genesishosting/infra/genesis-shield.md create mode 100644 documents/genesishosting/infra/infra-maintenance-windows.md create mode 100644 documents/genesishosting/infra/infra-monitoring-setup.md create mode 100644 documents/genesishosting/infra/server-naming-convention.md create mode 100644 documents/genesishosting/infra/zfs-strategy.md create mode 100644 documents/genesishosting/master_compliance_checklist.md create mode 100644 documents/genesishosting/pmgenesisiorealignment.md create mode 100644 documents/genesishosting/provisioning/checklist.md create mode 100644 documents/genesishosting/provisioning/post-deploy-verification.md create mode 100644 documents/genesishosting/provisioning/whmcs-integration.md create mode 100644 documents/genesishosting/security/incident-response.md create mode 100644 documents/genesishosting/security/logging-monitoring.md create mode 100644 documents/genesishosting/security/security-encryption-standards.md create mode 100644 documents/genesishosting/security/security-policy.md create mode 100644 documents/genesishosting/services/azuracast-policy.md create mode 100644 documents/genesishosting/services/directadmin-policy.md create mode 100644 documents/genesishosting/services/domain-management-policy.md create mode 100644 documents/genesishosting/services/ssl-certs.md create mode 100644 documents/genesishosting/services/teamtalk-policy.md rename incident_response.md => documents/incident_response.md (100%) rename {markdown => documents/procedures}/GROWL.md (100%) rename {markdown => documents/procedures}/OPS.md (100%) rename {markdown => documents/procedures}/decom.md (100%) rename {markdown => documents/procedures}/infrastructure.md (100%) rename {markdown => documents/procedures}/map.md (100%) rename {mastodonmd => documents/procedures/mastodon}/mastodon-content-policy.md (100%) rename {mastodonmd => documents/procedures/mastodon}/mastodon-maintenance-policy.md (100%) rename {mastodonmd => documents/procedures/mastodon}/mastodon-moderation-policy.md (100%) rename {mastodonmd => documents/procedures/mastodon}/mastodon-uptime-policy.md (100%) rename {mastodonmd => documents/procedures/mastodon}/mastodon-user-policy.md (100%) rename {markdown => documents/procedures}/planned_db_cluster_ZFS.md (100%) rename {markdown => documents/procedures}/runv1.md (100%) delete mode 100644 postmortem/genesisradiozfsmigration.md delete mode 100644 postmortem/mastoobjectstorage.md delete mode 100644 postmortem/pmgenesisiorealignment.md diff --git a/cheatsheets/rclone_cheat_sheet.md b/documents/cheatsheets/rclone_cheat_sheet.md similarity index 100% rename from cheatsheets/rclone_cheat_sheet.md rename to documents/cheatsheets/rclone_cheat_sheet.md diff --git a/cheatsheets/server_hardening_disaster_recovery.md b/documents/cheatsheets/server_hardening_disaster_recovery.md similarity index 100% rename from cheatsheets/server_hardening_disaster_recovery.md rename to documents/cheatsheets/server_hardening_disaster_recovery.md diff --git a/cheatsheets/zfs_cheat_sheet.md b/documents/cheatsheets/zfs_cheat_sheet.md similarity index 100% rename from cheatsheets/zfs_cheat_sheet.md rename to documents/cheatsheets/zfs_cheat_sheet.md diff --git a/documents/genesishosting/access/account-creation.md b/documents/genesishosting/access/account-creation.md new file mode 100644 index 0000000..12fd857 --- /dev/null +++ b/documents/genesishosting/access/account-creation.md @@ -0,0 +1,20 @@ +# Account Creation Policy + +## Customer Accounts + +- Created automatically via WHMCS upon signup +- Email verification is required before service activation +- Strong passwords (minimum 10 characters) are enforced +- 2FA is recommended and required for admin-facing services + +## Staff/Admin Accounts + +- Created manually by Super Admin only +- Must use SSH keys for server access +- Access logs are enabled and monitored +- Each staff account must be linked to an internal email + +## Account Naming Convention + +- Customers: `client_{username}` +- Admins: `admin.{firstname}` diff --git a/documents/genesishosting/access/account-deletion.md b/documents/genesishosting/access/account-deletion.md new file mode 100644 index 0000000..71fd0df --- /dev/null +++ b/documents/genesishosting/access/account-deletion.md @@ -0,0 +1,13 @@ +# Account Deletion Policy + +## Customer Accounts + +- Users may request account deletion via WHMCS support ticket +- Data is retained for 30 days post-termination (unless legally required) +- Backups including user data are purged after 30 days + +## Internal Accounts + +- Deactivated immediately upon staff departure or role change +- SSH keys, DirectAdmin access, and database credentials revoked +- Logs associated with the account are retained for audit purposes diff --git a/documents/genesishosting/access/least-priv.md b/documents/genesishosting/access/least-priv.md new file mode 100644 index 0000000..00f85ac --- /dev/null +++ b/documents/genesishosting/access/least-priv.md @@ -0,0 +1,20 @@ +# Least Privilege Policy + +Genesis Hosting enforces least privilege access for all systems. + +## Principles + +- Users are given the minimum level of access necessary to perform their work +- Admin tools are isolated by function (e.g., billing vs. system access) +- Escalation of privileges must be requested, documented, and time-bound + +## Tools in Use + +- WHMCS permissions are restricted by group +- SSH access is limited using `AllowUsers` and firewalled IPs +- TeamTalk server admins are rotated and audited monthly + +## Review Cycle + +- Access roles are reviewed quarterly +- Logs of access changes are stored and rotated every 90 days diff --git a/documents/genesishosting/access/user-roles.md b/documents/genesishosting/access/user-roles.md new file mode 100644 index 0000000..0f485f3 --- /dev/null +++ b/documents/genesishosting/access/user-roles.md @@ -0,0 +1,18 @@ +# User Roles + +Genesis Hosting Technologies uses Role-Based Access Control (RBAC) to ensure that users only have access to what they need. + +## Role Definitions + +| Role | Description | Examples | +|----------------|----------------------------------------------------------|----------------------------------| +| Customer | End users with access to services they’ve purchased | DirectAdmin clients, Streamers | +| Support Staff | Limited admin functions for resolving client issues | Helpdesk, WHMCS support agents | +| Administrator | Full access to provision, maintain, and modify services | Infrastructure admins | +| Super Admin | Root-level access to all systems | Owner/Lead Engineer | + +## Guidelines + +- Roles are assigned during onboarding. +- Access levels are reviewed quarterly. +- No one should hold higher access than required for their duties. diff --git a/documents/genesishosting/backups/backup-disaster-recovery.md b/documents/genesishosting/backups/backup-disaster-recovery.md new file mode 100644 index 0000000..18b8d67 --- /dev/null +++ b/documents/genesishosting/backups/backup-disaster-recovery.md @@ -0,0 +1,26 @@ +# Disaster Recovery Plan + +Genesis Hosting is prepared to recover core systems from catastrophic failure. + +## Recovery Objectives + +- **RPO (Recovery Point Objective)**: 24 hours +- **RTO (Recovery Time Objective)**: 4 hours for customer services + +## Full Recovery Flow + +1. Triage the affected systems +2. Identify last successful backup or snapshot +3. Restore individual services: + - DNS + - WHMCS + - DirectAdmin + - AzuraCast + - TeamTalk +4. Run post-restore validation scripts +5. Notify customers of incident and resolution + +## DR Testing + +- Simulated quarterly +- Logs retained in `/var/log/genesisdr.log` diff --git a/documents/genesishosting/backups/backup-integrity.md b/documents/genesishosting/backups/backup-integrity.md new file mode 100644 index 0000000..ced96f2 --- /dev/null +++ b/documents/genesishosting/backups/backup-integrity.md @@ -0,0 +1,23 @@ +# Backup Integrity + +We verify all backups regularly to ensure they are complete, uncorrupted, and restorable. + +## Weekly Tasks + +- ZFS scrubs for all pools +- Hash checks (SHA-256) for tarballs and dumps +- rsync `--checksum` verification for remote mirrors + +## Alerts + +- Email/Mastodon alert if: + - ZFS reports checksum errors + - Scheduled backup is missing + - Remote sync fails or lags > 24h + +## Tools Used + +- `zfs scrub` +- `sha256sum` + custom validation script +- rclone sync logs +- Telegram bot and Genesis Shield notifications diff --git a/documents/genesishosting/backups/backup-policy.md b/documents/genesishosting/backups/backup-policy.md new file mode 100644 index 0000000..6bd0de0 --- /dev/null +++ b/documents/genesishosting/backups/backup-policy.md @@ -0,0 +1,29 @@ +# Backup Policy + +Genesis Hosting Technologies maintains regular backups to ensure customer data and internal infrastructure are recoverable in the event of failure, corruption, or disaster. + +## Backup Schedule + +| System | Frequency | Retention | Method | +|----------------|-----------|-----------|------------------| +| DirectAdmin | Daily | 7 Days | rsync + tarball | +| WHMCS | Daily | 14 Days | Encrypted dump | +| AzuraCast | Daily | 7 Days | Docker volume snapshot + config export | +| TeamTalk | Daily | 7 Days | XML + config archive | +| Full VMs | Weekly | 4 Weeks | ZFS snapshots or Proxmox backups | +| Offsite Backups| Weekly | 4 Weeks | Rsync to remote ZFS or object storage | + +## Retention Policy + +- Daily: 7 days +- Weekly: 4 weeks +- Monthly: Optional, for specific business data + +## Encryption + +- Backups are encrypted at rest (AES-256) +- Transfers to remote locations use SSH or TLS + +## Notes + +- No backup occurs on client plans marked "opt-out" diff --git a/documents/genesishosting/backups/dr/assets-mastodon-bucket.md b/documents/genesishosting/backups/dr/assets-mastodon-bucket.md new file mode 100644 index 0000000..6a36a15 --- /dev/null +++ b/documents/genesishosting/backups/dr/assets-mastodon-bucket.md @@ -0,0 +1,45 @@ +## 2025-05-02 22:24:25 – MinIO Bucket Access Configuration for Mastodon + +**Bucket**: `assets-mastodon` +**Server**: `shredderv2` +**User**: `genesisuser` +**Permissions**: Read / Write / Delete +**Policy Name**: `assets-mastodon-rw-policy` + +### Commands Executed: + +```bash +mc alias set localminio http://localhost:9000 genesisadmin MutationXv3! + +cat > assets_mastodon_rw_policy.json < assets_mastodon_rw_policy.json <24h), credit may be issued +- All refund requests are reviewed manually by support + +## How to Request + +Submit a WHMCS ticket with reason for refund diff --git a/documents/genesishosting/company/company-code-of-conduct.md b/documents/genesishosting/company/company-code-of-conduct.md new file mode 100644 index 0000000..5c2ca65 --- /dev/null +++ b/documents/genesishosting/company/company-code-of-conduct.md @@ -0,0 +1,20 @@ +# Code of Conduct + +We maintain a respectful, safe, and inclusive environment for both staff and clients. + +## Expectations + +- Treat all clients and team members with professionalism and courtesy +- Communicate clearly and constructively — even during escalations +- Uphold privacy, security, and transparency at every level +- Follow internal and customer-facing policies at all times + +## Zero Tolerance + +We do not tolerate: + +- Harassment or abuse (verbal, written, or otherwise) +- Discrimination based on identity, ability, or belief +- Intentional sabotage of infrastructure or service integrity + +Violations may result in immediate termination of access or service. diff --git a/documents/genesishosting/company/company-mission-statement.md b/documents/genesishosting/company/company-mission-statement.md new file mode 100644 index 0000000..8ad1643 --- /dev/null +++ b/documents/genesishosting/company/company-mission-statement.md @@ -0,0 +1,12 @@ +# Mission Statement + +At Genesis Hosting Technologies, our mission is to provide secure, reliable, and transparent hosting services with a personal touch. + +We believe that even the smallest teams deserve enterprise-grade infrastructure — without enterprise-grade headaches. + +Our goal is to deliver: + +- Fast, stable hosting environments +- Fair pricing with no upsell games +- Transparent policies and proactive support +- A commitment to data ownership and user privacy diff --git a/documents/genesishosting/company/company-tos.md b/documents/genesishosting/company/company-tos.md new file mode 100644 index 0000000..2cddbcb --- /dev/null +++ b/documents/genesishosting/company/company-tos.md @@ -0,0 +1,25 @@ +# Terms of Service (TOS) + +By using services from Genesis Hosting Technologies, you agree to the following terms: + +## Service Provision + +- Services are delivered as-is, with best-effort uptime and technical support +- Users must abide by our Acceptable Use Policy (AUP) +- Access may be suspended for abuse, non-payment, or security issues + +## Billing & Renewals + +- All services are billed monthly or annually +- Automatic renewal is enabled by default +- Invoices are due within 5 days of issue unless otherwise agreed + +## Termination + +- You may cancel at any time via WHMCS +- We reserve the right to suspend or terminate accounts that violate our policies + +## Liability + +- We are not liable for data loss, service interruptions, or indirect damages +- Backups are provided as a best-effort courtesy unless contractually guaranteed diff --git a/documents/genesishosting/company/dmca.md b/documents/genesishosting/company/dmca.md new file mode 100644 index 0000000..b6a4097 --- /dev/null +++ b/documents/genesishosting/company/dmca.md @@ -0,0 +1,25 @@ +# DMCA Policy + +Genesis Hosting Technologies complies with the Digital Millennium Copyright Act (DMCA). + +## Filing a Takedown Notice + +Email dmca@genesishostingtechnologies.com with: + +- Your contact information +- Description of the copyrighted work +- URL or IP address of the infringing content +- A statement of good faith belief +- A statement of accuracy and authority + +## What Happens Next + +- We review and respond within 48 hours +- The client is notified and given a chance to respond +- If no valid counter-notice is filed, content may be removed or suspended + +## Filing a Counter Notice + +Clients who believe their content was wrongly removed may submit a counter notice with similar contact and justification information. + +We will not tolerate repeated infringement and may terminate accounts accordingly. diff --git a/documents/genesishosting/company/privacy-policy.md b/documents/genesishosting/company/privacy-policy.md new file mode 100644 index 0000000..380aee0 --- /dev/null +++ b/documents/genesishosting/company/privacy-policy.md @@ -0,0 +1,26 @@ +# Privacy Policy + +We respect your privacy and protect your data. + +## What We Collect + +- Account information: name, email, billing address +- Service usage data: IPs, access logs, system metrics +- Communications: support tickets and emails + +## How We Use It + +- Service provisioning and support +- Abuse prevention and system integrity +- Internal analytics (not shared or sold) + +## Data Sharing + +- We do not sell user data +- We may share limited data with trusted providers (e.g., payment processors) +- Law enforcement requests must include valid legal process + +## Data Retention + +- User data is retained as long as the account is active +- Backups are purged per the Backup Policy diff --git a/documents/genesishosting/disrec/zfsdestroycasestudy.md b/documents/genesishosting/disrec/zfsdestroycasestudy.md new file mode 100644 index 0000000..aa330ec --- /dev/null +++ b/documents/genesishosting/disrec/zfsdestroycasestudy.md @@ -0,0 +1,64 @@ +# 📛 Case Study: Why RAID Is Not a Backup + +## Overview + +On May 4, 2025, we experienced a production data loss incident involving the `nexus` dataset on `shredderv1`, a Linux RAID5 server. Despite no hardware failure, critical files were lost due to an unintended command affecting live data. + +This incident serves as a clear, real-world illustration of the maxim: + +> **RAID protects against hardware failure — not human error, data corruption, or bad automation.** + +--- + +## 🔍 What Happened + +- `shredderv1` uses RAID5 for media storage. +- The dataset `nexus/miniodata` (housing `genesisassets`, `genesislibrary`, etc.) was accidentally destroyed. +- **No disks failed.** The failure was logical, not physical. + +--- + +## 🔥 Impact + +- StationPlaylist (SPL) lost access to the Genesis media library. +- MinIO bucket data was instantly inaccessible. +- Temporary outage and scrambling to reconfigure mounts, media, and streaming. + +--- + +## ✅ Recovery + +Thanks to our disaster recovery stack: + +- Nightly **rsync backups** were synced to **The Vault** (backup server). +- **ZFS snapshots** existed on The Vault for the affected datasets. +- We restored the latest snapshot **from The Vault back to Shredder**, effectively reversing the loss. +- No data corruption occurred; sync validation showed dataset integrity. + +--- + +## 🎓 Takeaway + +This is a live demonstration of why: + +- **RAID is not a backup** +- **Snapshots without off-host replication** are not enough +- **Real backups must be off-server and regularly tested** + +--- + +## 🔐 Current Protection Measures + +- Production data (`genesisassets`, `genesislibrary`) now replicated nightly to The Vault via `rsync`. +- ZFS snapshots are validated daily via a **dry-run restore validator**. +- Telegram alerts notify success/failure of backup verification jobs. +- Future goal: full ZFS storage on all production servers for native snapshot support. + +--- + +## 🧠 Lessons Learned + +- Always assume you'll delete the wrong thing eventually. +- Snapshots are amazing — **if** they're somewhere else. +- Automated restore testing should be part of every backup pipeline. + diff --git a/documents/genesishosting/dns-check b/documents/genesishosting/dns-check new file mode 100644 index 0000000..14f5537 --- /dev/null +++ b/documents/genesishosting/dns-check @@ -0,0 +1,50 @@ +# 🌐 DNS Access Issues – Troubleshooting Guide + +If you're having trouble reaching **Genesis Radio** or the stream won't load, the issue may be with your DNS provider (the service that turns domain names into IP addresses). + +This happens more often than you'd think — and it's easy to fix. + +--- + +## ✅ Quick Fix: Change Your DNS + +We recommend switching to one of these trusted, fast, privacy-respecting DNS providers: + +| Provider | DNS Servers | +|--------------|-----------------------------| +| **Google** | `8.8.8.8` and `8.8.4.4` | +| **Cloudflare** | `1.1.1.1` and `1.0.0.1` | +| **Quad9** | `9.9.9.9` | + +--- + +## 💻 How to Change Your DNS + +### Windows 10/11 +1. Open **Settings → Network & Internet** +2. Click **Change adapter options** +3. Right-click your active connection → **Properties** +4. Select **Internet Protocol Version 4 (TCP/IPv4)** → Click **Properties** +5. Choose **"Use the following DNS server addresses"** +6. Enter: + - Preferred: `1.1.1.1` + - Alternate: `8.8.8.8` +7. Save and reconnect + +--- + +### macOS +1. Go to **System Preferences → Network** +2. Select your active network → Click **Advanced** +3. Go to the **DNS** tab +4. Click `+` and add: + - `1.1.1.1` + - `8.8.8.8` +5. Apply changes and reconnect + +--- + +### Linux (CLI) +For a quick test: +```bash +sudo resolvectl dns eth0 1.1.1.1 8.8.8.8 diff --git a/documents/genesishosting/infra/genesis-shield.md b/documents/genesishosting/infra/genesis-shield.md new file mode 100644 index 0000000..853f6d9 --- /dev/null +++ b/documents/genesishosting/infra/genesis-shield.md @@ -0,0 +1,24 @@ +# Genesis Shield – Security & Threat Monitoring + +Genesis Shield is our custom-built alert and ban system, integrated across our infrastructure. + +## Features + +- Aggregates Fail2Ban logs across all VMs +- Bans pushed in real-time via Mastodon DM and Telegram +- Scripts track: + - Repeated SSH failures + - API abuse + - Web panel brute force attempts + +## Interfaces + +- Terminal dashboard for live bans/unbans +- Role-based control (root/admin only) +- Daily threat summary via Mastodon bot + +## Roadmap + +- WHMCS integration for abuse tickets +- Live threat map by country/IP +- REST API for admin toolkit diff --git a/documents/genesishosting/infra/infra-maintenance-windows.md b/documents/genesishosting/infra/infra-maintenance-windows.md new file mode 100644 index 0000000..0f48e77 --- /dev/null +++ b/documents/genesishosting/infra/infra-maintenance-windows.md @@ -0,0 +1,25 @@ +# Maintenance Window Policy + +To maintain consistency and reduce customer impact, we adhere to a strict maintenance schedule. + +## Standard Window + +- **Every Sunday, 7 PM – 9 PM Eastern** +- Non-emergency changes must occur during this window + +## What’s Allowed + +- OS & kernel updates +- Docker/image upgrades +- ZFS snapshots & cleanup +- Rolling restarts of containers + +## Emergencies + +- Critical security patches can bypass the window +- All emergency changes must be logged and reviewed + +## Notifications + +- Posted on Mastodon at least 1 hour before the window begins +- Clients notified via WHMCS if it will affect their service diff --git a/documents/genesishosting/infra/infra-monitoring-setup.md b/documents/genesishosting/infra/infra-monitoring-setup.md new file mode 100644 index 0000000..e0f6c16 --- /dev/null +++ b/documents/genesishosting/infra/infra-monitoring-setup.md @@ -0,0 +1,25 @@ +# Monitoring Setup + +We use a layered monitoring approach to ensure full visibility and rapid response. + +## Stack + +- **Prometheus** for metrics collection +- **Grafana** for visualization dashboards +- **Fail2Ban** for intrusion attempts +- **Genesis Shield** for aggregated alerts (Telegram + Mastodon) + +## What We Monitor + +| System | Metric Examples | +|----------------|--------------------------------------------| +| PostgreSQL | Replication lag, disk usage, active queries | +| Web Servers | HTTP response time, TLS errors | +| MinIO / Assets | Cache hit ratio, sync status | +| Docker Hosts | Container uptime, memory pressure | + +## Alerting + +- Telegram: Real-time infra alerts +- Mastodon bot: Daily summaries and status posts +- Fallback email alerts for critical failures diff --git a/documents/genesishosting/infra/server-naming-convention.md b/documents/genesishosting/infra/server-naming-convention.md new file mode 100644 index 0000000..0097b1c --- /dev/null +++ b/documents/genesishosting/infra/server-naming-convention.md @@ -0,0 +1,19 @@ +# Server Naming Convention + +To reduce confusion and improve clarity, we follow a clear and themed naming structure. + +## Naming Style + +Examples: + +- `krang.internal` – Master backend server +- `replica.db3.sshjunkie.com` – Staging PostgreSQL replica +- `shredderv2` – ZFS backup server +- `anthony` – Ansible control node +- `nexus` – Main ZFS pool server for assets + +## Guidelines + +- Avoid generic names (`server1`, `host123`) +- Use themed names (e.g., TMNT characters for core infrastructure) +- Include environment tags where needed (`-test`, `-prod`) diff --git a/documents/genesishosting/infra/zfs-strategy.md b/documents/genesishosting/infra/zfs-strategy.md new file mode 100644 index 0000000..a69a1fa --- /dev/null +++ b/documents/genesishosting/infra/zfs-strategy.md @@ -0,0 +1,23 @@ +# ZFS Strategy + +ZFS is used across Genesis Hosting Technologies for performance, integrity, and snapshot-based backup operations. + +## Pool Layout + +- RAIDZ1 or mirrored vdevs depending on use case +- Dataset naming: `genesisassets-secure`, `genesisshows-secure`, etc. +- Dedicated pools for: + - Mastodon media + - Client backups + - Internal scripts and logs + +## Snapshots + +- Hourly: last 24 hours +- Daily: last 7 days +- Weekly: last 4 weeks + +## Send/Receive + +- Used for offsite replication to Servarica and backup nodes +- Verified using checksums and `zfs receive -F` diff --git a/documents/genesishosting/master_compliance_checklist.md b/documents/genesishosting/master_compliance_checklist.md new file mode 100644 index 0000000..10485bc --- /dev/null +++ b/documents/genesishosting/master_compliance_checklist.md @@ -0,0 +1,63 @@ +# ✅ Master Compliance Checklist +*(Status: ☐ = Not Started | 🟨 = In Progress | ✅ = Complete)* + +## 🧑‍💼 Access & User Management +- [ ] Role-Based Access Control (RBAC) in place (Customer, Admin, etc.) +- [ ] Account creation follows secure onboarding workflows +- [ ] Admin access restricted to SSH keys only +- [ ] Inactive accounts locked or removed quarterly +- [ ] Least privilege enforced across all services + +## 💾 Backups & Disaster Recovery +- [ ] Daily backups enabled for all key services (DirectAdmin, WHMCS, AzuraCast, TeamTalk) +- [ ] Weekly offsite backups with verification +- [ ] ZFS snapshots scheduled (hourly/daily/weekly) +- [ ] Backup integrity validated with checksums or scrubs +- [ ] Quarterly disaster recovery drill completed +- [ ] Restore instructions documented and tested + +## 🔐 Security +- [ ] 2FA enabled on all admin interfaces (WHMCS, SSH, DirectAdmin) +- [ ] SSH password auth disabled, key-only enforced +- [ ] Weekly patching or updates scheduled (Sunday 7–9 PM) +- [ ] Centralized logging active and stored 30–90 days +- [ ] Fail2Ban + Genesis Shield integrated and alerting +- [ ] TLS 1.2+ enforced for all public services +- [ ] AES-256 encryption at rest on backups and sensitive volumes + +## 🖥️ Provisioning & Automation +- [ ] WHMCS integrated with DirectAdmin, AzuraCast, TeamTalk +- [ ] All provisioning scripts tested and logged +- [ ] Post-deploy verification checklist followed +- [ ] DNS + SSL automation in place (Let's Encrypt) +- [ ] Monitoring added after provisioning (Prometheus/Grafana) + +## 📋 Client Policies +- [ ] Acceptable Use Policy posted and enforced +- [ ] Abuse response process defined and working +- [ ] DMCA policy publicly available and followed +- [ ] Suspension and refund rules defined in WHMCS +- [ ] Privacy Policy and Terms of Service available on client portal + +## 🌐 Services Configuration +- [ ] DirectAdmin quotas enforced (disk, bandwidth, email) +- [ ] AzuraCast listener/storage/bitrate limits respected +- [ ] TeamTalk server abuse protection and user limits enforced +- [ ] Domain registration/renewal workflows tested +- [ ] SSL auto-renew working correctly (Let's Encrypt + certbot) + +## ⚙️ Infrastructure +- [ ] ZFS pools configured for redundancy (RAIDZ1, mirrors) +- [ ] rclone mount points with caching working and monitored +- [ ] Genesis Shield actively alerting via Telegram/Mastodon +- [ ] All VMs named per convention (e.g., `krang`, `shredderv2`) +- [ ] Sunday maintenance window consistently followed +- [ ] Ansible playbooks used for provisioning/config consistency + +## 🛠️ Tools & Scripts +- [ ] All scripts version-controlled and documented +- [ ] Backups and restore tools tested and working +- [ ] Mastodon alert bot operating with secure tokens +- [ ] Rclone VFS stats monitored regularly +- [ ] Admin tools accessible only by authorized users +""" diff --git a/documents/genesishosting/pmgenesisiorealignment.md b/documents/genesishosting/pmgenesisiorealignment.md new file mode 100644 index 0000000..8789bca --- /dev/null +++ b/documents/genesishosting/pmgenesisiorealignment.md @@ -0,0 +1,83 @@ +# Postmortem: Genesis I/O Realignment + +**Date:** May 8, 2025 +**Author:** Doc +**Systems Involved:** minioraid5, shredder, chatwithus.live, zcluster.technodrome1/2, thevault +**Scope:** Local-first mirroring, permission normalization, MinIO transition + +--- + +## 🎯 Objective + +To realign the Genesis file flow architecture by: + +- Making local block storage the **primary source** of truth for AzuraCast and Genesis buckets +- Transitioning FTP uploads to target local storage instead of MinIO directly +- Establishing **two-way mirroring** between local paths and MinIO buckets +- Correcting inherited permission issues across `/mnt/raid5` using `find + chmod` +- Preserving MinIO buckets as **backup mirrors**, not primary data stores + +--- + +## 🔧 Work Performed + +### ✅ Infrastructure changes: +- Deployed block storage volume to Linode Mastodon instance +- Mirrored MinIO buckets (`genesisassets`, `genesislibrary`, `azuracast`) to local paths +- Configured cron-based `mc mirror` jobs: + - Local ➜ MinIO: every 5 minutes with `--overwrite --remove` + - MinIO ➜ Local: nightly pull, no `--remove` + +### ✅ FTP Pipeline Adjustments: +- Users now upload to `/mnt/spl/ftp/uploads` (local) +- Permissions set so only admins access full `/mnt/spl/ftp` +- FTP directory structure created for SPL automation + +### ✅ System Tuning: +- Set `vm.swappiness=10` on all nodes +- Apache disabled where not in use +- Daily health checks via `pull_health_everywhere.sh` +- Krang Telegram alerts deployed for cleanup and system state + +--- + +## 🧠 Observations + +- **High load** on `minioraid5` during `mc mirror` and `chmod` overlap + - Load ~6.5 due to concurrent I/O pressure + - `chmod` stuck in `D` state (I/O wait) while `mc` dominated disk queues + - Resolved after `mc` completion — `chmod` resumed and completed + +- **MinIO buckets were temporarily inaccessible** due to permissions accidentally inherited by FTP group + - Resolved by recursively resetting permissions on `/mnt/raid5` + +- **Krang telemetry** verified: + - Mastodon swap usage rising under asset load + - All nodes had Apache disabled or dormant + - Health alerts triggered on high swap or load + +--- + +## ✅ Outcome + +- Full Genesis and AzuraCast data now reside locally with resilient S3 mirrors +- Mastodon running on block storage, no longer dependent on MinIO latency +- FTP integration with SPL directory trees complete +- Cleanup script successfully deployed across all nodes via Krang +- Daily health reports operational with alerts for high swap/load + +--- + +## 🔁 Recommendations + +- Consider adding snapshot-based ZFS backups for `/mnt/raid5` +- Build `verify_mirror.sh` to detect drift between MinIO and local storage +- Auto-trigger `chmod` only after `mc mirror` finishes +- Monitor long-running background jobs with Krang watchdogs + +--- + +**Signed,** +Doc +Genesis Hosting Technologies + diff --git a/documents/genesishosting/provisioning/checklist.md b/documents/genesishosting/provisioning/checklist.md new file mode 100644 index 0000000..13741d3 --- /dev/null +++ b/documents/genesishosting/provisioning/checklist.md @@ -0,0 +1,23 @@ +# Provisioning Checklist + +This checklist is followed every time a new service is deployed. + +## Pre-Provisioning + +- [ ] Verify order and payment in WHMCS +- [ ] Confirm product mapping is correct +- [ ] Check available server resources + +## Provisioning + +- [ ] Trigger appropriate script/module +- [ ] Log provisioning result +- [ ] Assign DNS entries if applicable +- [ ] Generate Let’s Encrypt SSL if public-facing + +## Post-Provisioning + +- [ ] Send welcome email via WHMCS +- [ ] Confirm monitoring alert is active +- [ ] Test login credentials and endpoints +- [ ] Label service with client ID in Grafana/Prometheus diff --git a/documents/genesishosting/provisioning/post-deploy-verification.md b/documents/genesishosting/provisioning/post-deploy-verification.md new file mode 100644 index 0000000..d46f727 --- /dev/null +++ b/documents/genesishosting/provisioning/post-deploy-verification.md @@ -0,0 +1,22 @@ +# Post-Deployment Verification + +All services go through a post-deploy QA check to ensure they're live and stable. + +## Verification Tasks + +- [ ] Service reachable from public IP or internal route +- [ ] DNS resolves correctly (for domains/subdomains) +- [ ] SSL certificate is active and trusted +- [ ] Admin login works as expected +- [ ] Usage quotas correctly applied (disk, users, bandwidth) + +## Monitoring + +- [ ] Add to Prometheus for service-specific metrics +- [ ] Set alert thresholds (e.g., disk > 80%) +- [ ] Confirm Telegram/Mastodon alert webhook is functional + +## Documentation + +- [ ] Log final status in WHMCS admin notes +- [ ] Store internal service details in `genesis-inventory.yaml` diff --git a/documents/genesishosting/provisioning/whmcs-integration.md b/documents/genesishosting/provisioning/whmcs-integration.md new file mode 100644 index 0000000..3f5663d --- /dev/null +++ b/documents/genesishosting/provisioning/whmcs-integration.md @@ -0,0 +1,23 @@ +# WHMCS Integration + +WHMCS handles client billing, service provisioning, and support workflows. + +## Services Integrated + +| Service | Method | +|--------------|---------------------------------| +| DirectAdmin | Built-in WHMCS module | +| AzuraCast | Custom provisioning script | +| TeamTalk | API + XML user patching scripts | + +## Auto-Provisioning Steps + +1. Client signs up and completes payment +2. WHMCS triggers product-specific hook +3. Script/module provisions the service +4. Welcome email is sent with credentials + +## Logging & Troubleshooting + +- Logs stored at `/var/log/whmcs-hooks.log` +- Errors generate internal ticket automatically if provisioning fails diff --git a/documents/genesishosting/security/incident-response.md b/documents/genesishosting/security/incident-response.md new file mode 100644 index 0000000..29f7ce5 --- /dev/null +++ b/documents/genesishosting/security/incident-response.md @@ -0,0 +1,25 @@ +# Incident Response Policy + +This document defines how we detect, respond to, and report security incidents. + +## Response Workflow + +1. Detection via monitoring, alert, or client report +2. Triage severity and affected systems +3. Contain and isolate threat (e.g., suspend access) +4. Notify stakeholders if client-impacting +5. Perform root cause analysis +6. Patch, re-secure, and document the event + +## Timelines + +- Initial triage: within 2 hours +- Client notification (if impacted): within 24 hours +- Final report delivered internally within 72 hours + +## Tools Used + +- Fail2Ban +- Genesis Shield alerting +- Zabbix/Prometheus incident flags +- Manual log reviews (forensic-level) diff --git a/documents/genesishosting/security/logging-monitoring.md b/documents/genesishosting/security/logging-monitoring.md new file mode 100644 index 0000000..c305627 --- /dev/null +++ b/documents/genesishosting/security/logging-monitoring.md @@ -0,0 +1,24 @@ +# Logging & Monitoring Policy + +We collect and monitor system activity to detect threats, enforce accountability, and assist in incident resolution. + +## Log Types + +- SSH login attempts +- WHMCS access logs +- AzuraCast and TeamTalk server logs +- PostgreSQL query and connection logs +- Fail2Ban logs (ban/unban events) + +## Monitoring Tools + +- Prometheus for metrics +- Grafana dashboards for visual alerts +- Genesis Shield (Telegram + Mastodon alerting) +- Manual log review every 7 days + +## Retention + +- General logs: 30 days +- Security-related logs: 90 days minimum +- Logs archived to encrypted ZFS volume diff --git a/documents/genesishosting/security/security-encryption-standards.md b/documents/genesishosting/security/security-encryption-standards.md new file mode 100644 index 0000000..6d9139c --- /dev/null +++ b/documents/genesishosting/security/security-encryption-standards.md @@ -0,0 +1,23 @@ +# Encryption Standards + +Encryption is applied to all data in transit and at rest across Genesis Hosting Technologies infrastructure. + +## In Transit + +- HTTPS via TLS 1.3 (minimum TLS 1.2 for legacy fallback) +- SFTP for all file transfers +- SSH for all administrative access +- rclone with TLS for object storage replication + +## At Rest + +- ZFS encryption on backup pools +- PostgreSQL encryption at the database or filesystem level +- WHMCS and DirectAdmin credentials hashed and salted +- Backups encrypted with AES-256 before remote transfer + +## Key Management + +- SSH keys rotated every 6 months +- Let's Encrypt certs auto-renew every 90 days +- Master encryption keys stored offline and version-controlled diff --git a/documents/genesishosting/security/security-policy.md b/documents/genesishosting/security/security-policy.md new file mode 100644 index 0000000..7ed282f --- /dev/null +++ b/documents/genesishosting/security/security-policy.md @@ -0,0 +1,23 @@ +# Security Policy + +Genesis Hosting Technologies enforces strict security practices across all infrastructure and services to protect client data and maintain service integrity. + +## Core Principles + +- Least privilege for all users and services +- Regular audits and patching +- Encrypted communication and storage +- Real-time monitoring and alerting + +## Enforcement Areas + +- 2FA required for all admin portals +- SSH access limited to key-based logins +- Centralized log collection and review +- All critical assets monitored via Genesis Shield + +## Review Cycle + +- Policies reviewed quarterly +- Logs retained for 30–90 days depending on system +- Incidents reviewed post-mortem with improvements logged diff --git a/documents/genesishosting/services/azuracast-policy.md b/documents/genesishosting/services/azuracast-policy.md new file mode 100644 index 0000000..3bf3fc4 --- /dev/null +++ b/documents/genesishosting/services/azuracast-policy.md @@ -0,0 +1,32 @@ +# AzuraCast Streaming Policy + +## Features + +- Custom stream URLs (via relay or direct) +- Icecast or SHOUTcast available +- AutoDJ + scheduled playlists +- Web-based file upload + schedule + +## Plans & Limits + +| Plan | Storage | Listeners | Bitrate | +|----------|---------|-----------|---------| +| StreamLite | 2 GB | 25 | 128 kbps| +| StreamPro | 10 GB | 100 | 192 kbps| +| StreamMax | 50 GB | 250 | 320 kbps| + +## Fair Usage Policy + +- No nonstop streaming of static loops to inflate uptime +- Long-form live shows should rotate metadata periodically +- Content must not violate copyright laws + +## Backups + +- Daily backups of config + playlists +- Client media backup is optional (paid add-on) + +## Support + +- Stream diagnostics available in client panel +- WHMCS ticket support for outages or playlist issues diff --git a/documents/genesishosting/services/directadmin-policy.md b/documents/genesishosting/services/directadmin-policy.md new file mode 100644 index 0000000..7d238b2 --- /dev/null +++ b/documents/genesishosting/services/directadmin-policy.md @@ -0,0 +1,27 @@ +# DirectAdmin Hosting Policy + +## Features + +- FTP, webmail, MySQL, file manager, and site statistics +- Optional Let's Encrypt SSL enabled by default +- Nightly site + database backups (7-day retention) + +## Plans & Limits + +| Plan | Disk | Bandwidth | Domains | Email Accounts | +|------------|------|-----------|---------|----------------| +| Starter | 5 GB | 100 GB | 1 | 5 | +| Standard | 20 GB| 500 GB | 5 | 25 | +| Unlimited | 100 GB| ∞ | ∞ | ∞ | + +## Abuse Prevention + +- Email rate limits applied to prevent outbound spam +- CPU usage and inode caps enforced +- Suspicious files scanned automatically + +## Support + +- Available via WHMCS ticket system +- Response within 12 business hours + diff --git a/documents/genesishosting/services/domain-management-policy.md b/documents/genesishosting/services/domain-management-policy.md new file mode 100644 index 0000000..088a009 --- /dev/null +++ b/documents/genesishosting/services/domain-management-policy.md @@ -0,0 +1,22 @@ +# Domain Management Policy + +## Registration + +- Domains registered through our WHMCS interface are managed via third-party registrar API +- Registration typically completes within 5 minutes +- WHOIS privacy included by default (where available) + +## Renewals + +- Auto-renew is enabled by default +- Reminders sent 30, 7, and 1 day before expiration + +## Transfers + +- Domains can be transferred in or out with EPP code +- Support required if domain is locked or expired + +## DNS + +- Free DNS hosting included +- Custom DNS records managed through DirectAdmin or WHMCS panel diff --git a/documents/genesishosting/services/ssl-certs.md b/documents/genesishosting/services/ssl-certs.md new file mode 100644 index 0000000..f6e275c --- /dev/null +++ b/documents/genesishosting/services/ssl-certs.md @@ -0,0 +1,23 @@ +# SSL Certificate Policy + +## Free Certificates + +- Let’s Encrypt certificates issued automatically +- Applies to DirectAdmin, AzuraCast, and custom subdomains +- Auto-renews every 60 days with 30-day buffer + +## Premium SSL + +- Custom SSL certs (e.g., EV/OV) available for purchase +- Requires manual install via WHMCS ticket + +## Certificate Management + +- Certbot used for automation +- Custom certs must be supplied in `.crt` + `.key` format +- Broken SSL installs may be reverted to Let’s Encrypt fallback + +## Support + +- Certificate issues resolved within 24h of report +- DNS challenges supported for wildcard certs diff --git a/documents/genesishosting/services/teamtalk-policy.md b/documents/genesishosting/services/teamtalk-policy.md new file mode 100644 index 0000000..4d2e3ff --- /dev/null +++ b/documents/genesishosting/services/teamtalk-policy.md @@ -0,0 +1,26 @@ +# TeamTalk Hosting Policy + +## Features + +- Private and public servers +- Voice chat, file sharing, push-to-talk +- Admin access with room/channel management + +## Plans & Limits + +| Plan | Users | Bitrate Limit | Admin Access | +|--------------|-------|---------------|--------------| +| Basic Chat | 10 | 64 kbps | Yes | +| Pro Voice | 50 | 128 kbps | Yes | +| Broadcast+ | 100 | 256 kbps | Yes | + +## Rules + +- No harassment, spamming, or automated bots without permission +- Abuse may result in temp suspension or permanent ban +- Admins are responsible for moderating their own servers + +## Configuration + +- Clients may request config changes via WHMCS ticket +- Backups of XML configs stored nightly diff --git a/incident_response.md b/documents/incident_response.md similarity index 100% rename from incident_response.md rename to documents/incident_response.md diff --git a/markdown/GROWL.md b/documents/procedures/GROWL.md similarity index 100% rename from markdown/GROWL.md rename to documents/procedures/GROWL.md diff --git a/markdown/OPS.md b/documents/procedures/OPS.md similarity index 100% rename from markdown/OPS.md rename to documents/procedures/OPS.md diff --git a/markdown/decom.md b/documents/procedures/decom.md similarity index 100% rename from markdown/decom.md rename to documents/procedures/decom.md diff --git a/markdown/infrastructure.md b/documents/procedures/infrastructure.md similarity index 100% rename from markdown/infrastructure.md rename to documents/procedures/infrastructure.md diff --git a/markdown/map.md b/documents/procedures/map.md similarity index 100% rename from markdown/map.md rename to documents/procedures/map.md diff --git a/mastodonmd/mastodon-content-policy.md b/documents/procedures/mastodon/mastodon-content-policy.md similarity index 100% rename from mastodonmd/mastodon-content-policy.md rename to documents/procedures/mastodon/mastodon-content-policy.md diff --git a/mastodonmd/mastodon-maintenance-policy.md b/documents/procedures/mastodon/mastodon-maintenance-policy.md similarity index 100% rename from mastodonmd/mastodon-maintenance-policy.md rename to documents/procedures/mastodon/mastodon-maintenance-policy.md diff --git a/mastodonmd/mastodon-moderation-policy.md b/documents/procedures/mastodon/mastodon-moderation-policy.md similarity index 100% rename from mastodonmd/mastodon-moderation-policy.md rename to documents/procedures/mastodon/mastodon-moderation-policy.md diff --git a/mastodonmd/mastodon-uptime-policy.md b/documents/procedures/mastodon/mastodon-uptime-policy.md similarity index 100% rename from mastodonmd/mastodon-uptime-policy.md rename to documents/procedures/mastodon/mastodon-uptime-policy.md diff --git a/mastodonmd/mastodon-user-policy.md b/documents/procedures/mastodon/mastodon-user-policy.md similarity index 100% rename from mastodonmd/mastodon-user-policy.md rename to documents/procedures/mastodon/mastodon-user-policy.md diff --git a/markdown/planned_db_cluster_ZFS.md b/documents/procedures/planned_db_cluster_ZFS.md similarity index 100% rename from markdown/planned_db_cluster_ZFS.md rename to documents/procedures/planned_db_cluster_ZFS.md diff --git a/markdown/runv1.md b/documents/procedures/runv1.md similarity index 100% rename from markdown/runv1.md rename to documents/procedures/runv1.md diff --git a/postmortem/genesisradiozfsmigration.md b/postmortem/genesisradiozfsmigration.md deleted file mode 100644 index d2886f3..0000000 --- a/postmortem/genesisradiozfsmigration.md +++ /dev/null @@ -1,68 +0,0 @@ -# 🔧 Post-Mortem: Genesis Radio Storage Migration -**Date:** April 30, 2025 -**Prepared by:** Doc -**Systems Affected:** StationPlaylist (SPL), Voice Tracker, Genesis Radio media backend - ---- - -## 🧠 Executive Summary - -Genesis Radio’s backend was migrated from a legacy MinIO instance using local disk (ext4) to a new **ZFS-based, encrypted MinIO deployment on `shredderv2`**. This change was driven by a need for more stable performance, improved security, and a cleaner storage architecture with proper bucket separation. - -This migration was completed **without touching production** until final validation, and all critical services remained online throughout the transition. We also revamped the rclone caching strategy to reduce freeze-ups and playback hiccups. - ---- - -## ✅ What We Did - -- Created **three new secure buckets**: `genesislibrary-secure`, `genesisassets-secure`, and `genesisshows-secure` -- Migrated data from backup server using `rclone sync`: - - `genesislibrary` came directly from backup - - `genesisassets` and `genesisshows` were pulled from the same bucket, with de-duping and cleanup to be completed post-migration -- Retained **original SPL drive letters** (`Q:\\`, `R:\\`) to avoid changes to the playout config -- Switched rclone mounts to point to the new secure buckets, with **aggressive VFS caching** using SSD-backed cache directories -- Took a clean **ZFS snapshot** (`@pre-s3-switch`) before switching over -- Confirmed no regression in SPL, VT Tracker, or streaming audio - ---- - -## ⚙️ Technical Improvements - -- **VFS caching overhaul**: - - Increased read-ahead (`1G`), lowered write-back wait - - Split cache between `X:\\librarycache` and `L:\\assetcache` - - No more rclone choking on large files or freezing during transitions -- **Encrypted S3 storage** with isolated buckets per functional role -- **TLS-secured** Console and MinIO endpoints with automated renewal -- Mounted buckets at startup via batch script (future systemd equivalents to be implemented) -- Snapshot-based rollback in ZFS enabled post-deployment resilience - ---- - -## 🩹 What Went Weird (and We Fixed It) - -- SPL froze during initial `mc mirror` attempts — solution: switched to `rclone`, which performed exponentially faster -- Some hiccups during early cache tuning, including sparse file support issues — solved by switching to ZFS -- Missing media files in Mastodon were traced to uploads during sync; resolved with staged sync + retry before final switch -- Certbot automation wasn’t configured — resolved with a systemd timer that stops nginx, renews, and restarts nginx automatically - ---- - -## 🧯 What We Learned - -- MinIO is solid, but **rclone wins for bulk sync performance** -- VFS cache settings **make or break** media-heavy workloads like SPL -- ZFS is a game-changer: no sparse file errors, reliable snapshots, clean rollback -- Planning matters: pre-syncing from backup avoided downtime -- Not touching prod until ready keeps stress and screwups to a minimum - ---- - -## 📦 Next Steps - -- [ ] Clean `genesisassets-secure` of misplaced show files -- [ ] Sync `azuracast` from live system (no backup copy yet) -- [ ] Build automated snapshot send-to-backup workflow (`zfs send | ssh backup zfs recv`) -- [ ] Stage full failover simulation (optional but fun) - - diff --git a/postmortem/mastoobjectstorage.md b/postmortem/mastoobjectstorage.md deleted file mode 100644 index 6bbae4a..0000000 --- a/postmortem/mastoobjectstorage.md +++ /dev/null @@ -1,75 +0,0 @@ -# 🧾 Postmortem: Mastodon Object Storage Migration to Secure S3 (MinIO) -**Date:** April 30, 2025 -**Engineer:** Doc (Genesis Radio / Genesis Hosting) - ---- - -## 🎯 Objective - -Migrate Mastodon's object storage from an older MinIO bucket (`linodeassets`) to a new **ZFS-backed, encrypted** MinIO instance (`mastodonassets-secure`) on `shredderv2`, while maintaining uptime and improving storage performance and security. - ---- - -## 🧱 Infrastructure Touched - -- Mastodon (Docker-based, hosted on Linode) -- MinIO S3 Object Storage (`oldminio` → `secureminio`) -- Nginx (reverse proxy for Console + S3 endpoints) -- ZFS pool: `nexus/mastodonassets` -- Domains: - - `shredderv2.sshjunkie.com` (S3 API) - - `consolev2.sshjunkie.com` (MinIO Console UI) - ---- - -## ⚠️ Issues Encountered - -1. **403 Access Denied on Mastodon startup** - - ✅ Root cause: `genesisadminv2` MinIO user had no attached policy - - 🔧 Fixed via Console UI after re-enabling access - -2. **MinIO Console unreachable (`consolev2.sshjunkie.com`)** - - SSL cert for the domain was missing - - 🔧 Used `certbot certonly --standalone` to issue new cert, re-enabled full HTTPS proxy - -3. **Sync race conditions** - - Some media files were uploaded to the old bucket during the long transfer - - 🔧 Mitigated by running an additional `rclone sync` pass before cutover - -4. **Rclone performance bottlenecks** - - MinIO client (`mc mirror`) was too slow - - ✅ Switched to `rclone`, saw drastic speed improvement - -5. **SPL (StationPlaylist) freezing during asset access** - - Root cause: cache choking on sparse file writes under ext4 - - ✅ Fix: moved critical rclone mounts to ZFS-backed drives - ---- - -## ✅ Success Criteria Met - -- 🔒 All Mastodon assets are now stored in `mastodonassets-secure` with encryption -- 🪣 MinIO Console functional on `https://consolev2.sshjunkie.com` -- 🎯 Mastodon is running with zero visible user impact -- 💾 Snapshot (`nexus/mastodonassets@pre-s3-switch`) taken post-migration for rollback -- 🔁 Future syncs can now be performed cleanly from backup server instead of live system - ---- - -## 🧠 Lessons Learned - -- Always validate MinIO user policies before go-live -- Avoid redirects in `server_name` blocks during cert issuance -- ZFS dramatically improves caching performance with rclone VFS -- Post-cutover syncs are crucial for active upload systems like Mastodon -- UI access to MinIO is a lifesaver for emergency fixes — keep it working - ---- - -## 🔚 Follow-Up Actions - -- [ ] Schedule `certbot renew --standalone` with systemd timer -- [ ] Rotate MinIO user keys and audit access policies -- [ ] Monitor `/var/log/syslog` for VFS or sparse file errors -- [ ] Document your rclone mount and caching strategy for SPL and Mastodon - diff --git a/postmortem/pmgenesisiorealignment.md b/postmortem/pmgenesisiorealignment.md deleted file mode 100644 index fc9b83d..0000000 --- a/postmortem/pmgenesisiorealignment.md +++ /dev/null @@ -1,100 +0,0 @@ -# Postmortem: Genesis I/O Realignment - -**Date:** May 8, 2025 -**Author:** Doc -**Systems Involved:** minioraid5, shredder, chatwithus.live, zcluster.technodrome1/2, thevault -**Scope:** Local-first mirroring, permission normalization, MinIO transition - ---- - -## 🎯 Objective - -To realign the Genesis file flow architecture by: - -* Making local block storage the **primary source** of truth for AzuraCast and Genesis buckets -* Transitioning FTP uploads to target local storage instead of MinIO directly -* Establishing **two-way mirroring** between local paths and MinIO buckets -* Correcting inherited permission issues across `/mnt/raid5` using `find + chmod` -* Preserving MinIO buckets as **backup mirrors**, not primary data stores - ---- - -## 🔧 Work Performed - -### ✅ Infrastructure changes: - -* Deployed block storage volume to Linode Mastodon instance -* Mirrored MinIO buckets (genesisassets, genesislibrary, azuracast) to local paths -* Configured cron-based `mc mirror` jobs: - - * Local ➜ MinIO: every 5 minutes with `--overwrite --remove` - * MinIO ➜ Local: nightly pull, no `--remove` -* Prepared 5TB local drive for AzuraCast asset mirroring (pending full sync) - -### ✅ FTP Pipeline Adjustments: - -* Users now upload to `/mnt/spl/ftp/uploads` (local) -* Permissions set so only admins access full `/mnt/spl/ftp` -* FTP directory structure created for SPL automation - -### ✅ System Tuning: - -* Set `vm.swappiness=10` on all nodes -* Apache disabled where not in use -* Daily health checks via `pull_health_everywhere.sh` -* Krang Telegram alerts deployed for cleanup and system state - ---- - -## 🧠 Observations - -* **High load** on `minioraid5` during `mc mirror` and `chmod` overlap - - * Load \~6.5 due to concurrent I/O pressure - * `chmod` stuck in `D` state (I/O wait) while `mc` dominated disk queues - * Resolved after `mc` completion — `chmod` resumed and completed - -* **MinIO buckets were temporarily inaccessible** due to permissions accidentally inherited by FTP group - - * Resolved by recursively resetting permissions on `/mnt/raid5` - -* **Krang telemetry** verified: - - * Mastodon swap usage rising under asset load - * All nodes had Apache disabled or dormant - * Health alerts triggered on high swap or load - ---- - -## ✅ Outcome - -* Full Genesis and AzuraCast data now reside locally with resilient S3 mirrors -* Mastodon running on block storage, no longer dependent on MinIO latency -* FTP integration with SPL directory trees complete -* Cleanup script successfully deployed across all nodes via Krang -* Daily health reports operational with alerts for high swap/load - ---- - -## 🔁 Recommendations - -* Proceed with AzuraCast mirror only after: - - * Mastodon asset storage transition is confirmed stable - * All `/mnt/raid5` permission fixes are complete - -* Consider adding snapshot-based ZFS backups for `/mnt/raid5` - -* Build `verify_mirror.sh` to detect drift between MinIO and local storage - -* Auto-trigger `chmod` only after `mc mirror` finishes - -* Monitor long-running background jobs with Krang watchdogs - -* Finalize and launch AzuraCast 5TB mirror sync - ---- - -**Signed,** -Doc -Genesis Hosting Technologies