Auto-commit from giteapush.sh at 2025-05-09 16:41:41
This commit is contained in:
parent
378ea9723c
commit
23641442ae
@ -1,4 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:14:12</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:orange;'>Warning</td><td><pre>[shredder] ERROR: ZFS RAID check failed: Authentication failed.
|
||||
[shredder] ERROR: RAID check failed (ZFS+mdstat): Authentication failed.; Authentication failed.</pre></td></tr><tr><td>mastodon</td><td style='color:orange;'>Warning</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:orange;'>Warning</td><td><pre>[db1] ERROR: ZFS RAID check failed: Authentication failed.
|
||||
[db1] ERROR: RAID check failed (ZFS+mdstat): Authentication failed.; Authentication failed.</pre></td></tr><tr><td>db2</td><td style='color:orange;'>Warning</td><td><pre>[db2] ERROR: ZFS RAID check failed: Authentication failed.
|
||||
[db2] ERROR: RAID check failed (ZFS+mdstat): Authentication failed.; Authentication failed.</pre></td></tr></table></body></html>
|
||||
@ -1,4 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:15:26</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:orange;'>Warning</td><td><pre>[shredder] ERROR: ZFS RAID check failed: Authentication failed.
|
||||
[shredder] ERROR: RAID check failed (ZFS+mdstat): Authentication failed.; Authentication failed.</pre></td></tr><tr><td>mastodon</td><td style='color:orange;'>Warning</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:orange;'>Warning</td><td><pre>[db1] ERROR: ZFS RAID check failed: Authentication failed.
|
||||
[db1] ERROR: RAID check failed (ZFS+mdstat): Authentication failed.; Authentication failed.</pre></td></tr><tr><td>db2</td><td style='color:orange;'>Warning</td><td><pre>[db2] ERROR: ZFS RAID check failed: Authentication failed.
|
||||
[db2] ERROR: RAID check failed (ZFS+mdstat): Authentication failed.; Authentication failed.</pre></td></tr></table></body></html>
|
||||
@ -1,2 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:18:37</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:orange;'>Warning</td><td><pre>[shredder] WARNING: ZFS pool 'nexus' is not healthy: pool 'nexus' is healthy</pre></td></tr><tr><td>mastodon</td><td style='color:orange;'>Warning</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:orange;'>Warning</td><td><pre>[db1] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy</pre></td></tr><tr><td>db2</td><td style='color:red;'>Critical</td><td><pre>[db2] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy
|
||||
[db2] WARNING: No snapshots found in ZFS pool 'pgpool'</pre></td></tr></table></body></html>
|
||||
@ -1,2 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:19:40</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:orange;'>Warning</td><td><pre>[shredder] WARNING: ZFS pool 'nexus' is not healthy: pool 'nexus' is healthy</pre></td></tr><tr><td>mastodon</td><td style='color:orange;'>Warning</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:orange;'>Warning</td><td><pre>[db1] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy</pre></td></tr><tr><td>db2</td><td style='color:red;'>Critical</td><td><pre>[db2] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy
|
||||
[db2] WARNING: No snapshots found in ZFS pool 'pgpool'</pre></td></tr></table></body></html>
|
||||
@ -1,2 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:23:18</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:orange;'>Warning</td><td><pre>[shredder] WARNING: ZFS pool 'nexus' is not healthy: pool 'nexus' is healthy</pre></td></tr><tr><td>mastodon</td><td style='color:orange;'>Warning</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:orange;'>Warning</td><td><pre>[db1] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy</pre></td></tr><tr><td>db2</td><td style='color:red;'>Critical</td><td><pre>[db2] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy
|
||||
[db2] WARNING: No snapshots found in ZFS pool 'pgpool'</pre></td></tr></table></body></html>
|
||||
@ -1,2 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:25:55</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:red;'>Critical</td><td><pre>[shredder] WARNING: ZFS pool 'nexus' is not healthy: pool 'nexus' is healthy</pre></td></tr><tr><td>mastodon</td><td style='color:orange;'>Warning</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:red;'>Critical</td><td><pre>[db1] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy</pre></td></tr><tr><td>db2</td><td style='color:red;'>Critical</td><td><pre>[db2] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy
|
||||
[db2] WARNING: No snapshots found in ZFS pool 'pgpool'</pre></td></tr></table></body></html>
|
||||
@ -1,2 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:27:51</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:red;'>Critical</td><td><pre>[shredder] WARNING: ZFS pool 'nexus' is not healthy: pool 'nexus' is healthy</pre></td></tr><tr><td>mastodon</td><td style='color:orange;'>Warning</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:red;'>Critical</td><td><pre>[db1] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy</pre></td></tr><tr><td>db2</td><td style='color:red;'>Critical</td><td><pre>[db2] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy
|
||||
[db2] WARNING: No snapshots found in ZFS pool 'pgpool'</pre></td></tr></table></body></html>
|
||||
@ -1,2 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:29:37</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:orange;'>Warning</td><td><pre>[shredder] WARNING: ZFS pool 'nexus' is not healthy: pool 'nexus' is healthy</pre></td></tr><tr><td>mastodon</td><td style='color:orange;'>Warning</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:orange;'>Warning</td><td><pre>[db1] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy</pre></td></tr><tr><td>db2</td><td style='color:orange;'>Warning</td><td><pre>[db2] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy
|
||||
[db2] WARNING: No snapshots found in ZFS pool 'pgpool'</pre></td></tr></table></body></html>
|
||||
@ -1,2 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:30:53</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:orange;'>Warning</td><td><pre>[shredder] WARNING: ZFS pool 'nexus' is not healthy: pool 'nexus' is healthy</pre></td></tr><tr><td>mastodon</td><td style='color:orange;'>Warning</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:orange;'>Warning</td><td><pre>[db1] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy</pre></td></tr><tr><td>db2</td><td style='color:orange;'>Warning</td><td><pre>[db2] WARNING: ZFS pool 'pgpool' is not healthy: pool 'pgpool' is healthy
|
||||
[db2] WARNING: No snapshots found in ZFS pool 'pgpool'</pre></td></tr></table></body></html>
|
||||
@ -1 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:34:37</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>mastodon</td><td style='color:orange;'>Warning</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>db2</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr></table></body></html>
|
||||
@ -1 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:35:19</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>mastodon</td><td style='color:red;'>Critical</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>db2</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr></table></body></html>
|
||||
@ -1 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:36:34</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>mastodon</td><td style='color:red;'>Critical</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>db2</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr></table></body></html>
|
||||
@ -1 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:39:33</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>mastodon</td><td style='color:red;'>Critical</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>db2</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr></table></body></html>
|
||||
@ -1 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:40:39</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>mastodon</td><td style='color:red;'>Critical</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>db2</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr></table></body></html>
|
||||
@ -1 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:42:35</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>mastodon</td><td style='color:red;'>Critical</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>db2</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr></table></body></html>
|
||||
@ -1 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:43:10</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>mastodon</td><td style='color:red;'>Critical</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>db2</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr></table></body></html>
|
||||
@ -1 +0,0 @@
|
||||
<html><head><title>Genesis Radio Healthcheck</title><meta http-equiv='refresh' content='60'></head><body><h1>Genesis Radio System Health</h1><p>Last Checked: 2025-05-02 14:43:24</p><table border='1' cellpadding='5'><tr><th>System</th><th>Status</th><th>ZFS Details</th></tr><tr><td>shredder</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>mastodon</td><td style='color:red;'>Critical</td><td><pre>All pools healthy</pre></td></tr><tr><td>db1</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr><tr><td>db2</td><td style='color:green;'>Healthy</td><td><pre>All pools healthy</pre></td></tr></table></body></html>
|
||||
@ -1,20 +0,0 @@
|
||||
# Account Creation Policy
|
||||
|
||||
## Customer Accounts
|
||||
|
||||
- Created automatically via WHMCS upon signup
|
||||
- Email verification is required before service activation
|
||||
- Strong passwords (minimum 10 characters) are enforced
|
||||
- 2FA is recommended and required for admin-facing services
|
||||
|
||||
## Staff/Admin Accounts
|
||||
|
||||
- Created manually by Super Admin only
|
||||
- Must use SSH keys for server access
|
||||
- Access logs are enabled and monitored
|
||||
- Each staff account must be linked to an internal email
|
||||
|
||||
## Account Naming Convention
|
||||
|
||||
- Customers: `client_{username}`
|
||||
- Admins: `admin.{firstname}`
|
||||
@ -1,13 +0,0 @@
|
||||
# Account Deletion Policy
|
||||
|
||||
## Customer Accounts
|
||||
|
||||
- Users may request account deletion via WHMCS support ticket
|
||||
- Data is retained for 30 days post-termination (unless legally required)
|
||||
- Backups including user data are purged after 30 days
|
||||
|
||||
## Internal Accounts
|
||||
|
||||
- Deactivated immediately upon staff departure or role change
|
||||
- SSH keys, DirectAdmin access, and database credentials revoked
|
||||
- Logs associated with the account are retained for audit purposes
|
||||
@ -1,20 +0,0 @@
|
||||
# Least Privilege Policy
|
||||
|
||||
Genesis Hosting enforces least privilege access for all systems.
|
||||
|
||||
## Principles
|
||||
|
||||
- Users are given the minimum level of access necessary to perform their work
|
||||
- Admin tools are isolated by function (e.g., billing vs. system access)
|
||||
- Escalation of privileges must be requested, documented, and time-bound
|
||||
|
||||
## Tools in Use
|
||||
|
||||
- WHMCS permissions are restricted by group
|
||||
- SSH access is limited using `AllowUsers` and firewalled IPs
|
||||
- TeamTalk server admins are rotated and audited monthly
|
||||
|
||||
## Review Cycle
|
||||
|
||||
- Access roles are reviewed quarterly
|
||||
- Logs of access changes are stored and rotated every 90 days
|
||||
@ -1,18 +0,0 @@
|
||||
# User Roles
|
||||
|
||||
Genesis Hosting Technologies uses Role-Based Access Control (RBAC) to ensure that users only have access to what they need.
|
||||
|
||||
## Role Definitions
|
||||
|
||||
| Role | Description | Examples |
|
||||
|----------------|----------------------------------------------------------|----------------------------------|
|
||||
| Customer | End users with access to services they’ve purchased | DirectAdmin clients, Streamers |
|
||||
| Support Staff | Limited admin functions for resolving client issues | Helpdesk, WHMCS support agents |
|
||||
| Administrator | Full access to provision, maintain, and modify services | Infrastructure admins |
|
||||
| Super Admin | Root-level access to all systems | Owner/Lead Engineer |
|
||||
|
||||
## Guidelines
|
||||
|
||||
- Roles are assigned during onboarding.
|
||||
- Access levels are reviewed quarterly.
|
||||
- No one should hold higher access than required for their duties.
|
||||
@ -1,26 +0,0 @@
|
||||
# Disaster Recovery Plan
|
||||
|
||||
Genesis Hosting is prepared to recover core systems from catastrophic failure.
|
||||
|
||||
## Recovery Objectives
|
||||
|
||||
- **RPO (Recovery Point Objective)**: 24 hours
|
||||
- **RTO (Recovery Time Objective)**: 4 hours for customer services
|
||||
|
||||
## Full Recovery Flow
|
||||
|
||||
1. Triage the affected systems
|
||||
2. Identify last successful backup or snapshot
|
||||
3. Restore individual services:
|
||||
- DNS
|
||||
- WHMCS
|
||||
- DirectAdmin
|
||||
- AzuraCast
|
||||
- TeamTalk
|
||||
4. Run post-restore validation scripts
|
||||
5. Notify customers of incident and resolution
|
||||
|
||||
## DR Testing
|
||||
|
||||
- Simulated quarterly
|
||||
- Logs retained in `/var/log/genesisdr.log`
|
||||
@ -1,23 +0,0 @@
|
||||
# Backup Integrity
|
||||
|
||||
We verify all backups regularly to ensure they are complete, uncorrupted, and restorable.
|
||||
|
||||
## Weekly Tasks
|
||||
|
||||
- ZFS scrubs for all pools
|
||||
- Hash checks (SHA-256) for tarballs and dumps
|
||||
- rsync `--checksum` verification for remote mirrors
|
||||
|
||||
## Alerts
|
||||
|
||||
- Email/Mastodon alert if:
|
||||
- ZFS reports checksum errors
|
||||
- Scheduled backup is missing
|
||||
- Remote sync fails or lags > 24h
|
||||
|
||||
## Tools Used
|
||||
|
||||
- `zfs scrub`
|
||||
- `sha256sum` + custom validation script
|
||||
- rclone sync logs
|
||||
- Telegram bot and Genesis Shield notifications
|
||||
@ -1,29 +0,0 @@
|
||||
# Backup Policy
|
||||
|
||||
Genesis Hosting Technologies maintains regular backups to ensure customer data and internal infrastructure are recoverable in the event of failure, corruption, or disaster.
|
||||
|
||||
## Backup Schedule
|
||||
|
||||
| System | Frequency | Retention | Method |
|
||||
|----------------|-----------|-----------|------------------|
|
||||
| DirectAdmin | Daily | 7 Days | rsync + tarball |
|
||||
| WHMCS | Daily | 14 Days | Encrypted dump |
|
||||
| AzuraCast | Daily | 7 Days | Docker volume snapshot + config export |
|
||||
| TeamTalk | Daily | 7 Days | XML + config archive |
|
||||
| Full VMs | Weekly | 4 Weeks | ZFS snapshots or Proxmox backups |
|
||||
| Offsite Backups| Weekly | 4 Weeks | Rsync to remote ZFS or object storage |
|
||||
|
||||
## Retention Policy
|
||||
|
||||
- Daily: 7 days
|
||||
- Weekly: 4 weeks
|
||||
- Monthly: Optional, for specific business data
|
||||
|
||||
## Encryption
|
||||
|
||||
- Backups are encrypted at rest (AES-256)
|
||||
- Transfers to remote locations use SSH or TLS
|
||||
|
||||
## Notes
|
||||
|
||||
- No backup occurs on client plans marked "opt-out"
|
||||
@ -1,45 +0,0 @@
|
||||
## 2025-05-02 22:24:25 – MinIO Bucket Access Configuration for Mastodon
|
||||
|
||||
**Bucket**: `assets-mastodon`
|
||||
**Server**: `shredderv2`
|
||||
**User**: `genesisuser`
|
||||
**Permissions**: Read / Write / Delete
|
||||
**Policy Name**: `assets-mastodon-rw-policy`
|
||||
|
||||
### Commands Executed:
|
||||
|
||||
```bash
|
||||
mc alias set localminio http://localhost:9000 genesisadmin MutationXv3!
|
||||
|
||||
cat > assets_mastodon_rw_policy.json <<EOF
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Action": [
|
||||
"s3:GetBucketLocation",
|
||||
"s3:ListBucket"
|
||||
],
|
||||
"Effect": "Allow",
|
||||
"Resource": "arn:aws:s3:::assets-mastodon"
|
||||
},
|
||||
{
|
||||
"Action": [
|
||||
"s3:PutObject",
|
||||
"s3:GetObject",
|
||||
"s3:DeleteObject"
|
||||
],
|
||||
"Effect": "Allow",
|
||||
"Resource": "arn:aws:s3:::assets-mastodon/*"
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
mc admin policy add localminio assets-mastodon-rw-policy assets_mastodon_rw_policy.json
|
||||
mc admin policy set localminio assets-mastodon-rw-policy user=genesisuser
|
||||
```
|
||||
|
||||
### Outcome:
|
||||
|
||||
User `genesisuser` now has full authenticated access to `assets-mastodon` on `shredderv2`'s MinIO.
|
||||
@ -1,93 +0,0 @@
|
||||
## 2025-05-02 22:24:25 – MinIO Bucket Access Configuration for Mastodon
|
||||
|
||||
**Bucket**: `assets-mastodon`
|
||||
**Server**: `shredderv2`
|
||||
**User**: `genesisuser`
|
||||
**Permissions**: Read / Write / Delete
|
||||
**Policy Name**: `assets-mastodon-rw-policy`
|
||||
|
||||
### Commands Executed:
|
||||
|
||||
```bash
|
||||
mc alias set localminio http://localhost:9000 genesisadmin MutationXv3!
|
||||
|
||||
cat > assets_mastodon_rw_policy.json <<EOF
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Action": [
|
||||
"s3:GetBucketLocation",
|
||||
"s3:ListBucket"
|
||||
],
|
||||
"Effect": "Allow",
|
||||
"Resource": "arn:aws:s3:::assets-mastodon"
|
||||
},
|
||||
{
|
||||
"Action": [
|
||||
"s3:PutObject",
|
||||
"s3:GetObject",
|
||||
"s3:DeleteObject"
|
||||
],
|
||||
"Effect": "Allow",
|
||||
"Resource": "arn:aws:s3:::assets-mastodon/*"
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
mc admin policy add localminio assets-mastodon-rw-policy assets_mastodon_rw_policy.json
|
||||
mc admin policy set localminio assets-mastodon-rw-policy user=genesisuser
|
||||
```
|
||||
|
||||
### Outcome:
|
||||
|
||||
User `genesisuser` now has full authenticated access to `assets-mastodon` on `shredderv2`'s MinIO.
|
||||
|
||||
---
|
||||
|
||||
## 2025-05-02 22:43:00 – MinIO Transfer Log: AzuraCast Assets
|
||||
|
||||
**Source**: `thevault:/nexus/miniodata/assets_azuracast`
|
||||
**Destination**: `shredderv2 MinIO` bucket `assets-azuracast`
|
||||
|
||||
### Transfer Method:
|
||||
|
||||
```bash
|
||||
rclone sync thevault:/nexus/miniodata/assets_azuracast localminio:assets-azuracast \
|
||||
--progress \
|
||||
--transfers=8 \
|
||||
--checkers=8 \
|
||||
--s3-chunk-size=64M \
|
||||
--s3-upload-concurrency=4 \
|
||||
--s3-acl=private \
|
||||
--s3-storage-class=STANDARD
|
||||
```
|
||||
|
||||
### Outcome:
|
||||
|
||||
Data from AzuraCast backup (`assets_azuracast`) successfully synchronized to MinIO bucket `assets-azuracast` on `shredderv2`.
|
||||
|
||||
---
|
||||
|
||||
## 2025-05-02 23:05:00 – MinIO Transfer Log: Mastodon Assets
|
||||
|
||||
**Source**: `thevault:/nexus/miniodata/assets_mastodon`
|
||||
**Destination**: `shredderv2 MinIO` bucket `assets-mastodon`
|
||||
|
||||
### Transfer Method:
|
||||
|
||||
```bash
|
||||
rclone sync thevault:/nexus/miniodata/assets_mastodon localminio:assets-mastodon \
|
||||
--progress \
|
||||
--transfers=8 \
|
||||
--checkers=8 \
|
||||
--s3-chunk-size=64M \
|
||||
--s3-upload-concurrency=4 \
|
||||
--s3-acl=private \
|
||||
--s3-storage-class=STANDARD
|
||||
```
|
||||
|
||||
### Outcome:
|
||||
|
||||
Assets from `assets_mastodon` replicated to `assets-mastodon` bucket on `shredderv2`. No impact to production (`shredderv1`) during sync.
|
||||
@ -1,32 +0,0 @@
|
||||
# Restore Instructions
|
||||
|
||||
The following steps outline how to restore data for each supported service.
|
||||
|
||||
## DirectAdmin
|
||||
|
||||
1. Access DA panel as admin
|
||||
2. Go to Admin Backup/Transfer
|
||||
3. Select user and backup date
|
||||
4. Click "Restore"
|
||||
|
||||
## WHMCS
|
||||
|
||||
1. SSH into WHMCS server
|
||||
2. Restore from encrypted MySQL dump
|
||||
3. Restart `php-fpm` and `nginx`
|
||||
|
||||
## AzuraCast
|
||||
|
||||
1. Stop all Docker containers
|
||||
2. Replace `station_data` and `config` volumes
|
||||
3. Restart stack via `docker-compose up -d`
|
||||
|
||||
## TeamTalk
|
||||
|
||||
1. Replace configuration file (`tt5srv.xml`)
|
||||
2. Restart TeamTalk server
|
||||
|
||||
## VM-Level Restore (ZFS)
|
||||
|
||||
1. `zfs rollback poolname/dataset@snapshotname`
|
||||
2. Verify service health and logs
|
||||
@ -1,27 +0,0 @@
|
||||
# Abuse Handling Policy
|
||||
|
||||
We take reports of abuse seriously and aim to resolve them quickly.
|
||||
|
||||
## How to Report Abuse
|
||||
|
||||
Send an email to abuse@genesishostingtechnologies.com with:
|
||||
|
||||
- Description of the abuse
|
||||
- IP or domain involved
|
||||
- Any relevant logs or screenshots
|
||||
|
||||
## Internal Response Process
|
||||
|
||||
1. Triage within 12 hours
|
||||
2. Investigate logs and usage
|
||||
3. Contact the client with 24h to respond
|
||||
4. Temporary suspension may be issued to prevent further harm
|
||||
|
||||
## DMCA Takedowns
|
||||
|
||||
- We comply with valid DMCA requests
|
||||
- The client will be notified and given 48h to address or refute
|
||||
|
||||
## Escalation
|
||||
|
||||
Repeat offenders may be permanently banned.
|
||||
@ -1,22 +0,0 @@
|
||||
# Account Suspension Policy
|
||||
|
||||
Accounts may be suspended for violations of our Acceptable Use Policy, overdue invoices, or abuse complaints.
|
||||
|
||||
## Common Reasons
|
||||
|
||||
- Non-payment (after 5-day grace period)
|
||||
- Resource abuse or denial-of-service behavior
|
||||
- Hosting prohibited content
|
||||
- Violating community guidelines on TeamTalk
|
||||
|
||||
## Suspension Procedure
|
||||
|
||||
- Warning issued via WHMCS ticket and email
|
||||
- If no resolution within 24–48h, service is suspended
|
||||
- Admin note added to client profile for audit tracking
|
||||
|
||||
## Reinstatement
|
||||
|
||||
- Suspension is lifted upon payment or resolution
|
||||
- $5 reactivation fee may apply (for non-payment suspensions)
|
||||
- Services are not reinstated if terminated due to serious AUP violation
|
||||
@ -1,27 +0,0 @@
|
||||
# Acceptable Use Policy (AUP)
|
||||
|
||||
This policy outlines the acceptable use of services provided by Genesis Hosting Technologies.
|
||||
|
||||
## Prohibited Activities
|
||||
|
||||
Clients may not use our services to:
|
||||
|
||||
- Host or distribute malware, phishing sites, or spyware
|
||||
- Send unsolicited email (spam), whether direct or relayed
|
||||
- Host copyrighted content without permission (DMCA applies)
|
||||
- Promote hate speech, harassment, or targeted abuse
|
||||
- Overuse system resources in a way that affects others
|
||||
|
||||
## Special Notes
|
||||
|
||||
- Streaming via AzuraCast must comply with DMCA and public broadcast standards
|
||||
- TeamTalk users must not harass, dox, or spam other users
|
||||
- VPNs, proxies, and anonymizing services are not allowed without prior approval
|
||||
|
||||
## Enforcement
|
||||
|
||||
Violations will result in one or more of the following:
|
||||
|
||||
- Warning via email or WHMCS ticket
|
||||
- Service suspension
|
||||
- Permanent termination without refund (in egregious cases)
|
||||
@ -1,24 +0,0 @@
|
||||
# Refunds & Cancellations
|
||||
|
||||
Genesis Hosting Technologies offers a clear refund and cancellation policy.
|
||||
|
||||
## Cancellation
|
||||
|
||||
- Clients may cancel via WHMCS at any time
|
||||
- Cancellation before next billing date avoids future charges
|
||||
- No prorated refunds for unused time unless due to service failure
|
||||
|
||||
## Refunds
|
||||
|
||||
- Full refund within 7 days of initial purchase (DirectAdmin, AzuraCast, TeamTalk)
|
||||
- Domain registrations, SSL certificates, and add-ons are non-refundable
|
||||
- No refunds issued for abuse-related suspensions or policy violations
|
||||
|
||||
## Exceptions
|
||||
|
||||
- If we fail to deliver a service or suffer extended downtime (>24h), credit may be issued
|
||||
- All refund requests are reviewed manually by support
|
||||
|
||||
## How to Request
|
||||
|
||||
Submit a WHMCS ticket with reason for refund
|
||||
@ -1,20 +0,0 @@
|
||||
# Code of Conduct
|
||||
|
||||
We maintain a respectful, safe, and inclusive environment for both staff and clients.
|
||||
|
||||
## Expectations
|
||||
|
||||
- Treat all clients and team members with professionalism and courtesy
|
||||
- Communicate clearly and constructively — even during escalations
|
||||
- Uphold privacy, security, and transparency at every level
|
||||
- Follow internal and customer-facing policies at all times
|
||||
|
||||
## Zero Tolerance
|
||||
|
||||
We do not tolerate:
|
||||
|
||||
- Harassment or abuse (verbal, written, or otherwise)
|
||||
- Discrimination based on identity, ability, or belief
|
||||
- Intentional sabotage of infrastructure or service integrity
|
||||
|
||||
Violations may result in immediate termination of access or service.
|
||||
@ -1,12 +0,0 @@
|
||||
# Mission Statement
|
||||
|
||||
At Genesis Hosting Technologies, our mission is to provide secure, reliable, and transparent hosting services with a personal touch.
|
||||
|
||||
We believe that even the smallest teams deserve enterprise-grade infrastructure — without enterprise-grade headaches.
|
||||
|
||||
Our goal is to deliver:
|
||||
|
||||
- Fast, stable hosting environments
|
||||
- Fair pricing with no upsell games
|
||||
- Transparent policies and proactive support
|
||||
- A commitment to data ownership and user privacy
|
||||
@ -1,25 +0,0 @@
|
||||
# Terms of Service (TOS)
|
||||
|
||||
By using services from Genesis Hosting Technologies, you agree to the following terms:
|
||||
|
||||
## Service Provision
|
||||
|
||||
- Services are delivered as-is, with best-effort uptime and technical support
|
||||
- Users must abide by our Acceptable Use Policy (AUP)
|
||||
- Access may be suspended for abuse, non-payment, or security issues
|
||||
|
||||
## Billing & Renewals
|
||||
|
||||
- All services are billed monthly or annually
|
||||
- Automatic renewal is enabled by default
|
||||
- Invoices are due within 5 days of issue unless otherwise agreed
|
||||
|
||||
## Termination
|
||||
|
||||
- You may cancel at any time via WHMCS
|
||||
- We reserve the right to suspend or terminate accounts that violate our policies
|
||||
|
||||
## Liability
|
||||
|
||||
- We are not liable for data loss, service interruptions, or indirect damages
|
||||
- Backups are provided as a best-effort courtesy unless contractually guaranteed
|
||||
@ -1,25 +0,0 @@
|
||||
# DMCA Policy
|
||||
|
||||
Genesis Hosting Technologies complies with the Digital Millennium Copyright Act (DMCA).
|
||||
|
||||
## Filing a Takedown Notice
|
||||
|
||||
Email dmca@genesishostingtechnologies.com with:
|
||||
|
||||
- Your contact information
|
||||
- Description of the copyrighted work
|
||||
- URL or IP address of the infringing content
|
||||
- A statement of good faith belief
|
||||
- A statement of accuracy and authority
|
||||
|
||||
## What Happens Next
|
||||
|
||||
- We review and respond within 48 hours
|
||||
- The client is notified and given a chance to respond
|
||||
- If no valid counter-notice is filed, content may be removed or suspended
|
||||
|
||||
## Filing a Counter Notice
|
||||
|
||||
Clients who believe their content was wrongly removed may submit a counter notice with similar contact and justification information.
|
||||
|
||||
We will not tolerate repeated infringement and may terminate accounts accordingly.
|
||||
@ -1,26 +0,0 @@
|
||||
# Privacy Policy
|
||||
|
||||
We respect your privacy and protect your data.
|
||||
|
||||
## What We Collect
|
||||
|
||||
- Account information: name, email, billing address
|
||||
- Service usage data: IPs, access logs, system metrics
|
||||
- Communications: support tickets and emails
|
||||
|
||||
## How We Use It
|
||||
|
||||
- Service provisioning and support
|
||||
- Abuse prevention and system integrity
|
||||
- Internal analytics (not shared or sold)
|
||||
|
||||
## Data Sharing
|
||||
|
||||
- We do not sell user data
|
||||
- We may share limited data with trusted providers (e.g., payment processors)
|
||||
- Law enforcement requests must include valid legal process
|
||||
|
||||
## Data Retention
|
||||
|
||||
- User data is retained as long as the account is active
|
||||
- Backups are purged per the Backup Policy
|
||||
@ -1,64 +0,0 @@
|
||||
# 📛 Case Study: Why RAID Is Not a Backup
|
||||
|
||||
## Overview
|
||||
|
||||
On May 4, 2025, we experienced a production data loss incident involving the `nexus` dataset on `shredderv1`, a Linux RAID5 server. Despite no hardware failure, critical files were lost due to an unintended command affecting live data.
|
||||
|
||||
This incident serves as a clear, real-world illustration of the maxim:
|
||||
|
||||
> **RAID protects against hardware failure — not human error, data corruption, or bad automation.**
|
||||
|
||||
---
|
||||
|
||||
## 🔍 What Happened
|
||||
|
||||
- `shredderv1` uses RAID5 for media storage.
|
||||
- The dataset `nexus/miniodata` (housing `genesisassets`, `genesislibrary`, etc.) was accidentally destroyed.
|
||||
- **No disks failed.** The failure was logical, not physical.
|
||||
|
||||
---
|
||||
|
||||
## 🔥 Impact
|
||||
|
||||
- StationPlaylist (SPL) lost access to the Genesis media library.
|
||||
- MinIO bucket data was instantly inaccessible.
|
||||
- Temporary outage and scrambling to reconfigure mounts, media, and streaming.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Recovery
|
||||
|
||||
Thanks to our disaster recovery stack:
|
||||
|
||||
- Nightly **rsync backups** were synced to **The Vault** (backup server).
|
||||
- **ZFS snapshots** existed on The Vault for the affected datasets.
|
||||
- We restored the latest snapshot **from The Vault back to Shredder**, effectively reversing the loss.
|
||||
- No data corruption occurred; sync validation showed dataset integrity.
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Takeaway
|
||||
|
||||
This is a live demonstration of why:
|
||||
|
||||
- **RAID is not a backup**
|
||||
- **Snapshots without off-host replication** are not enough
|
||||
- **Real backups must be off-server and regularly tested**
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Current Protection Measures
|
||||
|
||||
- Production data (`genesisassets`, `genesislibrary`) now replicated nightly to The Vault via `rsync`.
|
||||
- ZFS snapshots are validated daily via a **dry-run restore validator**.
|
||||
- Telegram alerts notify success/failure of backup verification jobs.
|
||||
- Future goal: full ZFS storage on all production servers for native snapshot support.
|
||||
|
||||
---
|
||||
|
||||
## 🧠 Lessons Learned
|
||||
|
||||
- Always assume you'll delete the wrong thing eventually.
|
||||
- Snapshots are amazing — **if** they're somewhere else.
|
||||
- Automated restore testing should be part of every backup pipeline.
|
||||
|
||||
@ -1,50 +0,0 @@
|
||||
# 🌐 DNS Access Issues – Troubleshooting Guide
|
||||
|
||||
If you're having trouble reaching **Genesis Radio** or the stream won't load, the issue may be with your DNS provider (the service that turns domain names into IP addresses).
|
||||
|
||||
This happens more often than you'd think — and it's easy to fix.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Quick Fix: Change Your DNS
|
||||
|
||||
We recommend switching to one of these trusted, fast, privacy-respecting DNS providers:
|
||||
|
||||
| Provider | DNS Servers |
|
||||
|--------------|-----------------------------|
|
||||
| **Google** | `8.8.8.8` and `8.8.4.4` |
|
||||
| **Cloudflare** | `1.1.1.1` and `1.0.0.1` |
|
||||
| **Quad9** | `9.9.9.9` |
|
||||
|
||||
---
|
||||
|
||||
## 💻 How to Change Your DNS
|
||||
|
||||
### Windows 10/11
|
||||
1. Open **Settings → Network & Internet**
|
||||
2. Click **Change adapter options**
|
||||
3. Right-click your active connection → **Properties**
|
||||
4. Select **Internet Protocol Version 4 (TCP/IPv4)** → Click **Properties**
|
||||
5. Choose **"Use the following DNS server addresses"**
|
||||
6. Enter:
|
||||
- Preferred: `1.1.1.1`
|
||||
- Alternate: `8.8.8.8`
|
||||
7. Save and reconnect
|
||||
|
||||
---
|
||||
|
||||
### macOS
|
||||
1. Go to **System Preferences → Network**
|
||||
2. Select your active network → Click **Advanced**
|
||||
3. Go to the **DNS** tab
|
||||
4. Click `+` and add:
|
||||
- `1.1.1.1`
|
||||
- `8.8.8.8`
|
||||
5. Apply changes and reconnect
|
||||
|
||||
---
|
||||
|
||||
### Linux (CLI)
|
||||
For a quick test:
|
||||
```bash
|
||||
sudo resolvectl dns eth0 1.1.1.1 8.8.8.8
|
||||
@ -1,24 +0,0 @@
|
||||
# Genesis Shield – Security & Threat Monitoring
|
||||
|
||||
Genesis Shield is our custom-built alert and ban system, integrated across our infrastructure.
|
||||
|
||||
## Features
|
||||
|
||||
- Aggregates Fail2Ban logs across all VMs
|
||||
- Bans pushed in real-time via Mastodon DM and Telegram
|
||||
- Scripts track:
|
||||
- Repeated SSH failures
|
||||
- API abuse
|
||||
- Web panel brute force attempts
|
||||
|
||||
## Interfaces
|
||||
|
||||
- Terminal dashboard for live bans/unbans
|
||||
- Role-based control (root/admin only)
|
||||
- Daily threat summary via Mastodon bot
|
||||
|
||||
## Roadmap
|
||||
|
||||
- WHMCS integration for abuse tickets
|
||||
- Live threat map by country/IP
|
||||
- REST API for admin toolkit
|
||||
@ -1,25 +0,0 @@
|
||||
# Maintenance Window Policy
|
||||
|
||||
To maintain consistency and reduce customer impact, we adhere to a strict maintenance schedule.
|
||||
|
||||
## Standard Window
|
||||
|
||||
- **Every Sunday, 7 PM – 9 PM Eastern**
|
||||
- Non-emergency changes must occur during this window
|
||||
|
||||
## What’s Allowed
|
||||
|
||||
- OS & kernel updates
|
||||
- Docker/image upgrades
|
||||
- ZFS snapshots & cleanup
|
||||
- Rolling restarts of containers
|
||||
|
||||
## Emergencies
|
||||
|
||||
- Critical security patches can bypass the window
|
||||
- All emergency changes must be logged and reviewed
|
||||
|
||||
## Notifications
|
||||
|
||||
- Posted on Mastodon at least 1 hour before the window begins
|
||||
- Clients notified via WHMCS if it will affect their service
|
||||
@ -1,25 +0,0 @@
|
||||
# Monitoring Setup
|
||||
|
||||
We use a layered monitoring approach to ensure full visibility and rapid response.
|
||||
|
||||
## Stack
|
||||
|
||||
- **Prometheus** for metrics collection
|
||||
- **Grafana** for visualization dashboards
|
||||
- **Fail2Ban** for intrusion attempts
|
||||
- **Genesis Shield** for aggregated alerts (Telegram + Mastodon)
|
||||
|
||||
## What We Monitor
|
||||
|
||||
| System | Metric Examples |
|
||||
|----------------|--------------------------------------------|
|
||||
| PostgreSQL | Replication lag, disk usage, active queries |
|
||||
| Web Servers | HTTP response time, TLS errors |
|
||||
| MinIO / Assets | Cache hit ratio, sync status |
|
||||
| Docker Hosts | Container uptime, memory pressure |
|
||||
|
||||
## Alerting
|
||||
|
||||
- Telegram: Real-time infra alerts
|
||||
- Mastodon bot: Daily summaries and status posts
|
||||
- Fallback email alerts for critical failures
|
||||
@ -1,19 +0,0 @@
|
||||
# Server Naming Convention
|
||||
|
||||
To reduce confusion and improve clarity, we follow a clear and themed naming structure.
|
||||
|
||||
## Naming Style
|
||||
|
||||
Examples:
|
||||
|
||||
- `krang.internal` – Master backend server
|
||||
- `replica.db3.sshjunkie.com` – Staging PostgreSQL replica
|
||||
- `shredderv2` – ZFS backup server
|
||||
- `anthony` – Ansible control node
|
||||
- `nexus` – Main ZFS pool server for assets
|
||||
|
||||
## Guidelines
|
||||
|
||||
- Avoid generic names (`server1`, `host123`)
|
||||
- Use themed names (e.g., TMNT characters for core infrastructure)
|
||||
- Include environment tags where needed (`-test`, `-prod`)
|
||||
@ -1,23 +0,0 @@
|
||||
# ZFS Strategy
|
||||
|
||||
ZFS is used across Genesis Hosting Technologies for performance, integrity, and snapshot-based backup operations.
|
||||
|
||||
## Pool Layout
|
||||
|
||||
- RAIDZ1 or mirrored vdevs depending on use case
|
||||
- Dataset naming: `genesisassets-secure`, `genesisshows-secure`, etc.
|
||||
- Dedicated pools for:
|
||||
- Mastodon media
|
||||
- Client backups
|
||||
- Internal scripts and logs
|
||||
|
||||
## Snapshots
|
||||
|
||||
- Hourly: last 24 hours
|
||||
- Daily: last 7 days
|
||||
- Weekly: last 4 weeks
|
||||
|
||||
## Send/Receive
|
||||
|
||||
- Used for offsite replication to Servarica and backup nodes
|
||||
- Verified using checksums and `zfs receive -F`
|
||||
@ -1,63 +0,0 @@
|
||||
# ✅ Master Compliance Checklist
|
||||
*(Status: ☐ = Not Started | 🟨 = In Progress | ✅ = Complete)*
|
||||
|
||||
## 🧑💼 Access & User Management
|
||||
- [ ] Role-Based Access Control (RBAC) in place (Customer, Admin, etc.)
|
||||
- [ ] Account creation follows secure onboarding workflows
|
||||
- [ ] Admin access restricted to SSH keys only
|
||||
- [ ] Inactive accounts locked or removed quarterly
|
||||
- [ ] Least privilege enforced across all services
|
||||
|
||||
## 💾 Backups & Disaster Recovery
|
||||
- [ ] Daily backups enabled for all key services (DirectAdmin, WHMCS, AzuraCast, TeamTalk)
|
||||
- [ ] Weekly offsite backups with verification
|
||||
- [ ] ZFS snapshots scheduled (hourly/daily/weekly)
|
||||
- [ ] Backup integrity validated with checksums or scrubs
|
||||
- [ ] Quarterly disaster recovery drill completed
|
||||
- [ ] Restore instructions documented and tested
|
||||
|
||||
## 🔐 Security
|
||||
- [ ] 2FA enabled on all admin interfaces (WHMCS, SSH, DirectAdmin)
|
||||
- [ ] SSH password auth disabled, key-only enforced
|
||||
- [ ] Weekly patching or updates scheduled (Sunday 7–9 PM)
|
||||
- [ ] Centralized logging active and stored 30–90 days
|
||||
- [ ] Fail2Ban + Genesis Shield integrated and alerting
|
||||
- [ ] TLS 1.2+ enforced for all public services
|
||||
- [ ] AES-256 encryption at rest on backups and sensitive volumes
|
||||
|
||||
## 🖥️ Provisioning & Automation
|
||||
- [ ] WHMCS integrated with DirectAdmin, AzuraCast, TeamTalk
|
||||
- [ ] All provisioning scripts tested and logged
|
||||
- [ ] Post-deploy verification checklist followed
|
||||
- [ ] DNS + SSL automation in place (Let's Encrypt)
|
||||
- [ ] Monitoring added after provisioning (Prometheus/Grafana)
|
||||
|
||||
## 📋 Client Policies
|
||||
- [ ] Acceptable Use Policy posted and enforced
|
||||
- [ ] Abuse response process defined and working
|
||||
- [ ] DMCA policy publicly available and followed
|
||||
- [ ] Suspension and refund rules defined in WHMCS
|
||||
- [ ] Privacy Policy and Terms of Service available on client portal
|
||||
|
||||
## 🌐 Services Configuration
|
||||
- [ ] DirectAdmin quotas enforced (disk, bandwidth, email)
|
||||
- [ ] AzuraCast listener/storage/bitrate limits respected
|
||||
- [ ] TeamTalk server abuse protection and user limits enforced
|
||||
- [ ] Domain registration/renewal workflows tested
|
||||
- [ ] SSL auto-renew working correctly (Let's Encrypt + certbot)
|
||||
|
||||
## ⚙️ Infrastructure
|
||||
- [ ] ZFS pools configured for redundancy (RAIDZ1, mirrors)
|
||||
- [ ] rclone mount points with caching working and monitored
|
||||
- [ ] Genesis Shield actively alerting via Telegram/Mastodon
|
||||
- [ ] All VMs named per convention (e.g., `krang`, `shredderv2`)
|
||||
- [ ] Sunday maintenance window consistently followed
|
||||
- [ ] Ansible playbooks used for provisioning/config consistency
|
||||
|
||||
## 🛠️ Tools & Scripts
|
||||
- [ ] All scripts version-controlled and documented
|
||||
- [ ] Backups and restore tools tested and working
|
||||
- [ ] Mastodon alert bot operating with secure tokens
|
||||
- [ ] Rclone VFS stats monitored regularly
|
||||
- [ ] Admin tools accessible only by authorized users
|
||||
"""
|
||||
@ -1,83 +0,0 @@
|
||||
# Postmortem: Genesis I/O Realignment
|
||||
|
||||
**Date:** May 8, 2025
|
||||
**Author:** Doc
|
||||
**Systems Involved:** minioraid5, shredder, chatwithus.live, zcluster.technodrome1/2, thevault
|
||||
**Scope:** Local-first mirroring, permission normalization, MinIO transition
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Objective
|
||||
|
||||
To realign the Genesis file flow architecture by:
|
||||
|
||||
- Making local block storage the **primary source** of truth for AzuraCast and Genesis buckets
|
||||
- Transitioning FTP uploads to target local storage instead of MinIO directly
|
||||
- Establishing **two-way mirroring** between local paths and MinIO buckets
|
||||
- Correcting inherited permission issues across `/mnt/raid5` using `find + chmod`
|
||||
- Preserving MinIO buckets as **backup mirrors**, not primary data stores
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Work Performed
|
||||
|
||||
### ✅ Infrastructure changes:
|
||||
- Deployed block storage volume to Linode Mastodon instance
|
||||
- Mirrored MinIO buckets (`genesisassets`, `genesislibrary`, `azuracast`) to local paths
|
||||
- Configured cron-based `mc mirror` jobs:
|
||||
- Local ➜ MinIO: every 5 minutes with `--overwrite --remove`
|
||||
- MinIO ➜ Local: nightly pull, no `--remove`
|
||||
|
||||
### ✅ FTP Pipeline Adjustments:
|
||||
- Users now upload to `/mnt/spl/ftp/uploads` (local)
|
||||
- Permissions set so only admins access full `/mnt/spl/ftp`
|
||||
- FTP directory structure created for SPL automation
|
||||
|
||||
### ✅ System Tuning:
|
||||
- Set `vm.swappiness=10` on all nodes
|
||||
- Apache disabled where not in use
|
||||
- Daily health checks via `pull_health_everywhere.sh`
|
||||
- Krang Telegram alerts deployed for cleanup and system state
|
||||
|
||||
---
|
||||
|
||||
## 🧠 Observations
|
||||
|
||||
- **High load** on `minioraid5` during `mc mirror` and `chmod` overlap
|
||||
- Load ~6.5 due to concurrent I/O pressure
|
||||
- `chmod` stuck in `D` state (I/O wait) while `mc` dominated disk queues
|
||||
- Resolved after `mc` completion — `chmod` resumed and completed
|
||||
|
||||
- **MinIO buckets were temporarily inaccessible** due to permissions accidentally inherited by FTP group
|
||||
- Resolved by recursively resetting permissions on `/mnt/raid5`
|
||||
|
||||
- **Krang telemetry** verified:
|
||||
- Mastodon swap usage rising under asset load
|
||||
- All nodes had Apache disabled or dormant
|
||||
- Health alerts triggered on high swap or load
|
||||
|
||||
---
|
||||
|
||||
## ✅ Outcome
|
||||
|
||||
- Full Genesis and AzuraCast data now reside locally with resilient S3 mirrors
|
||||
- Mastodon running on block storage, no longer dependent on MinIO latency
|
||||
- FTP integration with SPL directory trees complete
|
||||
- Cleanup script successfully deployed across all nodes via Krang
|
||||
- Daily health reports operational with alerts for high swap/load
|
||||
|
||||
---
|
||||
|
||||
## 🔁 Recommendations
|
||||
|
||||
- Consider adding snapshot-based ZFS backups for `/mnt/raid5`
|
||||
- Build `verify_mirror.sh` to detect drift between MinIO and local storage
|
||||
- Auto-trigger `chmod` only after `mc mirror` finishes
|
||||
- Monitor long-running background jobs with Krang watchdogs
|
||||
|
||||
---
|
||||
|
||||
**Signed,**
|
||||
Doc
|
||||
Genesis Hosting Technologies
|
||||
|
||||
@ -1,23 +0,0 @@
|
||||
# Provisioning Checklist
|
||||
|
||||
This checklist is followed every time a new service is deployed.
|
||||
|
||||
## Pre-Provisioning
|
||||
|
||||
- [ ] Verify order and payment in WHMCS
|
||||
- [ ] Confirm product mapping is correct
|
||||
- [ ] Check available server resources
|
||||
|
||||
## Provisioning
|
||||
|
||||
- [ ] Trigger appropriate script/module
|
||||
- [ ] Log provisioning result
|
||||
- [ ] Assign DNS entries if applicable
|
||||
- [ ] Generate Let’s Encrypt SSL if public-facing
|
||||
|
||||
## Post-Provisioning
|
||||
|
||||
- [ ] Send welcome email via WHMCS
|
||||
- [ ] Confirm monitoring alert is active
|
||||
- [ ] Test login credentials and endpoints
|
||||
- [ ] Label service with client ID in Grafana/Prometheus
|
||||
@ -1,22 +0,0 @@
|
||||
# Post-Deployment Verification
|
||||
|
||||
All services go through a post-deploy QA check to ensure they're live and stable.
|
||||
|
||||
## Verification Tasks
|
||||
|
||||
- [ ] Service reachable from public IP or internal route
|
||||
- [ ] DNS resolves correctly (for domains/subdomains)
|
||||
- [ ] SSL certificate is active and trusted
|
||||
- [ ] Admin login works as expected
|
||||
- [ ] Usage quotas correctly applied (disk, users, bandwidth)
|
||||
|
||||
## Monitoring
|
||||
|
||||
- [ ] Add to Prometheus for service-specific metrics
|
||||
- [ ] Set alert thresholds (e.g., disk > 80%)
|
||||
- [ ] Confirm Telegram/Mastodon alert webhook is functional
|
||||
|
||||
## Documentation
|
||||
|
||||
- [ ] Log final status in WHMCS admin notes
|
||||
- [ ] Store internal service details in `genesis-inventory.yaml`
|
||||
@ -1,23 +0,0 @@
|
||||
# WHMCS Integration
|
||||
|
||||
WHMCS handles client billing, service provisioning, and support workflows.
|
||||
|
||||
## Services Integrated
|
||||
|
||||
| Service | Method |
|
||||
|--------------|---------------------------------|
|
||||
| DirectAdmin | Built-in WHMCS module |
|
||||
| AzuraCast | Custom provisioning script |
|
||||
| TeamTalk | API + XML user patching scripts |
|
||||
|
||||
## Auto-Provisioning Steps
|
||||
|
||||
1. Client signs up and completes payment
|
||||
2. WHMCS triggers product-specific hook
|
||||
3. Script/module provisions the service
|
||||
4. Welcome email is sent with credentials
|
||||
|
||||
## Logging & Troubleshooting
|
||||
|
||||
- Logs stored at `/var/log/whmcs-hooks.log`
|
||||
- Errors generate internal ticket automatically if provisioning fails
|
||||
@ -1,25 +0,0 @@
|
||||
# Incident Response Policy
|
||||
|
||||
This document defines how we detect, respond to, and report security incidents.
|
||||
|
||||
## Response Workflow
|
||||
|
||||
1. Detection via monitoring, alert, or client report
|
||||
2. Triage severity and affected systems
|
||||
3. Contain and isolate threat (e.g., suspend access)
|
||||
4. Notify stakeholders if client-impacting
|
||||
5. Perform root cause analysis
|
||||
6. Patch, re-secure, and document the event
|
||||
|
||||
## Timelines
|
||||
|
||||
- Initial triage: within 2 hours
|
||||
- Client notification (if impacted): within 24 hours
|
||||
- Final report delivered internally within 72 hours
|
||||
|
||||
## Tools Used
|
||||
|
||||
- Fail2Ban
|
||||
- Genesis Shield alerting
|
||||
- Zabbix/Prometheus incident flags
|
||||
- Manual log reviews (forensic-level)
|
||||
@ -1,24 +0,0 @@
|
||||
# Logging & Monitoring Policy
|
||||
|
||||
We collect and monitor system activity to detect threats, enforce accountability, and assist in incident resolution.
|
||||
|
||||
## Log Types
|
||||
|
||||
- SSH login attempts
|
||||
- WHMCS access logs
|
||||
- AzuraCast and TeamTalk server logs
|
||||
- PostgreSQL query and connection logs
|
||||
- Fail2Ban logs (ban/unban events)
|
||||
|
||||
## Monitoring Tools
|
||||
|
||||
- Prometheus for metrics
|
||||
- Grafana dashboards for visual alerts
|
||||
- Genesis Shield (Telegram + Mastodon alerting)
|
||||
- Manual log review every 7 days
|
||||
|
||||
## Retention
|
||||
|
||||
- General logs: 30 days
|
||||
- Security-related logs: 90 days minimum
|
||||
- Logs archived to encrypted ZFS volume
|
||||
@ -1,23 +0,0 @@
|
||||
# Encryption Standards
|
||||
|
||||
Encryption is applied to all data in transit and at rest across Genesis Hosting Technologies infrastructure.
|
||||
|
||||
## In Transit
|
||||
|
||||
- HTTPS via TLS 1.3 (minimum TLS 1.2 for legacy fallback)
|
||||
- SFTP for all file transfers
|
||||
- SSH for all administrative access
|
||||
- rclone with TLS for object storage replication
|
||||
|
||||
## At Rest
|
||||
|
||||
- ZFS encryption on backup pools
|
||||
- PostgreSQL encryption at the database or filesystem level
|
||||
- WHMCS and DirectAdmin credentials hashed and salted
|
||||
- Backups encrypted with AES-256 before remote transfer
|
||||
|
||||
## Key Management
|
||||
|
||||
- SSH keys rotated every 6 months
|
||||
- Let's Encrypt certs auto-renew every 90 days
|
||||
- Master encryption keys stored offline and version-controlled
|
||||
@ -1,23 +0,0 @@
|
||||
# Security Policy
|
||||
|
||||
Genesis Hosting Technologies enforces strict security practices across all infrastructure and services to protect client data and maintain service integrity.
|
||||
|
||||
## Core Principles
|
||||
|
||||
- Least privilege for all users and services
|
||||
- Regular audits and patching
|
||||
- Encrypted communication and storage
|
||||
- Real-time monitoring and alerting
|
||||
|
||||
## Enforcement Areas
|
||||
|
||||
- 2FA required for all admin portals
|
||||
- SSH access limited to key-based logins
|
||||
- Centralized log collection and review
|
||||
- All critical assets monitored via Genesis Shield
|
||||
|
||||
## Review Cycle
|
||||
|
||||
- Policies reviewed quarterly
|
||||
- Logs retained for 30–90 days depending on system
|
||||
- Incidents reviewed post-mortem with improvements logged
|
||||
@ -1,32 +0,0 @@
|
||||
# AzuraCast Streaming Policy
|
||||
|
||||
## Features
|
||||
|
||||
- Custom stream URLs (via relay or direct)
|
||||
- Icecast or SHOUTcast available
|
||||
- AutoDJ + scheduled playlists
|
||||
- Web-based file upload + schedule
|
||||
|
||||
## Plans & Limits
|
||||
|
||||
| Plan | Storage | Listeners | Bitrate |
|
||||
|----------|---------|-----------|---------|
|
||||
| StreamLite | 2 GB | 25 | 128 kbps|
|
||||
| StreamPro | 10 GB | 100 | 192 kbps|
|
||||
| StreamMax | 50 GB | 250 | 320 kbps|
|
||||
|
||||
## Fair Usage Policy
|
||||
|
||||
- No nonstop streaming of static loops to inflate uptime
|
||||
- Long-form live shows should rotate metadata periodically
|
||||
- Content must not violate copyright laws
|
||||
|
||||
## Backups
|
||||
|
||||
- Daily backups of config + playlists
|
||||
- Client media backup is optional (paid add-on)
|
||||
|
||||
## Support
|
||||
|
||||
- Stream diagnostics available in client panel
|
||||
- WHMCS ticket support for outages or playlist issues
|
||||
@ -1,27 +0,0 @@
|
||||
# DirectAdmin Hosting Policy
|
||||
|
||||
## Features
|
||||
|
||||
- FTP, webmail, MySQL, file manager, and site statistics
|
||||
- Optional Let's Encrypt SSL enabled by default
|
||||
- Nightly site + database backups (7-day retention)
|
||||
|
||||
## Plans & Limits
|
||||
|
||||
| Plan | Disk | Bandwidth | Domains | Email Accounts |
|
||||
|------------|------|-----------|---------|----------------|
|
||||
| Starter | 5 GB | 100 GB | 1 | 5 |
|
||||
| Standard | 20 GB| 500 GB | 5 | 25 |
|
||||
| Unlimited | 100 GB| ∞ | ∞ | ∞ |
|
||||
|
||||
## Abuse Prevention
|
||||
|
||||
- Email rate limits applied to prevent outbound spam
|
||||
- CPU usage and inode caps enforced
|
||||
- Suspicious files scanned automatically
|
||||
|
||||
## Support
|
||||
|
||||
- Available via WHMCS ticket system
|
||||
- Response within 12 business hours
|
||||
|
||||
@ -1,22 +0,0 @@
|
||||
# Domain Management Policy
|
||||
|
||||
## Registration
|
||||
|
||||
- Domains registered through our WHMCS interface are managed via third-party registrar API
|
||||
- Registration typically completes within 5 minutes
|
||||
- WHOIS privacy included by default (where available)
|
||||
|
||||
## Renewals
|
||||
|
||||
- Auto-renew is enabled by default
|
||||
- Reminders sent 30, 7, and 1 day before expiration
|
||||
|
||||
## Transfers
|
||||
|
||||
- Domains can be transferred in or out with EPP code
|
||||
- Support required if domain is locked or expired
|
||||
|
||||
## DNS
|
||||
|
||||
- Free DNS hosting included
|
||||
- Custom DNS records managed through DirectAdmin or WHMCS panel
|
||||
@ -1,23 +0,0 @@
|
||||
# SSL Certificate Policy
|
||||
|
||||
## Free Certificates
|
||||
|
||||
- Let’s Encrypt certificates issued automatically
|
||||
- Applies to DirectAdmin, AzuraCast, and custom subdomains
|
||||
- Auto-renews every 60 days with 30-day buffer
|
||||
|
||||
## Premium SSL
|
||||
|
||||
- Custom SSL certs (e.g., EV/OV) available for purchase
|
||||
- Requires manual install via WHMCS ticket
|
||||
|
||||
## Certificate Management
|
||||
|
||||
- Certbot used for automation
|
||||
- Custom certs must be supplied in `.crt` + `.key` format
|
||||
- Broken SSL installs may be reverted to Let’s Encrypt fallback
|
||||
|
||||
## Support
|
||||
|
||||
- Certificate issues resolved within 24h of report
|
||||
- DNS challenges supported for wildcard certs
|
||||
@ -1,26 +0,0 @@
|
||||
# TeamTalk Hosting Policy
|
||||
|
||||
## Features
|
||||
|
||||
- Private and public servers
|
||||
- Voice chat, file sharing, push-to-talk
|
||||
- Admin access with room/channel management
|
||||
|
||||
## Plans & Limits
|
||||
|
||||
| Plan | Users | Bitrate Limit | Admin Access |
|
||||
|--------------|-------|---------------|--------------|
|
||||
| Basic Chat | 10 | 64 kbps | Yes |
|
||||
| Pro Voice | 50 | 128 kbps | Yes |
|
||||
| Broadcast+ | 100 | 256 kbps | Yes |
|
||||
|
||||
## Rules
|
||||
|
||||
- No harassment, spamming, or automated bots without permission
|
||||
- Abuse may result in temp suspension or permanent ban
|
||||
- Admins are responsible for moderating their own servers
|
||||
|
||||
## Configuration
|
||||
|
||||
- Clients may request config changes via WHMCS ticket
|
||||
- Backups of XML configs stored nightly
|
||||
@ -1,2 +0,0 @@
|
||||
MASTODON_BASE_URL=https://chatwithus.live
|
||||
MASTODON_TOKEN=07w3Emdw-cv_TncysrNU8Ed_sHJhwtnvKmnLqKlHmKA
|
||||
Loading…
x
Reference in New Issue
Block a user