Toolvana: production-ready media tools platform (CodeIgniter 4)
- 134-tool registry with programmatic SEO (unique titles/H1/descriptions, JSON-LD graphs, sitemap index, canonical 301 enforcement via required filter) - DB-backed job queue (SKIP LOCKED) with drivers: Ffmpeg, Images (GD), Pdf (qpdf/gs/poppler), Youtube (thumbnails), Qr (server-side PNG) - Security: SSRF guard, MIME validation, rate limits, API-key auth, bcrypt admin login, security headers - Admin panel: dashboard, tools/categories/guides CRUD, SEO audit, analytics, job inspector with retry, system health, feature flags - Docker deployment (nginx + web/api FPM pools + scalable workers), PHPUnit suite (19 tests / 1139 assertions), PWA manifest + service worker
This commit is contained in:
deepseek
@@ -0,0 +1,125 @@
|
||||
# Toolvana — Your free online media toolbox
|
||||
|
||||
A production-ready, SEO-first all-in-one media tools platform built on **CodeIgniter 4.7** and **PHP 8.2**: YouTube tools, video/audio/image/GIF converters, PDF utilities, QR generation, programmatic SEO at scale (134+ root-level tool pages), a durable job queue with FFmpeg workers, anonymous analytics, and a complete admin panel.
|
||||
|
||||
## Feature overview
|
||||
|
||||
| Area | What ships |
|
||||
|---|---|
|
||||
| **Tools** | 134 seeded tools across YouTube / video / audio / image / GIF / PDF / utility categories; URL-kind, upload-kind and text-kind flows |
|
||||
| **Processing** | MySQL-backed queue (`SELECT … FOR UPDATE SKIP LOCKED`), driver architecture: `Ffmpeg`, `Images` (GD), `Pdf` (qpdf/Ghostscript/poppler), `Youtube` (thumbnails via i.ytimg.com; yt-dlp optional & policy-gated), `Qr` (server-side PNG) |
|
||||
| **Security** | SSRF guard for URL fetches, extension **and** real-MIME validation, UUID storage names outside the web root, per-session/IP hourly rate limits, API-key auth with hashed lookup, admin bcrypt login, strict security headers, canonical 301 enforcement |
|
||||
| **SEO** | Unique titles/H1s/descriptions generated per format pair, JSON-LD (WebApplication, BreadcrumbList, FAQPage, HowTo, Article), auto sitemap index split at 5 000 URLs, robots.txt blocking processing endpoints, noindex on search/API/admin, hreflang scaffolding, OG images rendered by GD |
|
||||
| **Admin** (`/admin`) | Overview dashboard, tools CRUD with pre-publish SEO gate, categories, guides (Markdown), automated **SEO audit** across every page, analytics dashboard (success-rate per tool), job inspector with retry, system health (binaries/storage/worker heartbeat), feature-flag settings |
|
||||
| **API v1** | `/api/v1/tools`, `/api/v1/jobs` (+ legacy `/api/*`), Bearer-token auth, per-key hourly quotas |
|
||||
| **PWA** | Web manifest, offline-capable service worker, offline shell |
|
||||
| **Privacy** | Anonymous session-hash analytics only, automatic media expiry (`media:cleanup`), DMCA/privacy/cookies pages |
|
||||
|
||||
## Requirements
|
||||
|
||||
- PHP 8.1+ (8.2 recommended) with `intl`, `mbstring`, `zip`, `pdo_mysql`, `gd`, `exif`
|
||||
- MariaDB/MySQL 10.6+
|
||||
- FFmpeg, ImageMagick (`convert`), poppler-utils (`pdftoppm`, `pdftotext`), qpdf, Ghostscript
|
||||
- Optional: `yt-dlp` (disabled by policy until enabled in Admin → Settings)
|
||||
- Composer 2
|
||||
|
||||
## Quick start (development)
|
||||
|
||||
```bash
|
||||
composer install
|
||||
cp env .env # then edit DB credentials
|
||||
php spark key:generate
|
||||
|
||||
# create schema + seed registry/content
|
||||
php spark migrate --all
|
||||
php spark db:seed InitialSeeder
|
||||
|
||||
# run
|
||||
php spark serve # http://127.0.0.1:8080 (falls back to 8081/8082 if busy)
|
||||
|
||||
# process jobs (keep one or more running)
|
||||
php spark queue:work --loop # add --sleep=2 to idle politely
|
||||
|
||||
# expire finished jobs + files (cron every 5 min in production)
|
||||
php spark media:cleanup
|
||||
```
|
||||
|
||||
Sign in to the admin panel at `/admin/login`. Set the password hash in `.env`:
|
||||
|
||||
```bash
|
||||
php -r 'echo password_hash("your-password", PASSWORD_DEFAULT), "\n";'
|
||||
# .env → site.adminPassword = '$2y$10$…'
|
||||
```
|
||||
|
||||
## Production deployment (Docker)
|
||||
|
||||
```bash
|
||||
cp .env.docker.example .env.docker # fill in secrets: encryption.key, site.adminPassword, DB password
|
||||
docker compose up -d --build
|
||||
docker compose exec web php spark migrate --all
|
||||
docker compose exec web php spark db:seed InitialSeeder
|
||||
```
|
||||
|
||||
Topology (`docker-compose.yml`):
|
||||
|
||||
```
|
||||
browser ── nginx ──► web (PHP-FPM, page rendering)
|
||||
└──► api (PHP-FPM, /api + processing endpoints isolated)
|
||||
workers × N (php spark queue:work --loop) ← scale replicas freely
|
||||
scheduler (media:cleanup every 5 minutes)
|
||||
db (MariaDB 11 + healthcheck)
|
||||
```
|
||||
|
||||
- The single `docker/Dockerfile` image contains PHP-FPM **and** the full media binary stack, so web/worker versions can never drift.
|
||||
- Media lives on named volumes (`media-storage`, `incoming-storage`); point them at shared NFS/EFS when running workers on separate hosts.
|
||||
- Put TLS termination at your edge/load balancer and set `app.forceGlobalSecureRequests = true`; nginx config includes HSTS-ready headers.
|
||||
- Recommended cron alternative to the scheduler container: `*/5 * * * * docker compose exec -T scheduler php spark media:cleanup`.
|
||||
|
||||
### Production checklist
|
||||
|
||||
- [ ] `CI_ENVIRONMENT = production`, strong `encryption.key` (`php spark key:generate`)
|
||||
- [ ] `site.adminPassword` set to a fresh bcrypt hash
|
||||
- [ ] `site.ytDlpEnabled` left `false` unless legal review is done
|
||||
- [ ] Review retention: `site.retentionHours` (default 2 h) matches your privacy policy
|
||||
- [ ] Point `/sitemap.xml` at Search Console; confirm canonical host in `.env` (`app.baseURL`)
|
||||
- [ ] Run Admin → **SEO Audit** and clear critical issues before launch
|
||||
|
||||
## Architecture notes
|
||||
|
||||
```
|
||||
app/
|
||||
├── Config/Site.php # brand, binaries map, limits, policy flags
|
||||
├── Controllers/ # Tools, Directory, Guides, Media (upload/start/download), Api/*, Admin/*
|
||||
├── Libraries/
|
||||
│ ├── Pipeline.php # enqueue guards + driver routing (by operation/format family)
|
||||
│ ├── Pipeline/{Ffmpeg,Images,Pdf,Youtube,Qr}.php
|
||||
│ ├── CatalogBuilder.php # generates the 134-tool registry with unique SEO copy
|
||||
│ ├── Seo.php # head tags, canonical, JSON-LD graphs
|
||||
│ ├── UrlGuard.php # SSRF protection for remote fetches
|
||||
│ ├── ToolSearch.php # weighted token search incl. aliases ("yt mp3")
|
||||
│ └── RelatedTools.php # internal-linking engine (format/category scoring)
|
||||
├── Commands/ # queue:work, media:cleanup
|
||||
└── Database/Migrations/ # categories, tools, guides, jobs, analytics_events, api_keys
|
||||
```
|
||||
|
||||
**Why a DB queue?** Zero extra infrastructure, transactional claims via `SKIP LOCKED`, trivial horizontal scaling of identical worker containers, and a built-in audit trail surfaced in the admin Jobs screen.
|
||||
|
||||
**Canonical URL policy:** every non-canonical variant (uppercase, trailing slash, doubled slashes) 301-redirects before routing — including unmatched routes — via a *required* filter.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
./vendor/bin/phpunit # 19 tests, ~1 100 assertions
|
||||
```
|
||||
|
||||
Suites:
|
||||
- `tests/unit/CatalogTest.php` — catalog integrity (unique slugs, complete SEO fields, format-specific titles), YouTube ID extraction, SSRF guard behaviour.
|
||||
- `tests/feature/SeoPagesTest.php` — the **SEO contract**: canonical/OG/Twitter tags, JSON-LD types present in first response, sitemaps contain only indexable URLs, robots.txt blocks processing endpoints, 301 canonicalisation, helpful 404s.
|
||||
|
||||
## Useful commands
|
||||
|
||||
```bash
|
||||
php spark queue:work [--loop] [--sleep=N] [--max-jobs=N] # process jobs
|
||||
php spark media:cleanup # expire jobs/files past retention
|
||||
php spark migrate --all && php spark db:seed InitialSeeder # rebuild registry (editorial fields preserved)
|
||||
```
|
||||
Reference in New Issue
Block a user