

DataNodes is an enterprise-grade distributed media ingestion, cloud storage virtualization, and sub-second encrypted video streaming platform powering MJ Academy. It aggregates 15TB+ of decoupled cloud storage accounts into a unified virtual storage tier with dynamic load-balancing, real-time AES-256-CTR random-access stream decryption, automated rate-limit circuit breakers, and zero-proxy serverless streaming handoffs.

"st1_" + Base64URL([12-byte IV] + [16-byte Auth Tag] + [AES-256-GCM Ciphertext]){ f: driveFileId, a: accountId, n: fileName, iat: issuedAt, exp: expirationTimestamp }). Any client tampering immediately triggers a 403 Forbidden./api/stream-url enforces per-IP token bucket rate limiting (40 req/min) combined with strict HTTP Referer and Origin validation to prevent hotlinking and third-party embedding.Block Index = floor(P_start / 16)
Sub-Block Offset = P_start % 16
Seek IV = (IV_original + Block Index) mod (2^128)
Upstream Range = [(Block Index * 16) + 16, P_end + 16]
Seek IV. If Sub-Block Offset > 0, initial fractional bytes are discarded, and the decrypted stream is immediately piped to the client under HTTP status 206 Partial Content.Promise.any(), auto-associating the file with the responding account.403 userRateLimitExceeded:
is_active = FALSE in PostgreSQL, ejects the account from the ingestion load balancer rotation, and routes incoming uploads to alternative active accounts.is_active = TRUE in PostgreSQL and re-admits the account into the load-balancer rotation without restarting any services.-mmt=on) into fixed 1GB split volumes (-v1024m). If under 1GB, output normalizes into a single clean Course_Materials.zip.dlengine) with 16 parallel TCP socket streams per part. Multi-part archives are unlinked from disk the millisecond extraction succeeds; loose materials are unlinked the moment 7-Zip split volumes finalize; split volumes and videos are unlinked immediately upon receiving 200 OK from cloud upload.| Engineering Vector | Architecture Strategy | Performance & Reliability Metric |
|---|---|---|
| Stream Cipher | AES-256-CTR with 128-bit modular block counter arithmetic | Sub-second HTTP 206 Partial Content random-access timeline seeking |
| Token Security | AES-256-GCM authenticated encryption (12-byte IV + 16-byte Auth Tag) | 2-Hour TTL hard expiration, HMAC-SHA256 signature verification |
| Access Control | Strict Referer & Origin validation with per-IP token bucket limits | Zero-leak file IDs; 40 req/min endpoint rate limiting prevents hotlinking |
| Storage Topology | Decoupled Multi-Account Google Drive & Object Storage virtualization | 15TB+ aggregated pooled capacity with automated quota balancing |
| Circuit Breaker | PostgreSQL account status flip with 60s probe recovery loop | Zero read downtime on 403 userRateLimitExceeded with automatic healing |
| Course Packaging | Multi-threaded 7-Zip (-mmt=on, -v1024m 1GB split volumes) | 99.9% inode drop (from 300,000 tiny files to ~30 clean archive parts) |
| Download Engine | Custom compiled Go crawler (dlengine) with 16 parallel sockets | 120GB+ course libraries ingested within ephemeral 40GB scratch disks |
| Web Architecture | Next.js 14 SSR App Router with decoupled Node.js streaming daemons | Bypasses serverless timeout limits and serverless egress bandwidth costs |