Collegium:Imperium System: Difference between revisions

Jump to navigation Jump to search
AdminIsidore (talk | contribs)
aether-access, Bureaucrats, contubadmin, duffers, Interface administrators, latin, liberty, mystery, proveit, Administrators (Semantic MediaWiki), Curators (Semantic MediaWiki), Editors (Semantic MediaWiki), Suppressors, swift, Administrators, Upload Wizard campaign editors, visualdata-admin, visualdata-editor, wayfinder
675 edits
Created page with "{{DISPLAYTITLE:Imperium System Mission Plan}} {{italic title}} == Overview == This document outlines the mission plan for building and testing the Imperium system, a distributed data processing pipeline. Each mission is designed to be executed in a separate thread, with phased OODA (Observe, Orient, Decide, Act) loops, and validated via independent tests upon completion. == Mission List == {| class="wikitable" |+ ! Mission Name ! Description ! Key Tools & Objectives |-..."
 
AdminIsidore (talk | contribs)
aether-access, Bureaucrats, contubadmin, duffers, Interface administrators, latin, liberty, mystery, proveit, Administrators (Semantic MediaWiki), Curators (Semantic MediaWiki), Editors (Semantic MediaWiki), Suppressors, swift, Administrators, Upload Wizard campaign editors, visualdata-admin, visualdata-editor, wayfinder
675 edits
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{DISPLAYTITLE:Imperium System Mission Plan}}
{{DISPLAYTITLE:Collegium:Imperium System}}
{{italic title}}
'''Imperium System Mission Plan'''
 
== Overview ==
== Overview ==
This document outlines the mission plan for building and testing the Imperium system, a distributed data processing pipeline. Each mission is designed to be executed in a separate thread, with phased OODA (Observe, Orient, Decide, Act) loops, and validated via independent tests upon completion.
The Imperium System Mission Plan outlines the phased construction and testing of the Imperium, a distributed data processing pipeline. Each mission is executed in a separate thread using OODA (Observe, Orient, Decide, Act) loops, with completion validated via independent tests. The plan adheres to the Lingua standard, using Latin nomenclature (e.g., ''aqua_datum'', ''grana_datum'', ''pomerium'', ''flamen_martialis'') to ensure script interoperability and support quarterly redundancy audits for AI training.


== Mission List ==
== Mission Plan ==
{| class="wikitable"
 
|+
The following table details the 14 missions, each with tools, dependencies, and objectives to build a unified, secure, and efficient system.
! Mission Name
 
{| class="wikitable sortable"
! Mission
! Description
! Description
! Key Tools & Objectives
! Tools/Dependencies
|-
! Objectives
! colspan="3" | Phase 1: Pomerium & Internal Network Foundation
! Status
|-
|-
| '''NFS Setup on Roma, Horreum, and Torta'''
| '''NFS Setup on Roma, Horreum, and Torta'''
| Configure NFS mounts to unify Roma, Horreum, and Torta's "grana" drive (~698 GB) as a single logical system within the Pomerium. This enables seamless file sharing and supports the "single machine" goal.
| Configure NFS mounts to unify ''Roma'', ''Horreum'', and ''Torta'' (smaller HDD, ~698 GB) as a single logical system within ''Pomerium'', enabling seamless file sharing for scripts and ''grana_datum''. Ensures no race conditions and supports the "single machine" goal.
|
| NFS (''nfs-kernel-server'', ''nfs-common''); configure ''/etc/exports'' on ''Torta'', mount on ''Roma''/''Horreum''
* '''Tools:''' <code>nfs-kernel-server</code>, <code>nfs-common</code>
| Read/write access across nodes with static IPs, tested via file creation/listing on ''pomerium_via'' paths
* '''Objective:''' Full read/write access to the "grana" share on Torta from both Roma and Horreum, verified via file creation and listing.
| Completed
|-
|-
| '''NFS-Plus GPU Dispatching'''
| '''NFS-Plus GPU Dispatching'''
| Extend the NFS setup to enable scripts on Roma or Torta to dispatch GPU-intensive tasks to Horreum’s NVIDIA RTX 5060 Ti, using the shared filesystem for data access.
| Extend NFS setup to enable ''Roma'' or ''Torta'' scripts to dispatch GPU-intensive tasks (e.g., AI processing) to ''Horreum''’s NVIDIA RTX 5060 Ti, preserving energy efficiency. Builds on NFS for unified data access.
|
| NFS mounts, CUDA toolkit on ''Horreum'', SSH-based job dispatching (e.g., ''ssh'' or SLURM)
* '''Tools:''' NFS, CUDA Toolkit, SSH
| Run a sample GPU task (e.g., Python/CUDA script) from ''Roma'' using ''Horreum''’s GPU
* '''Objective:''' Successfully run a sample GPU task (e.g., a small AI inference) initiated from Roma that reads/writes data on the NFS share and executes on Horreum's GPU.
| Pending
|-
| '''Preparing Dockers and Directories on Latium and Torta'''
| Set up Docker containers (''Pomerium'', ''Campus Martius'', ''Flamen Martialis'') on ''Latium'' and minimal directory structure on ''Torta'' (e.g., ''/mnt/lacus'', ''/mnt/aquaeductus'') for pipeline operations. Simplifies ''Torta'' by keeping it Docker-free.
| Docker, ''.bashrc'' modifications, directory scripts
| Functional containers and directories, tested by mock commands in each context
| Pending
|-
|-
! colspan="3" | Phase 2: Aquaeductus & External Pipeline Foundation
| '''NFS-Plus Setup on Torta Hard Drives and Pomerium on Latium'''
| Configure ''Torta''’s external HDDs (larger ~1.8 TB for ''lacus'', smaller ~698 GB for ''aquaeductus'') with NFS, integrating ''Latium''’s ''Pomerium'' Docker into the internal NFS network. Ensures secure data flow from external to internal zones.
| NFS, WireGuard, ''ufw''
| Read-only NFS access from ''Latium'' to ''Torta''’s smaller HDD, tested via mount and file read
| Pending
|-
|-
| '''Preparing Dockers and Directories on Latium and Torta'''
| '''Flamen Martialis and Salii Separation'''
| Set up the Docker containers (Pomerium, Campus Martius, Flamen Martialis) on Latium and the required directory structure on Torta's "aqua" drive (`/mnt/aqua/aqua_datum_raw`) for pipeline operations.
| Implement ''Flamen Martialis'' in ''Latium''’s ''Campus Martius'' Docker for external data collection/sanitation, with ''Salii'' on ''Roma'' for internal processing, reducing ''Latium''’s role and vulnerabilities. Ensures ''Salii'' is air-gapped, using ''Horreum''’s GPU.
|
| Python, NFS, SSH
* '''Tools:''' Docker, <code>.bashrc</code>, Shell scripts
| ''Flamen Martialis'' collecting ''aqua_datum'' and ''Salii'' processing to ''grana_datum'', tested with a mock dataset
* '''Objective:''' Functional containers and directories, tested by mock commands in each context.
| Pending
|-
| '''Simple Data Diodes'''
| Establish a one-way data flow from ''Latium'' to ''Torta'' (''Campus Martius'' to ''Pomerium'') to prevent reverse communication, mitigating security risks. Focuses on lightweight, secure transfer protocols.
| RSYNC, ''ufw'', WireGuard
| One-way ''aqua_datum'' push to ''/mnt/lacus'', tested by verifying no reverse access
| Pending
|-
| '''RSYNC Optimization'''
| Optimize RSYNC for fast, secure one-way data transfers over WireGuard, replacing SCP to avoid bottlenecks in pipelines like NOTAM. Tunes MTU and compression for performance.
| RSYNC, WireGuard, cron
| Transfer mock JSON files in <1s, tested by comparing transfer times
| Pending
|-
|-
| '''Simple Data Diodes & RSYNC Optimization'''
| '''Tar + Netcat (nc) Implementation'''
| Establish a fast, secure, one-way data flow (the Aquaeductus) from Latium to Torta using RSYNC over the WireGuard tunnel. This replaces SCP to avoid bottlenecks.
| Implement tar + nc for burst/large dataset transfers, comparing with RSYNC to determine the best tool per task (e.g., NOTAM vs. musica). Establishes a decision process for tool selection.
|
| Tar, Netcat, WireGuard
* '''Tools:''' RSYNC, WireGuard, <code>cron</code>
| Functional burst transfer with a decision matrix, tested with mock data
* '''Objective:''' A one-way push of `aqua_datum` to Torta's "aqua" drive, tested by verifying transfer speed and the inability to initiate a connection from Torta back to the originating service on Latium.
| Pending
|-
|-
| '''Firejail/Bubblewrap Sandboxing'''
| '''Firejail/Bubblewrap Sandboxing'''
| Deploy Firejail on Latium to sandbox the `Flamen Martialis` script, ensuring secure processing of external `aqua_datum` by restricting its filesystem and network access.
| Deploy Firejail (or Bubblewrap) on ''Latium'' to sandbox ''Flamen Martialis'' scripts, ensuring secure processing of external ''aqua_datum''. Avoids heavy Firecracker setup.
|
| Firejail, Python
* '''Tools:''' Firejail, Python
| Sandboxed mock script with restricted access, tested via confinement checks
* '''Objective:''' A sandboxed mock script with verifiably restricted access to the host system.
| Pending
|-
|-
! colspan="3" | Phase 3: End-to-End Pipeline Integration & Testing
| '''Supabase Integration'''
|-
| Integrate Supabase as a filtering buffer for ''aqua_datum'', using RLS and edge functions to validate data before transfer to ''Torta'' or ''Roma''. Enhances security and supports prototypes.
| '''Flamen Martialis and Salii Separation'''
| Supabase client libraries, REST API, WireGuard
| Implement the full Flamen/Salii workflow: Flamen on Latium collects and sanitizes `aqua_datum`, which is then transferred to Torta. Salii on Roma detects the new data and orchestrates internal processing.
| Validated data push/pull, tested with a mock schema
|
| Pending
* '''Tools:''' Python, RSYNC, NFS, SSH
* '''Objective:''' Flamen Martialis successfully collects `aqua_datum`, and Salii on Roma successfully processes it to `grana_datum` on the NFS share.
|-
|-
| '''JSONPlaceholder Data Pipeline Test'''
| '''JSONPlaceholder Data Pipeline Test'''
| Test the full, simple pipeline using JSONPlaceholder’s mock API, simulating the data flow from Latium -> Torta -> Roma -> OodaWiki. This validates the entire end-to-end architecture.
| Test the full pipeline using JSONPlaceholder’s mock API, simulating data flow from ''Latium'' to ''Torta'' to ''Roma''/''OodaWiki''. Validates end-to-end setup.
|
| Python, RSYNC/nc, NFS, pywikibots
* '''Tools:''' Python, RSYNC, NFS, pywikibots
| Complete data cycle, tested by verifying output on ''OodaWiki''
* '''Objective:''' A complete data cycle, verified by seeing the mock data correctly published on an OodaWiki page.
| Pending
|-
|-
| '''NOTAM Data Pipeline Test'''
| '''NOTAM Data Pipeline Test'''
| Test the pipeline with a more complex, authenticated source using the FAA's NOTAM API. This focuses on scheduled pulls and performance with real-world data.
| Test the pipeline with NOTAM API data, focusing on scheduled pulls and performance. Ensures reliable handling of time-sensitive data.
|
| Python, Supabase (optional), RSYNC/nc, NFS
* '''Tools:''' Python, RSYNC, NFS
| NOTAM ingestion to ''Roma'' SQL or ''OodaWiki'', tested by data accuracy
* '''Objective:''' Successful and reliable ingestion of NOTAM data, stored as `grana_datum` on the NFS share.
| Pending
|-
! colspan="3" | Phase 4: Optimization & Future Development
|-
|-
| '''Tar + Netcat (nc) Implementation'''
| '''RapidAPI via Supabase Test'''
| Implement and benchmark `tar` + `nc` for large, one-time "burst" transfers and compare its performance to RSYNC to establish a decision matrix for future pipeline tool selection.
| Test a basic RapidAPI endpoint via Supabase for filtering, integrating with the pipeline to store/publish results. Validates external API handling.
|
| Supabase, Python, RSYNC/nc, pywikibots
* '''Tools:''' Tar, Netcat, WireGuard
| API-to-Wiki flow, tested by published data on ''OodaWiki''
* '''Objective:''' A functional burst transfer and a documented decision matrix for when to use RSYNC vs. `tar` + `nc`.
| Pending
|-
| '''Supabase Integration'''
| Integrate Supabase as an optional, advanced filtering buffer for `aqua_datum`, using its edge functions and Row-Level Security (RLS) to validate data before it enters the Aquaeductus.
|
* '''Tools:''' Supabase client libraries, REST APIs
* '''Objective:''' A validated data push/pull through Supabase, tested with a mock schema and RLS policy.
|-
|-
| '''Automation/Standardized Deployment Script'''
| '''Automation/Standardized Deployment Script'''
| Develop a `creo_castellum.sh` CLI script to automate the setup of new data pipelines across Imperium, based on the lessons learned from the manual builds.
| Develop a CLI script to automate directory and tool setup for new projects (e.g., musica, NOTAM) across Imperium, using test lessons. Ensures consistent, customizable deployments.
|
| Bash/Python, Docker, NFS, Supabase
* '''Tools:''' Bash/Python, Docker, NFS
| Script for project setup with one command, tested by deploying a mock project
* '''Objective:''' A script that can provision a new data pipeline with a single command, tested by deploying a new mock project.
| Pending
|}
|}
== Execution Plan ==
Each mission will be executed in a dedicated thread using OODA loops:
* '''Observe''': Assess current system state (e.g., installed packages, configurations).
* '''Orient''': Plan configurations and identify dependencies.
* '''Decide''': Select specific tools and parameters.
* '''Act''': Implement and test the setup.
Upon thread completion, results are reported to the main strategic thread, where an independent test (e.g., file access, data transfer, script execution) confirms success. Successful missions are closed, and the next thread is initiated. The main thread tracks progress, ensuring coherence with the Imperium’s Lingua conventions and strategic goals.
== Notes ==
* Missions adhere to the Lingua standard, using Latin terms (e.g., ''pomerium_via'' for NFS paths, ''frumentarii_transfer'' for RSYNC jobs) to support script interoperability and quarterly audits for AI training.
* The plan prioritizes foundational infrastructure (NFS, GPU dispatching) before security mechanisms (data diodes, sandboxing) and pipeline tests, culminating in automation for scalability.
* Mission 1 (NFS Setup) was completed with fixed port configurations for NFS services and verified through multi-node file creation, concurrent writes, and cleanup tests, establishing unified access in Pomerium.
Retrieved from "https://www.ooda.wiki/wiki/Collegium:Imperium_System"