Personal Work
Projects
Applied machine learning and cloud data engineering, built end-to-end outside of professional engagements.
QGI: Quantitative Geopolitical Intelligence
Active DevelopmentAn applied-machine-learning platform that reads six decades of national indicators to find countries tracing the same trajectories that preceded past crises, and surfaces the real historical precedents behind every signal.
Status: research-grade and in active validation. QGI reports how strongly a country resembles historical precedents, ranked, evidenced, and honest about uncertainty, rather than issuing probabilistic forecasts.
QGI is a solo-built, seven-repository system running on a multi-account AWS backbone, live at qgintelligence.com. It has grown through two engines. The Classic Cascade, a pure-statistics correlation pipeline, mines billions of cross-country indicator correlations to assemble event “recipes.” The Trajectory Engine, the current standard, wraps each recipe in its own machine-learning model that learns which indicators matter, over what horizon, and how closely a country today resembles the historical cases.
Engine 1
Classic Cascade: the statistical core
A pure-statistics pipeline: for every pair of countries and every indicator it computes time-shifted Pearson correlations (SCDIs), aggregates them into multi-indicator Patterns, and binds those to real historical events to form Recipes, reusable fingerprints of what tends to precede a given kind of crisis. One full run computes on the order of billions of correlations across 180+ countries.
- Start
-
0
Phase 0: Hygiene
-
Janitor
Removes stale intermediate data before each run.
-
-
1
Phase 1: Ingest
-
Indicator Fetcher
Pulls macro indicators (World Bank, V-Dem, FAOSTAT) → Parquet → S3.
-
Dispatcher A
Fans out one Fargate Spot task per indicator.
-
Workers Download & Hash 21 min · hard
Pipeline A workers download and hash indicator data from S3.
-
Reporter
Tallies which indicators were updated; emits
updated_ids.
-
- updated_ids · updated_count
-
Updates Found?
yes → Phase 2 none → skip to Phase 5 -
2
Phase 2: Compute · SCDI Engine
-
Batch Orchestrator (Trigger B)
Submits Pearson cross-correlation jobs across all country pairs to Graviton Spot instances.
-
↻ every 30 min · until Batch completes
-
Poll Interval 30 min · poll
-
Batch Checker
Counts still-running Batch jobs.
-
Batch Done?
↻ jobs_running > 0, repeat all done, proceed
-
-
- SCDI Parquet → S3
-
3
Phase 3: Discovery · Glue Crawler
-
Start Crawler
Launches
qgi-scdis-partition-finderto auto-discover new Parquet partitions written by Phase 2. -
↻ every 5 min · until crawler is READY
-
Poll Interval 5 min · poll
-
Check Crawler
Reads crawler state from Glue API.
-
Crawler Ready?
↻ State ≠ READY, repeat READY, proceed
-
-
- partitions registered
-
4
Phase 4: Synthesis · Patterns & Recipes
-
Dispatcher C
Fans out workers via SQS to aggregate SCDIs into multi-indicator Patterns, then bind them to historical events as Recipes.
-
Workers Build Patterns & Recipes 60 min · hard
Pipeline C workers compute the Patterns dataset and event-bound Recipes, writing results to S3.
-
Repair Table
MSCK REPAIR TABLE patterns, registers new Parquet partitions for querying.
-
- patterns & recipes → queryable
-
5
Phase 5: Ops Report
-
DLQ Scanner
Surfaces failed messages and unprocessed items; produces final health report. Also the landing point when Phase 1 finds no updates.
-
- Workflow Complete
⏱ Min runtime ~2h+ (Batch-dependent) · Step Functions polls async, zero idle compute cost during waits
Engine 2
Trajectory Engine: the ML layer
currentEach recipe becomes its own supervised model. A per-recipe “walk” selects the handful of indicators that carry signal, searches for the horizon over which the pattern plays out, and validates everything with leave-one-country-out cross-validation, so a model is never scored on a country it trained on. Surviving models rank today’s countries by SHAP-weighted similarity to the historical precedents.
- Recipe + Labels
-
1
Feature Substrate
-
Feature Substrate
250 macro indicators (World Bank · V-Dem · FAOSTAT) as a shared columnar feature store, plus curated and verified event labels (positives).
-
- indicators + labels
-
2
Per-Recipe Walk F-ML-1
-
Univariate Pre-filter
Ranks 250 indicators by leave-one-country-out signal and keeps the top 40, capping the expensive search so the indicator pool can grow without exploding compute.
-
Indicator Selection + Window Search
Greedy forward selection over indicators, jointly searching the canonical horizon {3 … 25 years}; a candidate joins the recipe only if it lifts validated AUC.
-
Train + Leave-One-Country-Out AUC
XGBoost per recipe (logistic fallback for sparse classes), scored by leave-one-country-out cross-validation so no country is ever evaluated on itself.
-
Clears LOCO-AUC validation floor?
validated directional / experimental
-
- canonical shape + indicator weights
-
3
Similarity-as-Risk Scoring
-
Similarity-as-Risk Scoring
Each country’s recent trajectory is compared to the recipe’s canonical shape via SHAP-weighted cosine similarity, mapped to a 0–100 resemblance score with a confidence band.
-
- 0–100 resemblance + precedents
-
4
Recipe-Fleet · AWS Batch
-
Parallel Fleet Fan-Out
A canary-gated Step Functions Map fans every runnable recipe across right-sized 2-vCPU Graviton Spot instances (c7g / m7g), writing per-recipe models back to S3, a full fleet run costs a few dollars.
-
- Trend Tracker · live
Resemblance ranks, not probabilities · leak-guarded (leave-one-country-out) · research-grade & in validation