CI/CD Tower
Ohara
Mekari’s mobile engineering command center — build health, pipeline orchestration, release trains, and crash triage in one internal platform.
30-second summary
Disclosure: Anonymized case study
Problem
Build health lived in scattered logs, releases were tracked in a no-code spreadsheet app, and a 20-person mobile team had no single place to see what was failing, why, or what was shipping next.
Ownership
Sole designer and builder across the schema, webhook ingestion, dashboard UX, queue model, and AI failure analysis.
Key decision
Made webhooks the source of truth and the build queue a first-class, inspectable object so triggering, deduplication, and scheduling live in one coordination layer.
Result
- Opened daily by all 20 mobile developers as the team’s source of truth.
- Build success rose from 70% to 85%, while queue time fell from hours to minutes.
Trade-off
Kept Ohara in front of the existing CI as a coordination and visibility layer instead of replacing the CI system itself.
What I’d change now
Ship smaller and validate AI features sooner, then add predictive capacity warnings and richer team-level queue analytics.
Highlights
Context
Mobile build health at Mekari lived everywhere and nowhere. Build results were buried in logs, release planning happened in a no-code spreadsheet app, and a 20-person mobile team had no single place to answer basic questions: what's failing, why, and what's shipping next.
This sat inside a larger move — migrating 19 mobile projects off Bitrise onto in-house Jenkins running on two M4 Mac minis. Owning the infrastructure made the build queue a problem I could actually design for. Ohara started from one question: if build capacity and build health are shared, finite things, why is there no control tower for them?
It began as a tracking spreadsheet. It became a platform.
Approach
Ohara sits in front of the existing CI as a coordination and visibility layer, not a replacement. A few decisions shaped everything else:
- Webhooks are the source of truth. Every push and PR event flows through one rules layer before anything dispatches, so triggering logic lives in a single place instead of scattered across pipeline YAML.
- The queue is a first-class object. Jobs are explicit, inspectable entries with state — which is what makes deduplication and fair scheduling possible.
- Make failure legible. Raw build logs are hostile. Ohara pipes failures through an LLM to categorize root cause and likely responsible area, so triage starts from a summary instead of a scroll.
How it works
Ohara is a Next.js + Supabase web app, behind company SSO, with a few moving parts:
- Ingestion — a webhook endpoint normalizes build events from Jenkins and Bitbucket and writes them to Supabase (tens of thousands of build records and counting).
- Observability — success rate, average build time, stability scoring (success + volatility + recovery time), and automatic "at-risk project" detection.
- Orchestration — a queue layer between webhooks and runners: deduplication, priority scheduling, and a live view of what each runner (Sunny, Merry) is doing right now and what's next.
- AI failure analysis — build-log failures are summarized and tagged by category, so recurring problems surface instead of scrolling past.
- Release trains & crash triage — release planning and crash aggregation moved out of the old spreadsheet world and into the same app.
Outcome
The shift was from invisible and wasteful to observable and fair:
- Opened daily by all 20 mobile developers — the de facto source of truth.
- Build success rate climbed from 70% to 85% (~375 developer-hours saved from failed builds) as the runner and queue work landed.
- Queue time dropped from hours to minutes after re-architecting runners (VM-per-machine, 2 → 4 concurrent runners).
- Failure triage starts from an AI summary instead of a wall of logs.
An honest lesson lives in here too: the AI features I was proudest of were the least used — engineers preferred their own agents. Ship smaller, validate faster.
Want to feel how the queue behaves? The Ohara Release Lab is a public-safe, deterministic rebuild — tune runner capacity, caching, and duplicate cancellation against a synthetic release and watch the queue respond.
What I'd build next
Predictive capacity warnings before the queue backs up, richer per-team analytics to tune the fairness policy with data, and finishing the release-train manager (Water Seven) so shipping is as observable as building.