Lifecycle stage Capability Client tools Capture Logging, ingestion, scraping Falco, Logview, Scribe, Hive ETL, Scrapy, Playwright Move Pub/sub and streaming Scribe event streams, Scuba Refine Enriching, cleansing, aggregation Daiquery, Presto / Onyx, Hive, Confucius Store & secure Storage and access control Hive, Manifold, XDB, TAO, Everstore, AMP, Access Gate, CAT, OIDC Serve Semantic layer and visualization IData, Unidash, Vega, Nest Orchestration Scheduling, retries, backfills, SLAs, CI/CD Dataswarm, Sandcastle Configuration Feature flags, rollouts, experiments Configerator, Sitevars, JustKnobs, Gatekeeper Observability Monitoring, alerting, errors, on call ODS, Scuba, ErrorTool, Oncall Hub Development Source control, code review, IDE Sapling, Phabricator, Diffs, CodeHub, Paste, EdenFS, VS Code @ client Investigation Entity debugging, data discovery ID Tool, IData, Intern Search AI tooling . Tool Expected use Metamate Knowledge retrieval, document summaries, custom commands for repetitive tasks Confucius Agents for data collection, data quality monitoring, anomaly detection and reporting Claude (Opus, Sonnet) Pipeline code and code review, workflow scripting, documentation, compliance support Analytics Agent Finding data, analysing it and answering business questions Professional skills and qualifications 5 10 years of data engineering experience, with production pipelines at petabyte scale.