Squad PM - Data & ML Evaluation

You’ll be the product owner for all things product data and ML evaluation for Ask. You’ll define the data strategy, instrumentation, experimentation, and model evaluation loops that drive product decisions and model quality. You’ll lead a data and eval squad, partnering closely with PMs, data scientists, ML engineers, and platform teams to ensure we ship measurable, safe, and delightful AI features. You’ll also help ground our pricing strategy in data.

What You’ll Do

• Product Data & Analytics

  • Own the product data stack end-to-end across squads: telemetry, event schemas, tracking plans, data quality, warehousing, and accessibility of metrics both for our on-premise and our SaaS offerings.
  • Define the product metrics framework (north stars, input metrics, leading indicators) and help stakeholders build reliable dashboards.

• ML Evaluation & Experimentation

  • Build/own ML evaluation loops: curate test sets, define KPIs (quality, safety, bias, latency, cost), run offline/online evals, help evaluate agents, tools, finetunes, and retrieval changes; maintain an eval pipeline that’s reproducible and automated in CI.
  • Lead experimentation: design and analyze A/B tests - help create a framework for testing our customers’ agents and our “hyper-parameters.”

• Strategy & Governance

  • Ground pricing strategy in data: analyze feature/cost relationships, develop value-based pricing models, and create metrics that link usage patterns to pricing tiers.
  • Set data standards: event naming, governance, PII handling, retention, and privacy-by-design in collaboration with Security/Legal teams.

Requirements

• 5–8+ years in product roles with a strong engineering or data background (e.g., software, data engineering, ML engineering, analytics engineering) in tech product companies.
• Proven ownership of instrumentation, analytics, and experimentation for a product surface or platform.
• Deep comfort with SQL and experimentation stats (power, MDE, CUPED, guardrail metrics); able to sanity-check analyses and run your own when needed.
• Practical experience working with ML-powered products and their evaluation (offline test sets, online metrics, drift/monitoring).
• Strong product sense + stakeholder management; able to drive clarity across PM, DS, Eng, and Execs.
• Comfortable using AI tools to prototype analyses or small utilities (e.g., generate code for ETL, dashboards, or validators).

Nice-to-Haves

• Hands-on with Amplitude (or similar product analytics) and MLflow (or similar experiment tracking).
• Experience with RAG/LMM evals, safety/guardrail metrics, and red-teaming.
• Familiarity with data modeling (e.g., dbt), feature stores, and observability (e.g., Monte Carlo, Great Expectations).
• Experience in privacy-sensitive / regulated environments.

How You’ll Work

• Embedded in a “Data and Eval” within Ask, collaborating with the rest of the Ask team (~50 engineers, ~4 PMs, 1 PMM, and ~3 Product Designers).
• Close partnership with customers, Security/Legal (for data governance), Strategy and Finance teams.