Senior AI Systems Quality Engineer

Abacus Insights is moving beyond traditional software testing to solve the unique challenges of AI in healthcare. As a Senior AI Systems Quality Engineer, you won't just find bugs; you'll architect the platform that ensures agentic systems behave predictably. You’ll be responsible for embedding safety and trust into the heart of AI workflows, making sure that large language models provide accurate, reliable data for critical healthcare decisions.

This is an automation-first environment where your proficiency with Python or TypeScript meets the cutting edge of AI orchestration. You will work within a Databricks architecture, leveraging MLflow to track the lineage and performance of AI models. Because the role involves managing non-deterministic behavior and 'vibe coding' iterations, you’ll be at the forefront of defining how prompt engineering and agent logic are versioned, tested, and deployed at scale.

Core Responsibilities

• Architect Validation Pipelines: Build automated quality gates specifically designed for LLM-driven systems.
• Engineer AI Guardrails: Design custom harnesses to stress-test agentic tool-use, memory, and decision logic.
• Scale Evaluation: Create massive, scenario-based test suites that cover long-tail AI failures and edge cases.
• Integrate Quality Signals: Connect AI-specific metrics like hallucination rates and grounding directly into the CI/CD lifecycle.

Joining this team means shaping the industry standard for AI governance in a highly regulated space. You’ll have the autonomy to build reusable libraries that define the future of AI quality engineering, all while working in a flexible, remote-friendly environment that values curiosity and bold technical experimentation.