AI Engineer

Plexus has partnered with an early-stage AI company building an evaluation platform that helps enterprise teams validate generative AI models and agents before they go live. The product is already live, with a public leaderboard benchmarking leading LLMs, and the team is now growing. The role You will build the infrastructure that runs evaluations of frontier models and agents at scale. This is a hands-on role with real ownership, working directly with the founders. Responsibilities Build scalable pipelines for benchmark execution, scoring and results aggregation across multiple LLM providers and custom endpoints Develop systems that generate custom evals on demand from customer data Design secure, multi-tenant services and APIs to enterprise privacy standards Own orchestration, observability, performance and cost as usage grows Experience required Proven experience building AI systems and AI infrastructure in production Strong Python (or similar) and backend or distributed systems skills Hands-on experience with LLM APIs and open or self-hosted models Cloud (AWS, GCP or Azure), Docker/Kubernetes and infrastructure-as-code Comfortable in an early-stage, fast-moving environment Nice to have Experience with LLM evaluation, benchmarking or agent frameworks Startup or scale-up background