Hp Systems Evaluation Engineer to assess AI software performance and customer experience outcomes. Develop evaluation methodologies and interpret results to guide design and implementation decisions.
Responsibilities
We are looking for a Systems Evaluation Engineer to continuously assess whether our AI software delivers the customer experiences and system performance it is designed to achieve. You will connect user stories and product goals to measurable outcomes, develop evaluation methodologies, and interpret results throughout development.
Working closely with product, architecture, and engineering teams, you will evaluate AI workloads, compare model and runtime implementations, and explain how technical performance affects the customer experience. Your findings will help teams understand trade-offs, identify gaps, and make better design and implementation decisions.
This role requires deep technical understanding of AI inference and PC systems, combined with strong analytical judgment. You will own the evaluation approach and interpretation of evidence, providing continuous feedback that shapes the developing solution.
What You Might Do
Translate user stories and intended customer experiences into measurable success criteria, evaluation scenarios, and performance targets.
Design repeatable benchmarking frameworks that assess AI workload effectiveness and system behavior throughout development.
Compare model formats, model runners, runtime implementations, and hardware configurations using representative workloads and controlled measurements.
Analyze inference behavior, including prompt processing, time to first token, token generation rate, latency, and memory consumption.
Evaluate system responsiveness and resource utilization under individual and concurrent AI workloads, including effects on other PC activities.
Interpret benchmark and telemetry data to distinguish meaningful improvements, regressions, measurement variability, and gaps in evaluation coverage.
Provide actionable feedback to architects and engineers, connecting technical findings to customer impact and recommending further investigation.
Build automation and reporting tools that make evaluations reproducible and results accessible to development teams.
Qualification
Knowledge of attention mechanismsFamiliarity with CPUExperience with statistical analysisThe pay range for this role is $130
Preferred
Experience evaluating AI workloads on Windows and Linux PCs or other edge devices.
Knowledge of attention mechanisms, KV caching, quantization, and other inference optimization concepts and their implications for evaluation.
Experience analyzing device telemetry and combining benchmark results with observations from representative customer scenarios.
Familiarity with CPU, GPU, and NPU execution and the challenges of comparing performance across heterogeneous hardware.
Experience with statistical analysis, experimental design, and methods for separating real performance changes from measurement noise.
Background in advanced systems validation with substantial depth in AI inference, performance analysis, and benchmarking automation.
The pay range for this role is $130,700 to $205,200 USD annually with additional opportunities for pay in the form of bonus and/or equity (applies to United States of America candidates only). Pay varies by work location, job-related knowledge, skills, and experience.
