Model Test and Measurement Engineer at Defcon · LumiJobs
mid
$150k–$190k
H1B: no
Defcon Model Test and Measurement Engineer at Defcon AI owns the evaluation framework for AI-assisted platforms, ensuring model performance and trustworthiness. This role independently validates models and produces evidence for release decisions.
Responsibilities
You'll join the analytics and AI engineering team behind a system that genuinely matters: an AI-assisted platform that brings together records from dozens of disparate data sources, resolves them to the correct individual, highlights what analysts should review first, and provides transparent, explainable recommendations that users can trust. Operating within a secure government cloud environment, the platform tackles complex challenges in AI, data integration, and decision support where quality, trust, and accountability are mission-critical.
As a Model Test & Measurement Engineer, you'll own the evaluation framework that helps ensure those systems perform as intended. You'll build and maintain labeled ground truth datasets, design statistically sound audit and sampling methodologies, measure model performance across releases, and create the evidence packages that support deployment decisions. You'll independently validate both scoring and generative AI capabilities, helping the team understand not only whether a model works, but how confidently its outputs can be trusted.
This is a role with genuine influence. Your assessments will inform release decisions, drive improvement efforts, and provide the objective evidence customers rely on when evaluating system performance. You'll work closely with data scientists, AI engineers, and technical leadership while maintaining the independence needed to provide clear, defensible evaluations. You do not build the models you validate, and you do not approve thresholds or release authorization against your own evidence.
If you're energized by measurement, validation, and making complex AI systems more trustworthy, this is an opportunity to have an outsized impact on both the technology and the mission it supports.
This is a fully remote role with occasional travel to DEFCON AI headquarters, customer sites, and partner facilities as needed.
Construct labeled ground truth for model evaluation
Design and run sampled audits
Gate releases against model version; maintain version inventory, evaluation records, and rollback triggers
Run drift and override review
Produce human-oversight and fairness / disparate-effect evidence
Maintain independence from the build roles: produce evaluation evidence, but do not build the models, approve thresholds, or authorize releases against that evidence
Measure workflow improvement: review time, throughput, backlog movement, and override and rework rates
Qualification
Strong Python and SQLUS Citizenship RequiredActive US Secret clearanceNIST AI RMF or comparable practiceActive Top Secret clearanceWhat Success Looks LikeWorkflow metrics (review time
Required
5+ years with model validation as a named responsibility, not a side task
Direct experience with ground-truth construction and sampling design
Strong Python and SQL; comfort working independently from the teams whose models you evaluate
US Citizenship Required
Active US Secret clearance
Elevated personnel security requirements apply to portions of this work and are discussed during screening
Preferred
Regulated-industry or government model-risk background
Experience with fairness / disparate-effect testing and human-oversight documentation
NIST AI RMF or comparable practice
Active Top Secret clearance
What Success Looks Like
Evaluation evidence a customer can rely on, independent of the teams that build the models
Releases gated against a clear version and evaluation record
Workflow metrics (review time, throughput, backlog, override/rework rates) that give the program an honest read on whether it's working
