Senior Software Developer in Test, Foundation - Seattle

Seattle, WA
Permanent
Full-time

1 day ago

At Qualtrics, we create software the world’s best brands use to deliver exceptional frontline experiences, build high-performing teams, and design products people love. But we are more than a platform—we are the creators and stewards of the Experience Management category serving over 18K clients globally. Building a category takes grit, determination, and a disdain for convention—but most of all it requires close-knit, high-functioning teams with an unwavering dedication to serving our customers.When you join one of our teams, you’ll be part of a nimble group that’s empowered to set aggressive goals and move fast to achieve them. Strategic risks are encouraged and complex problems are solved together, by passing the mic and iterating until the best solution comes to light. You won’t have to look to find growth opportunities—ready or not, they’ll find you. From retail to government to healthcare, we’re on a mission to bring humanity, connection, and empathy back to business. Join over 5,000 people across the globe who think that’s work worth doing.Senior Software Developer in Test, Foundation - SeattleWhy We Have This RoleWe are looking for a Senior SDET to lead the development of our agentic evaluation framework within the ML team. As our AI capabilities expand, we need specialized quality engineering expertise to ensure robust testing processes and tooling for our agentic AI features. You will be responsible for designing, implementing, and scaling testing frameworks that ensure Qualtrics delivers responsible, high-quality AI solutions to our customers. This role is critical for standardizing how we test AI agents across the organization and establishing Qualtrics as a leader in responsible AI development.How You'll Find Success

Drive the quality assurance strategy for agentic features, ensuring standardization of testing approaches from a customer-centric perspective
Develop comprehensive evaluation tools and frameworks that enable automated testing of AI agent workflows
Define model launch readiness criteria for agentic features that align with both customer needs and regulatory requirements
Establish model risk management protocols that document AI system capabilities and limitations
Create knowledge transfer processes to disseminate best practices in AI testing across Qualtrics teams
Work in a collaborative environment with ML Engineers, Research Scientists, Software Engineers, and Product Managers

How You'll Grow

Become an expert in the emerging field of AI agent testing and evaluation
Lead the development of innovative testing methodologies for complex AI systems
Mentor other quality engineers on AI-specific testing approaches and tools
Build upon our existing LLM evaluation infrastructure to create next-generation testing capabilities
Expand your technical leadership through cross-functional collaboration and knowledge sharing
Stay at the forefront of AI testing practices by researching and implementing new techniques

Things You'll Do

Design and implement automated evaluation frameworks for testing AI agent behaviors and outputs
Build upon our existing AI test strategy and LLM-as-a-Judge infrastructure to create comprehensive testing solutions
Develop metrics and benchmarks to quantify AI system quality, reliability, and safety
Create reproducible test scenarios that simulate real-world user interactions with AI agents
Collaborate with product teams to integrate quality processes into their AI development workflows
Identify and mitigate potential risks in AI agent behaviors, including hallucination detection and security guardrails
Present testing results and quality metrics to stakeholders across engineering and product teams

What We're Looking For On Your Resume

Bachelor's degree in Computer Science or related field
5+ years of professional coding experience with strong Python skills
2+ years experience building test automation frameworks and tools from scratch
5+ years of relevant experience as part of an automated test team, with 1+ years as a senior contributor
Understanding of Machine Learning concepts (deep learning, categorization, classification, data mining)
Experience with Machine Learning evaluation frameworks
Experience with CI/CD deployment automation (Jenkins, etc.)
Proficiency with testing distributed systems and SOA (Service Oriented Architecture)
Experience applying various software testing techniques (equivalence class partitioning, boundary value testing, pairwise testing)
Strong critical thinking and root cause analysis skills

What You Should Know About This Team

The team works on cutting-edge AI technologies that power many of Qualtrics' product experiences.
Our team is pioneering quality engineering practices for AI systems and building the infrastructure that enables reliable, safe, and effective AI features.
We're actively working on several innovative AI projects, including security guardrails for AI models, LLM-based evaluation systems, and agentic workflows across various Qualtrics products.
The team has a strong culture of collaboration, innovation, and quality, and we're looking for someone who can help us elevate our testing practices for these complex systems.

Our Team’s Favorite Perks and Benefits

Wellness Reimbursement: $300 per quarter for wellness activities including gym memberships, spa massages, workout equipment, meditation apps, and much more.
Experience Bonus: $1800 to be used for an “Experience” of your choosing.
Amazing QGroup Communities: MOSAIQ, Green Team, Qualtrics Pride, Q&Able, Qualtrics Salute, and Women’s Leadership Development, which exist as places for support, allyship, and advocacy.

The Qualtrics Hybrid Work Model: Our hybrid work model is elegantly simple: we all gather in the office three days a week; Mondays and Thursdays, plus one day selected by your organizational leader. These purposeful in-person days in thoughtfully designed offices help us do our best work and harness the power of collaboration and innovation. For the rest of the week, work where you want, owning the integration of work and life.Qualtrics is an equal opportunity employer meaning that all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other protected characteristic.Applicants in the United States of America have rights under Federal Employment Laws: , ,Qualtrics is committed to the inclusion of all qualified individuals. As part of this commitment, Qualtrics will ensure that persons with disabilities are provided with reasonable accommodations. If reasonable accommodation is needed to participate in the job application or interview process, to perform essential job functions, and/or to receive other benefits and privileges of employment, please let your Qualtrics contact/recruiter know.Not finding a role that’s the right fit for now? Qualtrics Insiders is the one-stop shop for all things Qualtrics Life. Sign up for exclusive access to content created with you in mind and get the scoop on what we have going on at Qualtrics - upcoming events, behind the scenes stories from the team, interview tips, hot jobs, and more. No spam - we promise! You'll hear from us two times a month max with fresh, totally tailored info - so be sure to stay connected as you explore your best role and company fit.For full-time positions, this pay range is for base per year; however, base pay offered within this range may vary depending on location, job-related knowledge, education, skills, and experience. A sign-on bonus and restricted stock units may be included in an employment offer. Full-time employees are eligible for medical, dental, vision, life and disability, 401(k) with match, paid time off, a wellness reimbursement, mental health benefits, and an experience bonus. For a detailed look at our benefits, visit .Washington State Annual Pay Transparency Range$149,000—$214,000 USD

Qualtrics

Apply Now