Product Analyst, AI Solutions

Tyler Technologies

  • Yarmouth, ME
  • Permanent
  • Full-time
  • 8 days ago
DescriptionProduct Analyst, AI SolutionsThe Product Analyst, AI Solutions is responsible for shaping, optimizing, and maintaining advanced AI-driven features and user experiences within our product suite. This role sits at the intersection of business analysis, AI tuning, and user experience-combining expertise in software requirements gathering, prompt engineering, and AI model evaluation. The Analyst is heavily focused on refining how AI systems behave in real-world scenarios, ensuring outputs are accurate, trustworthy, and closely aligned with client needs.You partner with AI Engineers, Product Managers, UX/UI Designers, QA, and Subject Matter Experts (SMEs) to extract user requirements, translate those into actionable User Stories and evaluation metrics, and continuously tune AI systems for best-in-class performance and usability.This role is pivotal in ensuring our AI solutions provide maximum business value, aligning advanced technology with end-user needs while driving continuous improvement and reliability. The Analyst produces high-quality deliverables using systematic processes, with impact across multiple projects, deadlines, and teams.Principal DutiesAI Tuning & Output Quality
  • Design, evaluate, and refine prompts, templates, and conversation flows for reliability, cost effectiveness, and user alignment.
  • Maintain prompt libraries, track changes, and manage versioning.
  • Systematically test AI model outputs for accuracy, consistency, compliance, and tone, applying rerankers, guardrails, and validation layers to ensure business rules are met.
  • Collaborate with engineering and QA to promptly integrate feedback and resolve model drift or new scenarios.
Requirements Gathering & Documentation
  • Proactively gather requirements of moderate scope and complexity from diverse sources (clients, market analysis, SMEs) to drive product development and AI enhancements.
  • Document software and system requirements, including user stories, program functions, test cases, and steps required for new features or modifications.
  • Ensure all documentation supports future maintainability and growth of AI-enabled features.
Evaluation, Metrics, & Continuous Improvement
  • Establish systematic evaluation frameworks for AI model outputs, leveraging quantitative and qualitative benchmarks.
  • Run experiments, A/B tests, and structured evaluations to compare tuning strategies and track performance metrics (accuracy, coverage, hallucination rates, latency).
  • Translate business and user feedback into actionable improvements, owning continuous iteration for released solutions.
Collaboration & UX Alignment
  • Act as a bridge between technical and non-technical stakeholders, ensuring clear, effective communication and alignment.
  • Work closely with product, development, and UX/UI teams to ensure AI-driven features fit seamlessly into business workflows, are intuitive and user-friendly, and meet client needs.
  • Help shape the overall user experience for AI interactions, promoting explainability and transparency.
Education, Experience and Special Skills
  • Bachelor's degree in Computer Science, Management Information Science, Business, or related field OR equivalent work experience.
  • 1+ years' experience in software, AI/ML, or related product analysis roles.
  • Experience with AI/ML applications (LLMs, NLP, machine learning pipelines).
  • Strong understanding of prompt engineering and generative AI interaction design.
  • Basic knowledge of data modeling, relational database concepts, and SQL.
  • Familiarity with model evaluation techniques and performance metrics.
  • Proficiency with Python or JavaScript for prototyping and evaluation.
  • Experience with Agile SCRUM development processes.
  • Excellent planning, organization, and interpersonal communication skills.
  • Ability to work independently and collaboratively across functional groups.
  • Experience in ERP systems, enterprise software, or complex business domains.
  • Familiarity with RAG (retrieval-augmented generation), embeddings, MLOps, and responsible AI practices.
  • Experience with fine-tuning methods (LoRA, QLoRA, PEFT), running structured evaluation series, or human-in-the-loop systems.

Tyler Technologies