SOFTWARE ENGINEER - QA with AI focus (HYBRID - MONTEVIDEO)
ABIERTO The Software Engineer - QA, focused on AI testing, plays a critical role in the development and deployment of AI-powered solutions, ensuring that our artificial intelligence systems are
accurate, reliable, and aligned with business objectives and client expectations. As Software Engineer - QA specializing in AI testing, you will focus on validating AI models, data pipelines, and intelligent features to ensure they perform as intended, are free of defects, and deliver meaningful value. You will foster a collaborative and transparent environment where team members can share insights, raise concerns, and provide actionable feedback to continuously improve AI systems. Your testing expertise and feedback directly influence the quality, safety, and success of our AI-driven projects.
What you will do:
-
Focus on ensuring that AI systems, especially those driven by prompts, like LLMs — behave correctly, safely, and consistently.
-
Test prompt templates, used in applications (chatbots, agents, RAG systems, etc.) to ensure they generate accurate, consistent, and safe outputs based on functional requirements.
-
Validate prompt performance against defined acceptance criteria (e.g., correctness, coherence, relevance, tone).
-
Detect prompt regressions — when model updates or small prompt changes cause unexpected output shifts.
-
Create test suites with varied input cases to simulate real user queries.
-
Input–Output validation: Does the model handle malformed, ambiguous, or adversarial inputs gracefully.
-
Edge-case testing: How does the system behave under rare or extreme conditions.
-
Hallucination detection: Measuring factual correctness or hallucination frequency.
-
Security testing with different personas.
-
Maintain a prompt test suite that grows with product complexity.
-
Develop and apply metrics for evaluating LLM responses: Correctness (factual accuracy), Relevance, Helpfulness, Readability, Toxicity / Safety.
-
Suggest improvements based on observed failure patterns.
-
Document vulnerabilities for retraining.
-
Provide input and collaborates closely with AI software engineers, QA Engineers and Product Owner on functional and design specifications.
-
Regression Testing: Re-test known vulnerabilities after fixes have been implemented by the development team to ensure they are fully resolved.
-
Advocate for multiple personas and angles of quality for the business in each module including, but not limited to: Dealer Admins, Dealer Users, Dealer Consumers, Internal Supportability, Internal Implementation, and Internal Training, and Sales.
-
Participate in requirements gathering and write/assist in defining user story in sprint squad.
-
Facilitate scrum meetings and communicate changes (as needed).
-
Assist with providing training and guidance, both to QA, and other departments within the company.
-
Validate problems with products reported by customers and developers.
-
Troubleshoot Error and Read Logs; Enter, review, and update software defects in our defect tracking database in relation to automation.
-
Track differences in behavior across model versions and validates that system outputs remain stable and aligned with business requirements.
-
Build hybrid evaluation pipelines, combining automated scoring with human review workflows.
-
Test RAG systems (vector search, embeddings, chunking) for correctness, relevance, and hallucination reduction
-
2 to 5 years of related experience in Quality assurance and at least 1 year of experience in prompt QA engineering.
-
Good knowledge in UI and API automation using API Platform tools like Postman, Robot Framework, Cypress.
-
Knowledge of CI/CD pipelines and version control systems like Git.
-
Experience with databases like Postgres and MySql.
-
Must have a proven understanding of QA practices with regards to UI/API/Database testing.
-
Proven ability to think "outside the box" and approach problems from unconventional angles.
-
Familiarity with and working knowledge of the software development lifecycle.
Preferred:
-
2+ years of experience in Quality Assurance (QA) testing, especially for software or AI products.
-
Previous experience with Automation.
Excellent opportunity to join a company with international standards, work on complex projects, and grow professionally within an expanding team in Uruguay.
Hybrid work model with two days per week in our Carrasco office.