AI Evaluator - Remote (Mumbai)

About the Job:

As an AI Evaluator (both English and Hindi) you will work to help state-of-the-art generative AI models perform better using your language and critical thinking skills. You will be required to ensure quality in a high-investment area for our client’s program to evaluate LLM (Large Language Model) responses to human queries while checking for accuracy and policy sustainability.

There will be badge tests and an overall training process aimed at verifying the target skills and ensuring that any new candidate has the right aptitude for this work. All activities need to be completed within the defined SLA (service level agreement) and quality. All work will be in both Hindi and English.

This is a fully remote position, and candidates are required to have their own computer with a reliable high-speed internet connection to perform this role effectively.

Key Responsibilities:

• Proficiency in English and Hindi: intermediate or advanced language skills, which the candidate will use to:

• Analyze and understand the meaning of the model response, the user feedback and how it relates to the response.

• Identify messages that are inappropriate and trigger policy concerns.

• Define the target information (claims) in the response.

• Recognize information that isn’t clearly pointed out as wrong - e.g. missing details that make the response incomplete - or information that’s mixed up between multiple entities (disambiguation).

• Formulate appropriate strings for fact-checking and web research that helps them find the right sources quickly.

• Recognize any differences between the text in the input passage and the text generated in the response, thus identifying input issues.

• Understand the context of the user-model interaction, and recognize when the model doesn’t interpret that context correctly.

• Workflow-related: understand complex Guidelines instructions, ask well-formulated questions, leave informative justifications and task comments.

• Understand the client platform and follow team level protocols (like how/when to access the task, etc.).

• Strong logic, intuitive problem-solving and critical thinking; adaptability to the needs of the task. The candidate needs to be open-minded and welcome different viewpoints or solutions.

• Proactiveness, confidence to work independently without constant support. Reliability and being able to make good decisions on their own.

• Be a thought partner to delivery managers, input into the project delivery strategies, and share quality best practices to aid in achieving the project goals

Minimum Qualifications:

• Have completed a Bachelor’s degree.
• Fluency in Hindi at a level equivalent to C2 level.
• Fluency in English at C1 level.
• Digital skills for high levels of research to find accurate information from reliable internet sources.
• Strong critical thinking, reasoning, and exceptional problem-solving skills.
• Excellent attention to detail.
• Experience in GenAI is an added advantage.
• Knowledge of Indian cultural background.

Preferred Qualifications:

• Excellent written and oral communication skills in both English and Hindi.
• Hindi qualifications are preferred at Hindi Honors or MPhil levels.
• Demonstrable ability to perform well in a rapidly changing and extremely global team.
• Passion for our mission of ensuring a world-class support experience for our community and customers.
• Experience as an editor is an added advantage.

About Sama

Sama provides high-quality training data that powers AI technology for Fortune 2000 companies such as Google, Walmart, Ford, Microsoft, and Marriott. We’re experts in data curation and data annotation for 2D and 3D image, video, and sensor data for machine learning algorithms.. Sama offers the highest quality SLAs in the industry, along with cutting-edge ML-assisted annotation tools, QA processes, and security and compliance standards.

Founded in 2008 on the belief that “talent is equally distributed, but opportunity is not”, Sama is driven by the mission to expand opportunities for those who are underprivileged. As a certified B-corp, Sama has provided worker training programs to increase economic opportunity for more than 13,000 people from underserved communities. By connecting our customers with amazing talent in East Africa, we’ve impacted more than 59,000 workers and their dependents.

Today, our vision is to provide data scientists, ML engineers, and data operations teams with an indispensable, integrated platform for AI data preparation, labeling, and collection.

For more information, visit www.sama.com.

More information can be found at:

• Featured in Forbes: How Ethical Is Your AI?
• Sama Honored on Inc. Magazine’s Annual List of America’s Fastest-Growing Private Companies — the Inc. 5000
• Reversing Poverty - Ted Talk by our founder Leila Janah

Our Culture:

Sama is quite unique. We are a technology company with a social mission. People that thrive in a high-growth environment, love working on the bleeding edge of technology, and really care about having a positive impact on the world are a great fit for the Sama culture. Our core values are One Team, One Goal - Deliver. Period. - Trust & Transparency - Customer First - Humanity.

Our Benefits:

Sama is proud to offer competitive compensation corresponding to market data & level of experience. Sama also offers benefits packages based on country of employment, including but not limited to medical, dental, vision, and life insurance, holiday and vacation policies, professional development stipends and a paid sabbatical program. Our Talent Acquisition team would be happy to discuss our benefits packages with you in more detail during the interview process.

At Sama, we pride ourselves in being a diverse and equal opportunity employer.