All Openings

LLMOps/Evals Engineer

LLMOps/Evals Engineer

Bay Area, CA

|

Full-Time

|

Salary + Equity

About CodeRabbit

Current practices of ensuring quality in software workflows need an overhaul. They're often slow, inefficient, and costly, creating a major bottleneck. As developers, we have firsthand experience with these challenges and complexities. Our mission is to build an AI-first central code quality platform from the ground-up. That vision starts with augmenting manual code reviews using AI and eventually all other aspects of the quality workflows.

CodeRabbit is currently the most installed AI application on both GitHub and GitLab. It is used daily by thousands of developers and teams of all sizes, with over a thousand paying customers. We are backed by top-tier VCs and founders who have deep experience in developer workflows. If you are someone who cares about developer tooling and is interested in joining our team, we would love to hear from you.

Job description

We are looking for an experienced LLMOps/Evals Engineer for evaluating the application performance.

Challenges

  • Auto-evaluation of the output using LLMs.

  • Constructing the right datasets for evaluating the application output. 

  • Continuous tracking of the live application and spotting issues in real time.

What we’re looking for

Requirements
  • Experience constructing datasets from production data or existing sources to be used evals and few shot prompting. 

  • Experience in evaluation using LLMs to score application output. 

  • Setup regression testing to measure performance of the evaluation criteria by changes to the application.

  • Continuous tracking of the live application and spotting issues in real time.  

  • Experience with RAG embeddings and rerankers to improve the output quality.

  • Problem-solvers who can apply innovative ideas to solve the unknown challenges.

  • Ownership: You want to build something you are proud of. Monitor what you ship, learn, iterate, and make it better.

  • Velocity: Deliver high-quality software quickly through decisive action and effective execution.

Good-to-haves
  • 3+ years as an AI/ML Engineer

CodeRabbit􏰁コードレビュー􏰀方法を変革する革新的なAI駆動型プラットフォームです。そ􏰀自動化されたコードレビューによって従来􏰀手作業による時間と労力を大幅に削減しなが ら、コード品質を向上させます。

CodeRabbit􏰁コード􏰀変更に対する洞察に満ちたフィードバックを提供し、コード􏰀効率や堅牢 性を高める改善案を提供します。

ニュースレターへ登録

CodeRabbit © 2023。

CodeRabbit􏰁コードレビュー􏰀方法を変革する革新的なAI駆動型プラットフォームです。そ􏰀自動化されたコードレビューによって従来􏰀手作業による時間と労力を大幅に削減しなが ら、コード品質を向上させます。

CodeRabbit􏰁コード􏰀変更に対する洞察に満ちたフィードバックを提供し、コード􏰀効率や堅牢 性を高める改善案を提供します。

ニュースレターへ登録

CodeRabbit © 2023。

CodeRabbit􏰁コードレビュー􏰀方法を変革する革新的なAI駆動型プラットフォームです。そ􏰀自動化されたコードレビューによって従来􏰀手作業による時間と労力を大幅に削減しなが ら、コード品質を向上させます。

CodeRabbit􏰁コード􏰀変更に対する洞察に満ちたフィードバックを提供し、コード􏰀効率や堅牢 性を高める改善案を提供します。

ニュースレターへ登録

CodeRabbit © 2023。