Atharva Naik

Intro

I am Atharva Naik, a second-year PhD student (and former MLT) advised by Prof. Carolyn Rose at the Language Technologies Institute (LTI), Carnegie Mellon University (CMU). I also collaborate closely with Prof. Daniel Fried and Prof. David Mortensen. My research focuses on synthetic data generation and post-training/self-improvement methods for enhancing LLM-generated code quality, reasoning, and code safety/alignment.

I recently completed MetaLint, a project on training LLMs to flag non-idiomatic Python and Java code while adapting to evolving best practices by framing the task as instruction following and leveraging preference tuning. Part of this work was conducted during my summer 2025 internship at Oracle Labs. I also developed PBEBench, a dynamic, contamination-free, and scalable synthetic benchmark for evaluating knowledge-free inductive reasoning through multi-step string rewrites.

Previously, I worked on CRScore (NAACL 2025, Oral), a reference-free metric for evaluating code review comments using LLMs and static analysis by generating code quality rubrics and scoring them with embedding models across key dimensions. I also conducted an investigative study (ACL 2025, Oral, top 10%) framing sound law induction as programming by example and analyzing which synthetic data distributions yield the best downstream performance on low-resource tasks.

I also led the CMU team to the finals of the Amazon Nova AI Challenge in 2025, training LLMs to generate secure code by combining GRPO with deliberative alignment (chain-of-thought reasoning) and static analyzers such as AWS CodeGuru, Bandit, and CodeQL to produce useful, safe, and CWE-free code. I’ve also explored using LLMs to generate reflection questions in collaborative SQL programming activities to support peer learning. Our first study was accepted at AIED 2024 and received Best Paper and Best Student Paper nominations, and our follow-up study was accepted in the British Journal of Educational Technology (BJET), a high-impact educational technology journal.

Full CV

Education

Experience

2025

MetaLint: Generalizable Idiomatic Code Quality Analysis through Instruction-Following and Easy-to-Hard Generalization Atharva Naik, Lawanya Baghel, Dhakshin Govindarajan, Darsh Agrawal, Daniel Fried, Carolyn Rose Arxiv

PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics Atharva Naik, Prakam*, Darsh Agrawal*, Yash Mathur, Manav Kapadnis, Yuwei An, Clayton Marr, Carolyn Rose, David Mortensen Arxiv

Emergent (Mis)Alignment: Exploring the Hidden Link Between Code Security and AI Alignment Atharva Naik*, Abhinav Rao*, Alex Xie, Anmol Agarwal, Shubham Gandhi, Michael Hilton, Carolyn Rose Blogpost (WIP)

Programming by Examples Meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Induction Atharva Naik, Darsh Agrawal, Hong Sng, Clayton Marr, Kexun Zhang, Nathaniel R Robinson, Kalvin Chang, Rebecca Byrnes, Aravind Mysore, Carolyn Rose, David R Mortensen ACL 2025 (to appear)

CRScore: Grounding the Evaluation of Code Review Comments in Code Claims and Smells Atharva Naik, Marcus Alenius, Daniel Fried, Carolyn Rosé NAACL 2025

2024

Providing tailored reflection instructions in collaborative learning using large language models Atharva Naik, Jessica Ruhan Yin, Anusha Kamath, Qianou Ma, Sherry Tongshuang Wu, R. Charles Murray, Christopher Bogart, Majd Sakr, Carolyn P. Rosé British Journal of Educational Technology (BJET)

Can Large Language Models Code Like a Linguist?: A Case Study in Low Resource Sound Law Induction Atharva Naik, Kexun Zhang, Nathaniel Robinson, Aravind Mysore, Clayton Marr, Hong Sng, Rebecca Byrnes, Anna Cai, Kalvin Chang, David Mortensen Arxiv

On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generation Atharva Naik Arxiv

Generating Situated Reflection Triggers about Alternative Solution Paths: A Case Study of Generative AI for Computer-Supported Collaborative Learning Atharva Naik, Jessica Ruhan Yin, Anusha Kamath, Qianou Ma, Sherry Tongshuang Wu, Charles Murray, Christopher Bogart, Majd Sakr, Carolyn P. Rose 🏆 Best Paper Nominee 🏆 Best Student Paper Nominee AIED (2024)

Tricking LLMs into Disobedience: Understanding, Analyzing, and Preventing Jailbreaks Abhinav Rao, Atharva Naik* Sachin Vashistha*, Somak Aditya, Monojit Choudhury LREC-COLLING (2024): 16802-16830 (* - indicates equal contribution)

SkillCLIP: Skill Aware Modality Fusion Visual Question Answering (Student Abstract) Atharva Naik, Yash Parag Butala, Navaneethan Vaikunthan, Raghav Kapoor AAAI Student Abstracts (2024): 23592-23593 (* - indicates equal contribution)

2023

Data Augmentation for Code Translation with Comparable Corpora and Multiple References Yiqing Xie, Atharva Naik, Daniel Fried, Carolyn Rose EMNLP (2023) Findings

SYNC: A Structurally guided Hard Negative Curriculum for Generalizable Neural Code Search Atharva Naik, Soumitra Das, Jyothi Vedurada, Somak Aditya AACL (2023)

Weakly Supervised Explainable Phrasal Reasoning with Neural Fuzzy Logic Zijun Wu, Zi Xuan Zhang, Atharva Naik, Zhijian Mei, Mauajama Firdaus, Lili Mou ICLR (2023)

2022

Super-NaturalInstructions: Generalization via declarative instructions on 1600+ nlp tasks Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, ... , Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, Daniel Khashabi EMNLP (2022): 5085–5109

Representation Learning for Conversational Data using Discourse Mutual Information Maximization Bishal Santra, Sumegh Roychowdhury, Aishik Mandal, Vasu Gurram, Atharva Naik, Manish Gupta, Pawan Goyal NAACL (2022): 1718–1734

Towards Enabling Synchronous Digital Creative Collaboration: Codifying Conflicts in Co-Coloring Suryateja BV, Jeet Patel, Atharva Naik, Yash Parag Butala, Sristy Sharma, Niyati Chhaya CHI Extended Abstracts (2022/4/27): 1-7

2021

Understanding the role of affect dimensions in detecting emotions from tweets: a multi-task approach Rajdeep Mukherjee, Atharva Naik, Sriyash Poddar, Soham Dasgupta, Niloy Ganguly ACM SIGIR (2021): 2303-2307

Email :

arnaik [at] andrew [dot] cmu [dot] edu