Software Engineering Research

Latest papers

• AutoCompact: Learning When to Compact Context in Long-Horizon Coding Agents

  Xuan Zhang, Longtao Zheng, Cunxiao Du, Bo An, Xin Dong · 2 October 2026

  Coding agents solve repository-level software engineering tasks through long trajectories of code inspection, search, editing, and testing. As a task progresses, earlier exploration becomes stale, so managing context is more than avoiding overflow: an agent must decide when to compact, what working …

• Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models

  Juan S. Santillana · 2 October 2026

  Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics. A 661.6M parameter model (approx. 65% code/technical text; no d…

• When Does a Second Model Help? Cross-Model Review in LLM Verification

  Tae-Eun Song · 2 October 2026

  Large language models now generate code, documentation, and analyses, and are increasingly used to review such output. We ask when a second review by a different model helps. Building on the author's earlier preprints, which varied context, repetition, and role structure within one model, we test mo…

• Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

  Junyu Guo, Shangding Gu, Ming Jin, Javad Lavaei · 2 October 2026

  Coding agents can return plausible patches that omit required behavior. These failures are hard to review because long traces and confident summaries often hide what was missed. We ask when a nominally weaker reviewer can reliably decide whether a patch solves its issue. We study 411 execution-label…

• Cross-Benchmark Transfer from RL on Agentic Coding Tasks

  Sushant Mehta, Logan Ritchie, Edwin Chen · 2 October 2026

  Coding agents often fail in the last mile: they build most of a feature but drop a requirement, test only the cases their implementation already handles, break behavior that was supposed to stay intact, or validate against an unchecked assumption. We ask whether reinforcement learning (RL) on expert…

• Self-Evolving Coding Rules for AI Coding Agents

  Zhengyuan Jiang, Reachal Wang, Yuepeng Hu, Yupu Wang, Yuqi Jia, Neil Zhenqiang Gong · 2 October 2026

  The performance of AI coding agents is highly dependent on their underlying coding rules. However, existing coding rules are typically hand-crafted and fixed, making the process labor-intensive and often suboptimal. In this work, we propose RuleEvolve, a self-evolving framework for coding rules. Rul…

• Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software

  Bhanu Prakash Vangala, Tanu Malik · 2 October 2026

  Code generation has emerged as a central capability of large language models, with coding agents now able to produce functionally correct software projects from natural language prompts. However, functional correctness alone does not capture a critical dimension of generation quality: environment sp…

• Rules to Tools: Executable Checks for LLM Agents in Scientific Computing

  Jingjie Ning, Guojiang Zhao, Chen Xu, Shanshan Zhong, Xiaochuan Li, Ji Zeng, Guolin Ke · 2 October 2026

  Scientific coding agents receive equations, boundary conditions, and output requirements in writing, then must assess the programs they revise. Rules to Tools (R2T) supplies prepared executable checks of public scientific requirements. Matched SciCode repair groups share written checks, starting pro…

• LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems

  Yun Peng, Zihan Wu, Zeyang Zhuang, Xin Zhou, Rui Shu, Xu Han, Chun Yong Chong, Yuan Wang, Jiakun Liu · 1 October 2026

  Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks with large reference implementations. Many benchmarks evaluate coding agents' implementation capability to produce correct code edits from detailed speci…

• Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agents

  Jiangrui Zhao, Chenglong Li, Meng Zhang, Xiaoting Du · 1 October 2026

  Coding agents solve repository-level tasks through sequences of actions, where a single erroneous action can misdirect subsequent decisions and increase recovery costs. Existing approaches use execution feedback for recovery or specialized checks to block errors, but deciding before execution whethe…

• DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation?

  Frances Liu, Manny Silva, Paige Calvert, Ayu Adiati, Sarah Sanders · 1 October 2026

  We introduce DoGBENCH (Documentation Generation Benchmark), to our knowledge, the first benchmark for generating and maintaining real user-facing software documentation. It asks whether an agent can produce documentation that experienced technical writers would accept in review. The benchmark contai…

• Self-Spec Verifiable Code Generation

  Jiaru Qian, Yihong Dong, Yongmin Li, Hao Zhu, Bin Gu, Ge Li · 1 October 2026

  Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable guarantees. Recently, researchers have proposed several benchmarks to evaluate the capabilities of LLMs in generating formally verifiable code, where L…

• EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses

  Zhixuan Tan, Pengjie Gu, Zhao Li, Yihan Hu, Xu He, Dong Li, Jianye Hao · 1 October 2026

  While large language models have achieved remarkable success in isolated code generation, authentic software engineering requires sustained reasoning, complex state management, and continuous cross-domain abstraction. However, current evaluations of skill evolution in autonomous agents suffer from a…

• Trustworthy Runtime Error Healing in Real-World Repositories: A Benchmark and Guardrail

  Gou Tan, Pengfei Chen, Zhensu Sun, Jieke Shi, Junkai Chen, Ting Zhang, Weifeng Sun, Junda He, Shuai Liang, Chuanfu Zhang, Lwin Khin Shar, David Lo · 1 October 2026

  Runtime error healing lets a crashed program continue by generating code that repairs its live runtime state. Recent work shows that LLMs can generate such healing code, but it is evaluated only on small competition programs, and executing LLM-generated code inside a live process raises safety conce…

• E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

  Hantian Ding, Chloe Bi, Jiacheng Zhu, John Yang, Matt Deitke, Pengcheng Yin, Zijian Wang, Rui Hou · 1 October 2026

  Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behavior…

• Zero2Repo: Can Coding Agents Build Repositories from Scratch?

  Pei Yang, Tianyu Shi, Yuhang Yao, Wanyi Chen, Tongyun Yang, Dun Pei, Haonan Wang, Pengbin Feng, Guanxu Yu, Jingchun Huang, Zeyu Zhang, Shuhan Sun, Hao Li, Xiang Li, Jie Xiao, Xinyu Wang, Hanxin Chen, Daqi Li, Qi Jia, Hongshan Lin, Zhizhou Gu, Zijun Tian, Weizhi Du, Lynn Ai, Eric Yang · 1 October 2026

  Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limited to a single language and depend on manually curated tasks. We introduce Zero2Repo, a benchmark in which an agent receives a product requirements doc…

• How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

  Kirill Brilliantov, Alejandro Hern\'andez-Cano, Emmanuel Abb\'e · 1 October 2026

  Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machin…

• Papers Without Code: Availability of GitHub Repositories Linked in *CL Publications

  Selina Meyer, Michael Roth · 30 September 2026

  Source code and data published at computational linguistics (*CL) venues are increasingly being shared via GitHub. While this generally is a favourable development for the accessibility and potential reusability of research artifacts in natural language processing (NLP), the long-term availability o…

• Opera: A Verbal Critic Framework for Long-horizon Coding Agents

  Kai Mei, Zhiyuan Hu, Yutong Dai, Juntao Tan, Yifan Zhang, Dingjie Song, Dimitris N. Metaxas, Silvio Savarese, Ran Xu, Zeyuan Chen · 30 September 2026

  Long-horizon coding agents need timely corrections, yet feedback can be ineffective or even harmful when it misjudges ongoing work or fails to address the underlying problem. Existing critics focus on evaluating trajectories and generating feedback, but rarely track what happens after feedback is de…

• Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL

  Jinhao Dong, Liang Zhao, Zihao Yue, Wenhan Ma, Linghao Zhang, Lei Li, Shicheng Li, Yifan Song, Bowen Ye, Fuli Luo · 30 September 2026

  Reinforcement learning (RL) for code agents often uses executable tests to provide binary rewards. With these rewards, Group Relative Policy Optimization (GRPO) assigns identical advantages to test-passing trajectories within each rollout group, overlooking differences in implementation quality and …

• Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection

  Sabrina Kaniewski, Tim Kr\"amer, Julius B\"achle, Markus Enzweiler, Michael Menth, Tobias Heer · 30 September 2026

  Retrieval-Augmented Generation (RAG) is increasingly used to enhance Large Language Model (LLM)-based software vulnerability detection by grounding predictions in retrieved vulnerability knowledge, such as vulnerability reports. However, existing RAG-based software vulnerability detection (RAG4SVD) …

• Complexity-Aware Evaluation of LLM Comprehension

  Ali Mohammadi Esfahani, Nafiseh Kahani, Samuel A. Ajila · 30 September 2026

  Large language models (LLMs) are increasingly used for software engineering tasks that require understanding existing source code, including behavior prediction, function explanation, debugging, and code review. However, aggregate benchmark accuracy can conceal how model reliability changes as sourc…

• CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?

  Yue Pan, Jiawei Li, Ziyuan Zhang, Xiangxin Zhao, He Ye · 30 September 2026

  Large language models can generate plausible code-review comments, but such comments may contain technically incorrect claims that mislead developers. We study technical trustworthiness judgment: determining whether a review comment's core technical claims are correct and applicable to the reviewed …

• From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agents

  Tianyu Liu, Dingyuan Dai, Yufan Du, Zhen Yang · 30 September 2026

  Can coding agents restore software that no longer runs while preserving its underlying methods, and reconstruct industrial software engines from open specifications? Here we introduce ReviveBench, a benchmark with two task families evaluated by hidden verifiers calibrated against native execution en…

• Evaluating Name-Only Directory Routing for One-Shot Code Search

  Manoj Bajaj · 30 September 2026

  Finding the right files is an early challenge for coding agents. We test whether a language model can follow directory and file names to find annotated code files missed by fixed lexical queries. Across 82 audited issues from 11 repositories at pinned pre-fix commits, name-only directory routing rec…