An industry case study, conducted at Amazon Prime Video, on assessing the risk of software deployments using diff-aware features derived from code changes. We show how features extracted from code diffs can be used to predict the likelihood that a given deployment introduces a regression, helping teams prioritize review and testing effort.
@inproceedings{kurup2026deployment,title={Deployment Risk Assessment Using Diff-Aware Features: A Case Study at Prime Video},author={Kurup*, Mayur and Suh*, Hyunjae and Vaidyanathan, Swathi and Vyas, Pranesh and Madabhushi, Srinidhi and Silyutin, Yegor},year={2026},booktitle={IEEE/ACM International Conference on Automated Software Engineering (ASE), Industry Showcase},}
TOSEM
An Empirical Study on Evaluating Accessible Code Generation Capabilities of LLMs
Hyunjae Suh, Mahan Tafreshipour, Sam Malek, and Iftekhar Ahmed
ACM Transactions on Software Engineering and Methodology (TOSEM), 2026
We study how well Large Language Models generate web-accessible code compared to human developers. Comparing code generated by GPT-4o and Qwen2.5-Coder against human-written code, we find that LLMs frequently produce more accessible code for basic features but struggle with complex issues such as ARIA attributes. We evaluate advanced prompting strategies (Zero-Shot, Few-Shot, Self-Criticism) and introduce FeedA11y, a feedback-driven ReAct-based approach that outperforms the other methods, underscoring the need for feedback-based techniques to improve accessibility.
@article{hyunjae2026accessible,title={An Empirical Study on Evaluating Accessible Code Generation Capabilities of LLMs},author={Suh, Hyunjae and Tafreshipour, Mahan and Malek, Sam and Ahmed, Iftekhar},year={2026},journal={ACM Transactions on Software Engineering and Methodology (TOSEM)},doi={10.1145/3820782},}
2025
ICSE
An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far Are We?
Hyunjae Suh, Mahan Tafreshipour, Jiawei Li, Adithya Bhattiprolu, and Iftekhar Ahmed
In IEEE/ACM International Conference on Software Engineering (ICSE) , 2025
As LLMs become popular for code generation, detecting whether source code was written by a human or an AI is increasingly important for quality and licensing reasons. We empirically evaluate existing AI-generated code detectors and find that they all perform poorly and lack generalizability. We propose improved approaches based on fine-tuning LLMs and machine-learning classifiers using static code metrics and AST-derived embeddings. Our best model outperforms the state-of-the-art detector GPTSniffer, achieving an F1 score of 82.55, and we conduct an ablation study on the contribution of different code features.
@inproceedings{hyunjae2025aidetector,title={An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far Are We?},author={Suh, Hyunjae and Tafreshipour, Mahan and Li, Jiawei and Bhattiprolu, Adithya and Ahmed, Iftekhar},year={2025},booktitle={IEEE/ACM International Conference on Software Engineering (ICSE)},}
2024
arXiv
Does the Order of Fine-tuning Matter and Why?
Qihong Chen, Jiawei Li, Hyunjae Suh, Lianghao Jiang, Zheng Zhou, Jingze Chen, Jiri Gesi, and Iftekhar Ahmed
We present the first empirical study analyzing how the ordering of intermediate fine-tuning tasks affects target-task performance in Software Engineering. Unlike prior NLP work that considered only a single intermediate task, we study multi-task orderings and find that ordering influences results by up to 6% performance gain and up to 4% performance loss. We examine dataset-, model-, and task-level factors (syntactic/semantic similarity, dataset size, probing and attention analysis, and task affinity) to explain these effects and to guide cost-effective ordering choices.
@article{chen2024finetuning,title={Does the Order of Fine-tuning Matter and Why?},author={Chen, Qihong and Li, Jiawei and Suh, Hyunjae and Jiang, Lianghao and Zhou, Zheng and Chen, Jingze and Gesi, Jiri and Ahmed, Iftekhar},year={2024},journal={arXiv preprint arXiv:2410.02915},}