Nilay Yilmaz

I am a Ph.D. student at Arizona State University, working under the supervision of Professor Yezhou Yang.

My research centers on evaluating the multimodal abstract reasoning capabilities of vision-language models (VLMs) and large language models (LLMs), with a particular emphasis on spatial visualization and visual analogy problems. These cognitively demanding tasks reflect core aspects of human intelligence, and my goal is to advance our understanding of human cognition by developing models that can mirror such abilities in artificial systems.

I have extensive experience in developing dynamic dataset creation pipelines that include 3D scene construction grounded in physical principles and modular framework structures for flexible task generation. I believe that evaluating model capabilities should go beyond static benchmarks and limited multiple-choice evaluation formats, which fail to reflect a model’s true reasoning ability. Instead, evaluation must be open-ended, comprehensive, and human-aligned to uncover not only what a model gets wrong, but why and how that differs from human reasoning patterns.

In addition, I explore parameter-efficient fine-tuning and preference-based reinforcement learning as strategies to enhance the reasoning and problem-solving abilities of multimodal foundation models in complex, abstract domains.

News

Dec 10, 2025	✨ Released `Unwritten Benchmark` - a new challenge designed to probe this abstract perceptual and cognitive ability. Check out our paper and page!
Nov 15, 2025	✨ Released `MentalBlackboard` - an open-ended spatial visualization benchmark. Check out our paper and page!
Jan 15, 2025	🚀 VOILA is accepted at ICLR 2025!

Selected Publications

VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning

Nilay Yilmaz, Maitreya Patel, Yiran Lawrence Luo, and 4 more authors

In ICLR, 2025

Abs Bib PDF Code Data

Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information. Despite their exceptional performance on visual understanding benchmarks, measuring their ability to reason abstractly across multiple images remains a significant challenge. To address this, we introduce VOILA, a large-scale, open-ended, dynamic benchmark designed to evaluate MLLMs’ perceptual understanding and abstract relational reasoning. VOILA employs an analogical mapping approach in the visual domain, requiring models to generate an image that completes an analogy between two given image pairs, reference and application, without relying on predefined choices. Our experiments demonstrate that the analogical reasoning tasks in VOILA present a challenge to MLLMs. Through multi-step analysis, we reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning. Notably, we observe that performance improves when following a multi-step strategy of least-to-most prompting. Comprehensive evaluations on open-source models and GPT-4o show that on text-based answers, the best accuracy for challenging scenarios is 13% (LLaMa 3.2) and even for simpler tasks is only 29% (GPT-4o), while human performance is significantly higher at 70% across both difficulty levels.
@inproceedings{yilmaz2025voila, title = {{VOILA}: Evaluation of {MLLM}s For Perceptual Understanding and Analogical Reasoning}, author = {Yilmaz, Nilay and Patel, Maitreya and Luo, Yiran Lawrence and Gokhale, Tejas and Baral, Chitta and Jayasuriya, Suren and Yang, Yezhou}, booktitle = {ICLR}, year = {2025}, publisher = {OpenReview}, data = {https://huggingface.co/datasets/nlylmz/VOILA}, }