Research

Below is all currently published research work I'm an author on. Each entry includes a link to the paper, a short summary, and some personal thoughts.

1. Lag and Duration of Leader–Follower Relationships in Mixed Traffic Using Causal Inference

   Summary: This paper implements a causal inference approach to analyze leader-follower dynamics in an arterial road in Chennai, India. We quantify the temporal lag and duration of interactions using transfer entropy metrics.

   My thoughts: This was my first paper. I learned a lot about how to write research in general from here, and in the future I think I would like to do more papers in this style—analyzing some weird real world phenomenon with an interesting method. Published in Chaos.

2. Herding as an emergent behaviour in harem groups of feral Garrano ponies

   Summary: We use transfer entropy to measure herding in a harem group of feral Garrano ponies in Serra D'Arga, Portugal. We characterize leader-follower relationships, validate transfer entropy against traditional clustering methods, and provide evidence that herding emerges through indirect propagation of leadership influence among mares.

   My thoughts: I did it for fun, and it started out of an idle conversation with my supervisor. Published in Journal of The Royal Society Interface.

3. Batayan: A Filipino NLP benchmark for evaluating Large Language Models

   Summary: This paper introduces Batayan, a benchmark to evaluate LLMs on NLP tasks in Filipino.

   My thoughts: Most of the work here was in actually writing/re-translating the entries. Would be nice to do some in-depth error analysis ala Parser Showdown at the Wall Street Corral. Published in ACL 2025 Main Conference.

4. Identifying a Circuit for Verb Conjugation in GPT-2

   Summary: Looking for a circuit in GPT-2 that does subject-verb agreement. We find one, but it gets progressively larger as the SVA task gets more complicated.

   My thoughts: Final project for L193: Explainable Artificial Intelligence. Thinking of a place to submit this.

5. Learning Modular Exponentiation with Transformers

   Summary: We teach a small 4-layer transformer modular exponentiation. PCA on embeddings doesn't show any clear structure, but we do find a cool example of grokking by multiples of moduli. Also, we find a small circuit that performs regular/normal exponentiation.

   My thoughts: Final project for R252: Theory of Deep Learning. Thinking of a place to submit this.

6. Learning Dynamics of Meta-Learning in Small Model Pretraining

   Summary: If you replace half of the steps in language model pretraining with a meta-task, what does the model learn? Model achieves better loss, improves the vanilla model's F1 on NER, and has this really interesting phase transition.

   My thoughts: This is one half of my MPhil thesis. Really proud of Figure 6 here.

7. Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages

   Summary: At what point in pretraining does meta-pretraining start to improve zero-shot cross-lingual named entity recognition (NER) in Filipino and Tagalog? If you fine-tune every checkpoint from pretraining step 0 to 6000, you find some actual reuse of knowledge from the model's backbone.

   My thoughts: This is the other half of my MPhil thesis. Have submitted this to a workshop somewhere. I think Figures 4 to 7 look nice.

8. No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes

   Summary: Can we predict the accuracy of LLM answers using model internals, even before the answer is generated? We find that a simple linear probe on activations can achieve surprisingly good performance.

   My thoughts: Worked on this with MARS 2.0 people. Nice graphs.

9. Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models

   Summary: We study the effects of ReLoRA on the learning dynamics of small language models. Our experiments show that ReLoRA isn't that helpful.

   My thoughts: Yuval's thesis. I like the conclusions.

10. Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time

    Summary: We investigate the phenomenon of inoculation, where appending even a short system prompt in fine-tuning suppresses this behaviour in general deployment.

    My thoughts: Daniel Tan is very agentic.

11. Understanding AI Trajectories: Mapping the Limitations of Current AI Systems

    Summary: A comprehensive overview of the current limitations of AI systems on the way to AGI. Nice taxonomy of factors.

    My thoughts: I was only a minor contributor here, giving detailed review and helping write substantial portions of the continual learning section.

12. Does Self-Evaluation Enable Wireheading in Language Models?

    Summary: We investigate the potential for wireheading in language models through self-evaluation mechanisms. We find that self-evaluation can inadvertently lead to wireheading, and formalize some conditions under which this occurs.

    My thoughts: I wrote this as a way to upskill in agent foundations research, and ended up with something paper shaped.