Neel Rajani

Email: Neel.Rajani@ed.ac.uk

Website: https://neelectric.github.io/

Research keywords: LLM post-training, AI Safety, Alignment, Interpretability

Bio:

Neel is a PhD student working on AI safety. He has a background in Computing Science, earning his First Class Honours degree from the University of Glasgow in 2024. His Bachelor’s thesis explored how to augment LLMs for biomedicine with retrieval of relevant medical papers, winning the prize for the best project in his year. In industry, he has learned how to apply LLMs to practical contexts, such as at Kodex AI where he fine-tuned Llama 3.1 for financial tasks, or at Binaere Bauten GmbH, where he built an internal coding assistant for the company’s projects. Furthermore, he was a visiting researcher at the Mila institute in Montreal, where he worked on understanding the shallowness of safety guardrails in LLMs. His research investigates how post-training algorithms affect models, and what they learn about safety during their alignment.

PhD research:

As the capabilities of LLMs continue to advance, our ability to understand their inner workings and control their actions is not keeping pace. Critically, the field has converged towards ‘aligning’ LLMs to human values in post-training, to provide them with guardrails and to ensure their outputs are safe. However, a growing body of work has shown these safety guardrails to be brittle: they can be bypassed via jailbreaks, fine-tuning on completely benign data, or even just sampling with the same prompt sufficiently often. My research aims to understand how alignment is typically operationalised in practice, why safety training is so shallow, and what kinds of risks this can introduce.

Supervisors: Ivan Titov, Pasquale Minervini