I am pleased to announce that The Fitzwilliam is beginning a partnership with the Advanced Research and Invention Agency (ARIA). My friend Neil Shevlin and I were selected as one of ten groups sponsored to start what they call an ‘Innovator Circle’. We decided to call ours the Fitzwilliam AI Circle.
If you’re not aware, ARIA is a new and exciting science funding agency in the UK. It has been allocated over £1 billion and is structured in such a way as to insulate it from the political pressures and bureaucratic proceduralism that it is argued – quite credibly, in my opinion – have slowed down scientific progress. For example, ARIA can distribute grants outside academia, peer review, and outside the UK. It is explicitly modelled on the success of DARPA, the legendary research agency responsible for funding such activities as the existence of the internet. One of the characteristic features of the ARPA model is individually empowered programme directors, who in ARIA’s case each control budgets of around £50 million.
It’s too soon to say whether ARIA will be a success, but the vibes are good. Our friends at Works in Progress made this cool video about them.1
To the best of my knowledge, me and Neil messing around will be the first presence by ARIA in the Republic of Ireland. This follows a storied tradition in our friend group of pursuing random curiosities until one of us accidentally does something important.2
ARIA states that their goal with the innovator circles is to “support individuals who want to convene their own informal, self-organised, technical peer groups”. I gather the idea is that intellectual history has been disproportionately shaped by close-knit social circles of exceptional talent. Insofar as one can help such groups coalesce and thrive, it’s one of the most useful things you can do. And the UK government apparently thinks I am among the top ten most qualified people to do this. ¯\_(ツ)_/¯
ARIA provides funding, a scientific network, and capacity to support our subsequent projects.
The Fitzwilliam AI Circle will live alongside the Fitzwilliam Maths Circle and Fitzwilliam Reading Group. The AI Circle won’t be about ethics, philosophy, or safety; those things are important, but this is not the right outlet for them.3
We intend this to be much more than just another reading group.4 Starting this month, we’ll be meeting regularly for two sets of meetups. The first will be a journal club anchored around a core curriculum; we were inspired both by Ilya Sutskever’s list and Elicit’s list of foundational machine learning texts.5 You can read a more detailed (but tentative) reading list on our AI Circle GitHub. The second session of the month will be more hands-on: pair programming, implementing papers, and building things. We will also be able to provide some access to compute and other resources for experiments.
Each month will have a theme. For April, the topic is Foundations. The reading group will be on Saturday, April 25th, from 3pm, and we will be discussing:
Vaswani et al. (2017), Attention is All You Need
Radford et al. (2018), Improving Language Understanding by Generative Pre-Training
Radford et al. (2019), Language Models are Unsupervised Multitask Learners
Bonus: Andrej Karpathy, Deep Dive into LLMs like ChatGPT
April’s technical lab will be on Thursday, April 30th, from 6pm, and we will be implementing GPT-1 from scratch.
For May, the topic is Scaling. The reading group will be on Saturday, May 16th, from 3pm, and we will be discussing:
Kaplan et al. (2020), Scaling Laws for Neural Language Models
Brown et al. (2020), Language Models are Few-Shot Learners
Hoffman et al. (2022), Training Compute-Optimal Large Language Models
Bonus: Richard Sutton, The Bitter Lesson
May’s technical lab will be on Tuesday, May 26th, from 6pm, and we will be reproducing the scaling laws for GPT-2 using Andrej Karpathy’s nanoGPT repository.
These papers are complicated and difficult to read. If they don’t make sense, we recommend working through a textbook; Neil and I are fans of Bishop and Bishop’s Deep Learning: Foundations and Concepts.
Future topics will include ‘reasoning’, ‘learning’, ‘interpretability’, ‘agents’, and ‘creativity’. We’ll also have a string of special guests.
One of the topics Neil and I are obsessed with is how passive reading is as a learning method, and how rarely people apply genuinely effective pedagogical methods. We are planning to run ‘pair studying’ sessions, inspired by this video of Dwarkesh Patel watching how insanely thoughtful Andy Matuschak is about reading a textbook.
Our hope is that our group will converge on a shared research direction. But we’re open to going wherever it leads us. We’re also looking for people from different backgrounds. If you are working on a difficult problem – in business, research, or policy – and you are struggling to solve it with current AI, you could be a good fit.
The main reading group will take place in Dogpatch Labs in central Dublin, with some larger labs and talks happening at the Baseline coworking space and in Trinity College Dublin. Many thanks to these generous venues. Later this year, we’ll also be running some events in Belfast; never fear, British taxpayer!
The Fitzwilliam AI circle now has its own section on The Fitzwilliam homepage. We also have a GitHub here.
Finally, the Fitzwilliam community has grown to be quite large, and WhatsApp has become a bit unwieldy as a communication platform. So we’ll also be creating the Fitzwilliam Slack. This will be a server with channels for the reading group, maths circles, AI circles, and general chatter.
If you are interested in joining, please send an email to sam@thefitzwilliam.com with a brief bio, and we can add you to the mailing list, Slack, and GitHub. The extremely rapid progress in AI is, in many ways, very scary, but it’s also terrifically exciting. I don’t think there’s any other period in human history where I’d rather be a young person with too much time on his hands. I look forward to seeing many of you there!
Sam Enright is editor-in-chief of The Fitzwilliam, and Innovation Policy Lead at Progress Ireland. You can follow him on Twitter here or read his personal blog here.
Eric Gilliam also conducted an oral history interview with Ilan Gur, the first CEO of ARIA.
The only reason this group exists is because of a chance encounter with Pranay Shah at the Progress Conference, when he told me to apply.
Neil and I also recently started a (less organised, more informal) group we’re calling ‘The Fitzwilliam Exercise Society’, for coordinating runs, swims, and gym visits in Dublin, with a focus on weightlifting. No one can stop us in our cultivation of Hibernian Renaissance people.
An Irish special: “Is there anything to be said for another reading group?”
Supposedly, Ilya Sutskever provided a list to the video game designer John Carmack when he was trying to get more into AI, for which “If you read these 30 things, you’ll understand 90% of what matters today”. I have been reading this book about Ilya’s list, which is being released in a serial format. There is a lot of mythology beyond “Ilya’s List”, the exact contents hands-onof which have never been disclosed.
!!!