Faculty at top CS departments consider recruiting talented PhD students one of the most important and difficult parts of their job. Almost everyone I have spoken to who has served on an admissions committee expresses some level of frustration with the process.
The Process
When I served on admissions committee, I reviewed around 200 applications over three weeks during winter break. Each packet got 3-10 minutes. I looked at the applicant’s school and GPA, their publication list: title, venue, authorship. Then I skimmed the recommendation letters for broadly glowing language and keywords like “independent” or “one of my best students in the last n years.” Then I skimmed the SOP for decent grammar and some faint signal of research taste. I gave scores on rubrics, wrote tiny comments, and clicked to the next packet.
50-75% of applicants got that review from one to three readers and nothing more. Some of the rest received informal 20-minute interviews and were gauged for faculty advising interest.
A PhD admission is a six-year $500k commitment, made mostly based on an application packet of a CV, cover letter and reference letters. Most established companies require more process for a summer internship.
The Evidence on Paper
In most CS subfields, the primary evidence of research ability is recommenders attesting that the applicant is good at research. Publications provide more evidence, especially in subfields like ML where they are common. Though even then, reviewers can mostly only evaluate venue prestige.
We want to measure an applicant’s research potential. We can primarily answer this by: (i) does their recommender say they show great research potential, and (ii) do I trust their recommender to claim this?
A friend, before joining a committee, thought elite programs were simply admitting their friends’ students and perhaps he could do better. Once on committee, he found himself doing the same, not out of favoritism, but because he could not figure out anything meaningful about anyone else from the packet. Many/most admissions committee members are similarly idealistic, and often overwhelmed with the volume of applications to finish reading over Christmas.
For applicants from top-10 or top-20 departments who start research early, the process still works reasonably well. Their recommenders are known, they sometimes have publications in respected venues, and reviewers can calibrate. This is maybe 10–20% of the applicant pool.
For applicants from state schools, or international students (especially from smaller countries): they may have done genuine research with sincere letters attesting to it. Those letters are uninterpretable to reviewers under time pressure. The very best from these programs get in when some additional signal happens to exist (e.g., they already made a major research breakthrough). When it doesn’t, the application is indistinguishable from noise.
Thoughts on Better Evidence
The strongest predictor of job performance is watching someone do the job. Academia has a weak structural equivalent to industry internships in REUs, but expanding such programs is hard.
Many schools these days have some interview component these days. However, during my own admission process, a couple of professors went further: conducting technical research interviews, giving me papers to critique and asking me to come up with potential projects we might work on together.
E.g. Flavio Calmon sent me a prepared handout in advance outlining his interview structure. We talked about my prior research experience, then I presented my solutions to some technical problems he had provided, then we critiqued and brainstormed problems in his research area based on papers he had asked me to read. We both learned a lot about what it would be like to work with each other. It was awesome, I loved it.
If I am faculty one day, I would like to do something very similar. I wish this were more common.