Structured Prediction

Paper Reviews

Reading research papers

One of the goals of this class is to make you comfortable with reading CS research papers. Reading a research paper is a learned skill. Here are some pointers that may help you on that front.

• Philip WL Fong. 2009. Reading a computer science research paper. ACM SIGCSE Bulletin, 41(2):138–140.
• Srinivasan Keshav. 2007. How to read a paper. ACM SIGCOMM Computer Communication Review, 37(3):83–84.
• Michael Mitzenmacher. 2015. How to read a research paper.

Reviews

Most assignments for this class will take the form of paper reviews. Typically, you will be given a list of papers (taken from the lecture pages), from which you can choose one paper to review. You can submit your review any time before the deadline via Canvas.

I am hoping that the reviews will be like peer reviews for conference or journal papers. We will use a Canvas form for this purpose.

For each paper, you should first start off with a one paragraph description of the core contribution of the paper in your own words, followed by a list of pros and cons of the approach. After this, you should be creative and discuss how you would extend or develop this paper, if you had been the author. End the review with questions, doubts, or any thoughts about the paper. You could also list experiments or results that you think would strengthen the paper.

Some tips

• Do not comment about the grammar and typos. We are reading the paper as computer scientists and are interested in the underlying ideas. Please be constructive and remember that a critical review is meant to critique, not merely criticize.
• Summarize the paper in your own words in a couple of sentences. This will help you understand the paper better.
• Identify the main scientific contribution of the paper
  • A good paper will try to answer a specific question. Try to identify this question. For example, this could be a new idea, a unification of previous ideas into a more general framework, or an implementation of an idea to show that it works.
  • Sometimes the question asked by the paper might be its most important contribution
  • Sometimes the novelty of the approach could be its biggest selling factor
• Identify the problem being solved by the paper
  • Every paper tries to solve a problem. What is it for this one?
  • Was the problem worth solving? Are there simpler solutions that the authors did not consider and might have worked just as well? Does the solution have limitations?
• Identify the positive aspects of the paper
  • What are the good ideas? What is their potential influence?
  • How could you generalize the ideas?
  • If you start research in this direction, what will be your next step?
• Identify the assumptions and claims made by the paper
  • Are they reasonable? Will they ever hold?
  • Given the assumptions, what does the paper claim?
  • Given the assumptions, are the claims of the paper reasonably justified? What is the flow of reasoning?
• If the paper has experiments, identify the value of the experimental evaluation
  • Does the paper use the right set of experiments?
  • Is the experimental setup convincing? What other experiments/results would strengthen the paper?
  • Do the experiments really persuade you about the paper’s claims? Do the results persuade you?
• Always try to think about how this paper would be situated among other relevant papers.
  • Are the ideas novel, or have they appeared before?

Some resources to help you write better reviews

• Mark Allman. 2008. Thoughts on reviewing. ACM SIGCOMM Computer Communication Review, 38(2):47–50.

• Graham Cormode. 2009. How not to review a paper: The tools and techniques of the adversarial reviewer. ACM SIGMOD Record, 37(4):100–104.

• Alan Jay Smith. 1990. The task of the referee. IEEE Computer, 23(4):65–71.