Research and Funding
Inside NIA: A Blog for Researchers

Percentiling at NIA: sometimes we do. And sometimes we don’t.

Percentiling at NIA: sometimes we do. And sometimes we don’t.

Posted on April 9, 2014 by Robin Barr, Director of the Division of Extramural Activities. See Robin Barr's full profile.

Why do some grant applications receive a percentile rank and others not?

Which NIA applications get a percentile rank?

Applications for an NIA grant that are reviewed at the NIH Center for Scientific Review, or CSR, get a percentile rank.

Why are some NIA applications reviewed at CSR while others are reviewed at the NIA?

Many NIA grant applications are reviewed at the NIH Center for Scientific Review, or CSR. CSR handles receipt and review of 80% of NIH grant applications. CSR staff work hard to match your application to the most expert reviewers and, when they can, assign your application to more than one Institute to consider for funding.

Other NIA applications are reviewed by NIA study sections, including career development awards (K awards), multi-site clinical trials, cooperative agreements, large program project and centers awards, and others. NIA organizes these review groups, which require special coordination or expertise not available through the CSR process. Read more…

This percentile ranking orders these investigator-initiated research grant applications, with those with the best scores at the top of the list for funding. CSR-reviewed applications constitute the bulk of our funding. (For newbies, read about the NIH peer review process here.)

Applications for an NIA grant that are reviewed at the National Institute on Aging, through review panels organized by our NIA scientific review staff, do not receive percentile ranks.

If an application is reviewed by NIA, we give it a priority score, not a percentile. This includes the R01s and investigator-initiated research grant applications that are reviewed at NIA. But this seems inconsistent! Either this means that we at NIA are crazy (which many have claimed) or that the use of percentiling needs explaining. Well, today we’re going try to clear away all the clouds.

Why do we use percentiles for the NIA applications reviewed by CSR?

During each Council review round, we’re combining data from as many as 100 different committees from all over CSR in the same list.

Committees develop different patterns of behavior. Some concentrate their scoring at the good end of the range: all the applications are above average! Others make a conscientious effort to spread their scores throughout the range. And there are a few that hold to the maxim that nothing is perfect and so vote that way.

How is my percentile determined?

Here is a great summary from NIAID on exactly how a percentile is calculated.

Percentiling, then, adjusts for these different patterns. Percentiles are based on three rounds of a particular panel’s behavior. Because the base is calculated from reviews by the same panel then the tendency of a panel to be generous or stingy with scoring is canceled by the percentile.

Percentiles allow the NIA to order applications reviewed by panels that behave very differently from each other and could otherwise have distorted a funding list.

Why don’t we do this for applications reviewed at NIA?

We tried this and it no longer works well.

In earlier days, we did percentile program project applications and other research grant applications reviewed at NIA. We used the trick of percentiling them against the average of all the CSR panels as there were too few applications in our own panels to percentile them against themselves. This method missed some of the advantages of percentiling but it did allow us to integrate NIA-reviewed applications with CSR-reviewed applications in the same list. And we saw that as an important advantage.

Then the scoring system changed.

We went from a score range of 100 to 500 to a range of 10 to 90. That increased the problem of “ties” and brought our percentiling solution crashing down on us.

And the challenges get even bigger when panels start handing out perfect scores of 10 like candy. Normally a score of 10 would be at the first percentile. But when a panel gives a lot of tens, then the percentile for a score of 10 drops to maybe the 5th or 6th percentile and a score of 11 would drop to the 10th or 12th percentile. That means an application with an almost perfect score would drop out of the funding range of many ICs. So CSR panels really have a disincentive to give out many 10s. (And there are other disincentives that deter CSR panels from giving out too many 10s.)

But, NIA panels aren’t in the same situation when it comes to penalties for assigning many 10s. All applications for which they assigned a score of 10 would receive a first percentile rating based on the CSR-averaged-percentile-score chart. And a score of 11 would be at the 2nd percentile and sometimes still at the 1st percentile.

That sounds like an esoteric problem—surely there are more pressing things to worry about? Until it happens. Then it becomes a real headache. Been there. Done that.

We tried other ways to percentile. To cut a long story short, they don’t work either.

Applications reviewed at NIA now get a priority score, not a percentile rank.

So, we chose to use the priority score itself to rank applications. We make a major effort to give different NIA review committees strong guidance on spreading scores to try to achieve consistent scoring across different committees. And we now have separate funding lines for NIA-reviewed and CSR-reviewed applications as we cannot integrate the two sets of applications.

So, the answer to why we percentile some applications and not others comes down to trying to keep an even playing field for science across quite different review environments.

Have I answered all of your questions about percentiling? Get in touch with me by commenting below.

 

Read next:

NIA Funding Policy for Fiscal Year 2014

6 Comments
Share this:
Email Twitter Linkedin Facebook

Posted by Jen on Apr 09, 2014 - 1:03 pm
How do you determine which applications are reviewed via CSR vs NIA?

Posted by Robin Barr on Apr 09, 2014 - 1:21 pm
Most investigator-initiated (parent announcement) research applications (R01, R21, R03, R15) are assigned to CSR for review. Complex research applications (like P01s) are assigned to NIA for review as are career development awards and scientific meeting applications. You can find a fuller description at: www.nia.nih.gov/research/dea-scientific-review/applicants.

Posted by Anonymous on Apr 09, 2014 - 2:59 pm
I have an R03 with an impact score of 8%. A1 was submitted in early 2013 and so far I still have not received any response back. Is this normal? Thanks.

Posted by Robin Barr on Apr 09, 2014 - 5:56 pm
I encourage you to contact your program officer. If you do not receive a response within a few days then send me an email with details about the application, and I will make sure that you receive a response.

Posted by Anonymous on Apr 09, 2014 - 7:20 pm
How long after council review does it usually take for funding notices to be initiated? Our application falls within the published pay-line but at the "top" end.

Posted by Robin Barr on Apr 10, 2014 - 8:53 am
Six to eight weeks is normal but it takes longer when our grants staff have difficulty having messages or calls returned. So please help them to help you!