Great question.  Somewhere between 0 and 98%. We kid.  But then again we don't.  

Let's break this down.  Your goal in checking references on candidates is to make sure you get great hires who are going to stay with your company long term.  The most dangerous place to be in the reference-checking process is to have a system that checks references at an inch-deep level, then rejects no one.

Many companies live this reality.  The reference-check process is an after-thought, something to check off after an offer is made.  

In those circumstances, no one gets declined, kind of like this dialog shared by Kris Dunn at HR Capitalist related to the folly of weak reference checking:

Kris: "Hey gang, I know you dig into references, but candidly, we've never rejected anyone based on information we've uncovered through those reference checks.  So, I think we need to discontinue doing that and either a) use a commodity vendor who can do the basics so we say we have done it, or b) get someone who really specializes in getting us quality info so we get better data."

Team: .  "That's bull, Kris.  Doing those is core to our team and part of our cultural check.  It's who we are."

Kris: "When is the last time you rejected someone based off of your calls?" 

Team: 

Kris: "If you want to keep it, we need to institutionalize our ability and willingness to go after negative information in reference checks.  For example, I'd start with a pretty simple question:  "What type of environment would you never put this person into and why?  What type of manager would you never put this person working with and why?"

Team:

Moral of the story:  Reference checking does nothing to protect your culture and the talent standard at your company if you reject no one as a result of those references checks.

Back to the question - what's the right % of candidates to reject in your hiring process due to bad references?

The answer isn't a percentage.  It's ensuring you have a reference checking process that asks tough questions designed to get real information, then being brave enough to put the reference checking system at a point in the system when there's still a chance to compare and contrast feedback on multiple candidates.

Check references post offer on one candidate, and the system is incented to reject no one.  You've checked the legal box but are inclined to move forward with all candidates who have been made an offer.

Insert a reference checking process that asks real questions earlier in the process - say to all finalists (average group size - 2-3) - and you'll find yourself actually using information that comes in to make hiring decisions.

Isn't that what we tell ourselves we use references for?  Make the change and get the rewards - better talent, higher retention, etc.

Or keep your reference rejection percentage at 0% - and wonder what you're missing.  Good luck out there...