Pay Back the Taxpayers Act of 2014 - Prohibits the Federal National Mortgage Association (Fannie Mae) and the Federal Home Loan Mortgage Corporation (Freddie Mac) (government sponsored enterprises or GSEs), during the term of any conservatorship or receivership, from making any contribution or transfer to, or allocating or setting aside any amounts for, the Housing Trust Fund or the Capital Magnet Fund.

Requires any amounts paid or repaid to the Secretary of the Treasury by a GSE in any form, during any conservatorship or receivership of the GSE, to be covered into the General Fund of the Treasury and used only for reducing the federal budget deficit.