State-backed RBS sold the securities to two credit unions, which failed after the US housing bubble burst in 2008.
The settlement is with the National Credit Union Administration Board (NCUA), which regulates credit unions.
RBS, which under the deal does not admit fault, said payment was covered "substantially by existing provisions".
Among outstanding civil and criminal actions against RBS are claims from the US Department of Justice (DoJ) and the Federal Housing Finance Agency.
RBS chief executive Ross McEwan said at a conference in London on Tuesday that the bank was working to resolve outstanding claims, but that this could create "additional conduct provisions and noise" over the next few months.
The bank's existing provisions totalled $3.8bn at the end of June.
RBS shares have been hit recently after the DoJ slapped a $14bn penalty on Deutsche Bank for mis-selling mortgage-backed securities, sparking fears that the demand would cripple the German firm.
The NCUA has so far collected $4.3bn in settlements of various civil lawsuits over mis-sold mortgage securities. The proceeds are used to pay claims against five failed credit unions.
NCUA chairman Rick Metsger said the regulator was pleased with the RBS settlement and plans to continue "to pursue recoveries against financial firms that we maintain contributed to the corporate crisis".