Appeals of Material Supervisory Determinations: Guidelines & Decisions
SARC- 2005-03 (November 10, 2005)
I. Background
This appeal arises from a dispute over the development and validation of a
credit scoring model used in underwriting consumer loan applications by
individuals with little or no credit history. On December 21, 2004, the
Regional Director of the *** Regional Office (the “Regional Director”) of
FDIC’s Division of Supervision and Consumer Protection (“DSC”) issued a
material supervisory determination reached following a Compliance
Examination on January 21, 2003 of *** (“the Bank”). The Regional Director determined
that the Bank’s policies and practices discriminated against applicants on
the basis of age, violating the Equal Credit Opportunity Act (“ECOA”) 15
U.S.C. §§ 1691 et seq. and Sections 202.4(a) and 202.6(b)(2) of ECOA’s
implementing provision, Regulation B. On February 18, 2005, the Bank timely
sought review, and on March 18, 2005, DSC’s Director in Washington (the
“Director”) affirmed the Regional Director’s decision. The Bank timely filed
an appeal with the Supervision Appeals Review Committee (the “Committee”) by
letter dated June 30, 2005.
The Bank argues that (1) FDIC has interpreted the provisions of ECOA and Regulation B addressing the use of credit scoring models in an overly restrictive manner not required by the law; (2) the Bank ceased using the model when it had sufficient evidence to find its predictive qualities deficient; (3) no credit applicant was harmed by use of the model; and (4) as no credit applicant was “discouraged” by any pattern or practice of the Bank, the matter should not be referred to the U.S. Department of Justice (the “Department of Justice,” “DOJ”).
DSC responds that (1) its interpretation of Regulation B is in line with that of the Federal Reserve Board, which promulgated the Regulation, and that the Bank’s model was not statistically validated as required; (2) the Bank continued to use the model for at least three years after actual results of its use should have demonstrated its insufficient predictive ability to meet the validation requirements of the Regulation; (3) if applicants for credit were treated differently on the basis of age without a valid statistical basis justifying the difference in treatment, the Bank has violated the Regulation, regardless of the outcome of the underwriting decision; and (4) as the model overtly and impermissibly treated credit applicants differently on the basis of age without appropriate validation and adjustment to maintain predictive ability, FDIC must refer the matter to the Department of Justice.
In accordance with the Guidelines for Appeals of Material Supervisory Determinations1, the Committee reviews the appeal for consistency with the policies, practices and mission of FDIC, as well as the reasonableness of, support for, and respective positions of the parties. The Committee has considered the written submissions of the Bank. The scope of the Committee’s review is limited to the facts and circumstances extant at the time of the examination. The Committee found, as asserted by the Bank, that the evidence fails to demonstrate that the NORT credit scoring model was statistically unsound as developed. However, the Committee also found that the Bank did not properly revalidate and adjust the model on a continuing basis as required by Regulation B. This finding compels a referral to the Department of Justice as required by the Equal Credit Opportunity Act, 15 U.S.C. § 1691e(g).
A. Development of the Model.
In 1999, the Bank contracted with Company X (“X”), an acknowledged expert in
predictive modeling, scoring and analysis, to develop a set of “scorecards”
for use in underwriting consumer loan applications. For applicants with
established credit histories, five scorecards were fashioned, a separate one
for each of five products – (1) home equity loans, (2) home equity lines of
credit, (3) automobile loans, (4) other secured personal loans, and (5)
unsecured personal loans.
The Bank also sought to create a scoring system that would enable it to extend credit to applicants who otherwise would be denied credit due to a lack of credit history. A single scorecard was formulated to underwrite consumer loan applications by individuals who either had no credit bureau record or had no trade lines on their credit bureau reports. This “no record/no trade line” (“NORT”) scorecard was the only model of the six developed that considered age as a variable directly scored. For the vast majority of NORT applicants, only two variables other than age were scored – length of time at present residence, and length of time at present employment. The fact that a single scorecard was created for applicants with no credit history meant that the NORT model was developed using repayment data aggregated from five different loan products. Loan type was not a variable in the NORT model, and the Bank did not track loan product data for NORT applicants.
According to X, the NORT model was developed using widely accepted statistical principles and modeling techniques and was constructed in a manner similar to other NORT models that it has developed, some of which explicitly take age into account. X asserts that it commonly develops a single scorecard that applies to multiple loan products. The Bank adopted the model only after X evaluated its statistical validity. At the outset, the results reported a Kolmogorov-Smirnoff statistic – a commonly used measure of scorecard performance – of 29.9, a score that is, though within acceptable bounds, considered marginal.
B. Continuing Validation of the Model.
The Bank conducted semiannual validations of the NORT credit scoring model
until October 2003. The Bank’s continuing validation reports indicate that
the NORT model was statistically weak in two key respects: (1) the actual
applicant pool (customers who “walked in the door”) varied from the
development applicant pool (the profile of customers used to develop the
model), and (2) the model did not adequately separate good from bad loans.
The Bank never adjusted the model, but finally abandoned its use in February
2004.
II. Analysis.
A. ECOA and Regulation B.
The Equal Credit Opportunity Act was enacted in 1974 to prevent
discrimination in the extension of consumer credit. The Act makes it
“unlawful for any creditor to discriminate against any applicant, with
respect to any aspect of a credit transaction on the basis of . . . age.” 15
U.S.C. § 1691(a). Regulation B proscribes discrimination against an
applicant on a prohibited basis regarding any aspect of a credit
transaction. 12 C.F.R. § 202.4(a). Although age is a prohibited basis2,
Regulation B permits age to be used as a predictive variable
[i]n an empirically derived, demonstrably and statistically sound, credit scoring system . . .
12 C.F.R. § 202.6(b)(2)(ii).
To qualify as “empirically derived, demonstrably and statistically sound,” Regulation B mandates, among other requirements, that the system be
(iii) Developed and validated using accepted statistical principles and methodology; and
(iv) Periodically revalidated by the use of appropriate statistical principles and methodology and adjusted as necessary to maintain predictive ability.
12 C.F.R. § 202.2(p)(1)(iii), (iv).
B. The Development Phase of the Model.
DSC contends that the population characteristics of applicants who apply for
one credit product tend to vary considerably from the characteristics of
those who apply for a different product. Similarly, performance tends to
vary appreciably from one product to another. For example, applicants
perform significantly better on home equity loans than on unsecured consumer
loans. DSC reasons that because the NORT model combines applicants for five
diverse credit products and does not track performance by product, it is
impossible to tell, using accepted statistical methodology, whether the
scorecard is valid for any of the five products considered individually.
The Bank, citing analysis performed by X, counters that there is not always a predictive benefit to having a sub-model developed for each product:
The segmentation analysis for the NORT model showed that superior predictive lift was obtained from having a sub-model specifically for the NORT population regardless of product.
Letter from X to the Bank, dated August 13, 2004.
The evidence presented fails to demonstrate to the Committee that it was known that the model was not statistically sound as developed. Thus, the Committee declines to find a violation with respect to the development phase of the NORT credit scoring model.
C. The Validation Phase of the Model.
The Bank had ample information prior to October 2002 to conclude that the
NORT scorecard had insufficient predictive ability to meet Regulation B
requirements. The Bank’s first validation report in June 2000 revealed a
significant population shift from the “development” population to the actual
loan applicants. This shift was due largely to a difference in age
distribution. Further, the Bank staff viewed the model as commanding low
predictive power, as pointed out in its October 2002 validation review:
In only the [NORT] model is the PSI3 higher than the acceptable threshold of (0.150). Unlike the [NORT] model in Dealer Finance, the [NORT] model in the Direct line of business scores slightly lower than expected. Since each line of business considers the [NORT] model to be an important tool in the auto-decline process, the statistical validation of the model is not of primary importance. Again, the predictive power of this model is extremely low considering the lack of available credit history on the customers (sic) credit bureau file.
Later validation testing demonstrated that the statistics remained extremely poor and that the model was not performing well. The Bank admitted in its own review that the validation statistics could not separate good from bad loans at a statistically significant level, nor did the model meet the Bank’s validation benchmark. In addition to demonstrating poor statistical performance, the validation reports contained statements by the Bank staff that the Bank was aware of the model’s poor performance, but nonetheless continued to use it. That is, beginning in October 2000 and continuing through May 2003, the validation reports note that the “predictive power of [the NORT] model is intuitively low,” but state that “business reasons” still justify using the model for the Bank’s “auto-decline” process.
In sum, the Bank’s validation reports confirm that, despite conducting regular examinations of the NORT credit scoring model, the Bank did not “revalidate using appropriate statistical principles and methodology and adjust as necessary to maintain predictive ability.” 12 C.F.R. § 202.2(p)(1)(iv). Despite poor results from 2000 forward, the Bank never adjusted the NORT model and, in fact, continued its use until February 2004.4
For these reasons, we conclude that the Bank has failed to show that the NORT model was periodically revalidated using demonstrably and statistically sound principles, as required by Regulation B.
D. Referral to the Department of Justice.
Since 1991, FDIC and the federal agencies with responsibility for
administrative enforcement of ECOA, have been required to refer violations
of that statute to DOJ, where there is reason to believe that one or more
creditors has engaged in a pattern or practice of denial or discouragement
on a prohibited basis.
Although the Bank appears to have been well-intended in attempting to construct an automated scoring system that would enable it to extend credit to an underserved population, under NORT, younger applicants were treated less favorably than older applicants. The validation statistic could not separate good from bad loans, and the model had low predictive power. The Bank did not adjust the model based upon its consistently weak statistical performance. Accordingly, in our view, the Bank’s model violated ECOA and Regulation B: it directly considered age in a credit scoring model that was not revalidated on a demonstrably and statistically sound basis. Such a finding compels a referral to the Department of Justice.
III. Conclusion.
For the reasons set forth above, the Committee finds, as asserted by the Bank, that the evidence fails to demonstrate that the NORT credit scoring model was statistically unsound as developed, but also finds that the Bank failed to properly revalidate and adjust the model on a continuing basis as required by Regulation B. This decision is considered a final supervisory decision by FDIC.
By direction of the Supervision Appeals Review Committee of FDIC, dated November 10, 2005.
Valerie J. Best
Assistant Executive Secretary
1 The Guidelines are set out at 69 Fed. Reg. 41479 (July 9, 2004) and
in FDIC Financial Letter (“FIL”) 113-2004 (Oct. 13, 2004).
2 12 C.F.R. §202.2(z).
3 PSI is the conventional
statistical method of comparing the actual “in-the-door” applicants to the
data used to develop
the model.
4 ***