Shedding Light on the United States Pretrial Services Agency’s Pretrial Risk Assessment Tool

Download the tables accompanying this article:
Bureau of Justice Statistics Compendium of Federal Justice Statistics and More

A federal defendant’s release or detention pending trial is governed by 18 U.S.C. § 3142. It provides that in determining whether to release or detain a defendant, a judicial officer must consider the nature and circumstances of the offense charged; the weight of the evidence against the defendant; the defendant’s history and characteristics, including specifically their character, condition, family ties, employment, financial resources, length of residence, community ties, past conduct, substance abuse, and criminal history; their past performance on conditions of release; whether they were under supervision at the time of the current offense; and the nature and seriousness of the danger that they pose. The judicial officer must detain a defendant if, after holding a hearing, they find that no combination of conditions will reasonably assure the defendant’s appearance or the safety of the community.

In the Speedy Trial Act of 1974, Congress created experimental pretrial services agencies in 10 judicial districts to investigate and advise judicial officers regarding bail, and to supervise defendants released on bail. In the Pretrial Services Act of 1982, Congress authorized the expansion of such agencies to every district. See 18 U.S.C. § 3152. Section 3154 specifies Pretrial Services’ powers and functions. They include to

• Collect, verify, and report to the judicial officer information relating to pretrial release, including information about any danger a defendant may pose;
• Make recommendations about whether a defendant should be released or detained, and if released, on what conditions;
• Supervise defendants released on conditions;
• Inform the court and US Attorney of all apparent violations of conditions of release, arrests, and any danger a defendant may come to pose;
• Prepare pretrial detention reports; and
• Prepare reports to improve the bail process.

In 2009, the Department of Justice’s Office of Federal Detention Trustee (OFDT) recommended that the US Pretrial Services Agency and Pretrial Services Department develop an actuarial risk assessment tool to inform recommendations regarding pretrial release. The recommendation was motivated by the increasing number of federal defendants who were being detained pretrial in contravention of the presumption of innocence, by the cost of detaining them, and by the “logistical nightmares” that their detention caused. Luminosity Data Driven Justice Solutions conducted the original research. The OFDT and the Administrative Office of US Courts sponsored the research in order to identify predictors of pretrial outcomes, identify defendants suited to pretrial release without jeopardizing the integrity of the criminal justice system or the safety of the community, and support alternatives to detention. Those alternatives to detention were third-party custody, substance abuse treatment, location monitoring, housing and shelter, mental health treatment, and sex offender treatment.

The understanding of the development and validation of the PTRA reflected in this article is based on the following sources: Christopher Lowenkamp & Jay Whetzel, The Development of an Actuarial Risk Instrument for U.S. Pretrial Services, 73 Fed. Prob., no. 2, Sept. 2009; Marie VanNostrand, Gena Keebler & Luminosity, Pretrial Risk Assessment in the Federal Court, 73 Fed. Prob., no. 2, Sept. 2009; James Johnson et al., The Construction and Validation of the Federal Post Conviction Risk Assessment (PCRA), 75 Fed. Prob., no. 2, Sept. 2011; Timothy Cadigan & Christopher Lowenkamp, Implementing Risk Assessment in the Federal Pretrial Services System, 75 Fed. Prob., no. 2; Thomas Cohen, Christopher Lowenkamp & William Hicks, Revalidating the Federal Pretrial Risk Assessment Instrument (PTRA): A Research Summary, 82 Fed. Prob., no. 2, Sept. 2018. VanNostrand and Keebler are referred to as the “developers” of the PTRA. The others are referred to as the “proponents” of the PTRA.

The developers constructed the tool by studying 85,369 defendants who entered the federal criminal justice system between 2001 and 2007, who were released on conditions, and about whom there were sufficient data. The developers identified measures or independent variables that they thought would predict the dependent variables or outcomes of interest, namely failures to appear and revocations due to an arrest for new criminal conduct. The independent variables of interest for their predictive value that the developers initially identified were a defendant’s demographics, offense details, criminal history, substance use, mental health, residential status, education, and employment.

The developers reviewed the case files of the sample population of defendants. They coded those files for the presence of the independent variables, and for the defendants’ performance on supervision or the dependent variables. They used multivariate regression analysis to identify those independent variables that had policy-relevant and empirically related value, and they weighted the independent variables based on the magnitude of their relationship to the dependent variables/outcomes. The independent variables they found had predictive value were felony convictions, prior failures to appear, whether or not there were pending charges at the time of the current offense, the type of the current offense (those with firearms, drug, or immigration charges were more likely to fail), the level of the current offense (those with felony charges were more likely to fail), the age at time of interview (younger defendants were more likely to fail), education, employment, home ownership, and substance use.

The developers used this information to formulate a battery of questions, each of which generates a certain number of points. For example, the question about number of felony convictions generates 0 points if the defendant has no prior felony convictions, 1 point if they have one to four prior felony convictions, and 2 points if they have five or more. The question about highest level of education attained generates 0 points for a college degree, 1 point for a high school diploma or the equivalent, and 2 points for something less than a high school diploma. The question about employment status generates 0 points if the defendant is employed or disabled and receiving benefits, and one point if they are unemployed, a student, a homemaker, or retired.

The result is a score on a 15-point scale that is divided into five risk levels. The higher the point total, the greater the risk level. Defendants who score a total of 0–4 points are at risk level 1, 5–6 points are a level 2, 7–8 points are a level 3, 9–10 points are a level 4, and 11–15 points are a level 5. Among the developers’ findings and recommendations were that

• Current practices were not ideal in that Pretrial Services recommended detention too often;
• Low-risk defendants tended to fail because of oversupervision;
• Alternatives to detention were most appropriate for moderate-risk defendants;
• Moderate- to high-risk defendants were most suited to release for programmatic and economic reasons, and their failures on supervision could be minimized by participation in alternatives to detention;
• Release on conditions was consistent with the 1984 Bail Reform Act and strikes the appropriate balance between the rights of defendants and the needs of the community and courts;
• Alternatives to detention were more effective, less costly, more fair, and more true to the presumption of innocence and the Eighth Amendment prohibition of excessive bail; and
• Best practice would be for PTS to use a standardized, empirically based, risk assessment tool to inform its recommendations.

The PTRA is part of a larger agenda that includes the Post Conviction Risk Assessment (PCRA) tool in service of a goal of reducing future criminal behavior by identifying high-risk, high-need offenders and tailoring treatment to their criminogenic needs. PTS first deployed the PTRA in September 2009 in two pilot districts: Nebraska and the Western District of North Carolina. By August 2011, PTS had deployed it in most federal judicial districts.

In 2011, PTS assessed the PTRA’s performance. Among its conclusions were the following:

• Technical conditions of release do not promote appearance or reduce new offenses.
• Community ties are not predictive of risk.
• Violent offenders perform better than others in terms of re-arrest, failure to appear, and technical violations leading to revocation.
• Pretrial detention has negative consequences for defendants at subsequent phases of the criminal justice system such as disposition and sentence because detention reduces their access to their attorney, reduces their participation in their defense, exerts pressure on them to plead guilty, and creates a negative impression in the minds of judges and juries.
• Risk assessment tools are useful but have limitations. In particular, they are not totally accurate at the individual level and must be applied with discretion.

In 2014, PTS began the Detention Reduction Outreach Program (DROP), a program to reduce unnecessary detention by educating stakeholders about the appropriate use and interpretation of the PTRA. Even so, federal prosecutors, defense attorneys, and judges may not appreciate the PTRA and its significance. In 2018, PTS revalidated the PTRA. Among its conclusions were that the PTRA effectively predicts pretrial violations and PTS can lower the pretrial detention rate while producing positive outcomes.

In 2019, PTS reported the failure rates of defendants released on conditions by risk level and type of failure:

Compared to the same statistics in 2010, the rate of failure to appear was identical across all five risk levels. The 2019 rate of new criminal arrests (3 percent) was higher for level 1 defendants (it was 1 percent in 2010), as was the rate of technical violations for levels 1 and 2 defendants (they were 1 percent and 4 percent, respectively, for 2010 defendants).

The developers of the PTRA appear to have had an agenda apart from accurately and reliably predicting defendants’ performance on conditions of release. That agenda may have affected the tool’s accuracy and reliability. For example, the agenda may have influenced the way the developers coded the sample population. The problem is that no outsider has had access to those data and coding decisions. At various points, the proponents of the PTRA have described that agenda as

• An assumption that too many defendants were being detained;
• A goal to save money by reducing the number of defendants detained;
• A belief that detaining defendants is contrary to the presumption of innocence and the prohibition on excessive bail; and
• A goal of shortening sentences by giving defendants an opportunity to prove themselves by complying with conditions of pretrial release.

As regards these items, it is not clear that money is saved by releasing defendants pretrial. If a defendant gets the same sentence whether released or detained pretrial and gets credit for the time they served in pretrial detention, then pretrial release is more expensive because in addition to the costs of incarceration, society incurs the cost of pretrial supervision. Courts have consistently upheld the constitutionality of the pretrial detention provisions of the Bail Reform Act. As regards the goal of shortening sentences, the proponents base this argument in favor of increased pretrial release on the observation that defendants released pretrial tend to receive shorter sentences, but that may be because those defendants are less risky in the first place.

There appears to have been little independent study of the PTRA’s development or implementation. What studies exist seem to have been conducted by the developers, PTS, or persons associated with them. Indeed, the data that Luminosity used to develop the tool is not public and has not been subject to critical review. What is known is that the original sample was one of convenience. The items available for consideration were those that could be distilled from the 2001–2007 PSRs. It might have been better if the developers had identified items they thought predictive of the outcomes of interest, applied them to a valid sample of defendants, and tracked how well those items predicted the sample’s performance on release. However, that would have taken more time.

In the process of identifying the items of interest and narrowing them down, the developers eliminated some items that seemed promising predictors of performance on release such as offense conduct and gender. They applied the items they selected to the sample’s history of performance on release. It does not appear that any of these decisions have been subjected to independent review.

The studies that have verified the PTRA may have been influenced by confirmation bias. PTS has a significant amount of control and discretion over whether a defendant violates conditions of their release. It decides whether to report technical violations such as positive drug tests. In response to a failure to check in, an absconding, and even a new arrest, PTS can and does broker self-surrenders that avoid a failure to appear or a violation. Notwithstanding 18 U.S.C. § 3148(b)’s provision that a government attorney initiate revocation proceedings, it appears that in some districts, PTS has assumed greater responsibility regarding when a defendant is in violation of conditions of release and whether to file a motion to revoke, sometimes without the knowledge of the government. To the extent that PTS underreports violations, it skews the assessment of the PTRA in favor of validation.

Some aspects of bail laws and practices are being challenged in court and otherwise. For example, suits allege that Texas’s and New Mexico’s bail systems detain defendants based on their indigency in violation of the Equal Protection clause, deprive defendants of their liberty without serving a substantial government interest in violation of substantive due process, and deprive defendants of their liberty without procedural due process. However, those suits do not challenge the validity, accuracy, and reliability of the risk assessment tools in use in those states.

In any event, the PTRA has not been subjected to the adversarial process using the tests for admissibility of expert evidence, such as the tests announced in Daubert and Kumho Tire or articulated in Federal Rule of Evidence 702 governing the admissibility of expert testimony. Although PTS is an arm of the court, not a party before it, and the rules of evidence do not apply to bond hearings, still the PTRA is a matter affecting an important judicial determination. There is a relevant scientific community in which risk assessment is a generally accepted technique. That technique has principles and methods. The PTRA’s techniques, principles, and methods have not been subjected to that community’s critical peer review. While the PTRA can be tested, it has not been tested by an independent examiner to ensure that the principles and methods of risk assessment were properly applied.

In creating the PTRA, the developers do not seem to have distinguished between outcomes besides binary not-fail (abide by conditions of release) and fail (violate conditions). By grouping together different outcomes such as failures to appear and new criminal activity into one outcome, fail, the developers may have increased the apparent predictive value of the tool. Historically, defendants released pending trial commit technical violations of their release most frequently, engage in new criminal conduct next most frequently, and fail to appear least often. For example, according to the BJS 2001 Compendium, in fiscal year 2000, they failed to appear 2.6 percent of the time, engaged in new criminal conduct 3.8 percent of the time, and committed technical violations of their conditions of release 17.3 percent of the time, for a total rate of violation of conditions of release of 23.7 percent. Using one score to predict the risk of three different types of outcome obscures, and may exaggerate, the PTRA’s ability to predict a particular outcome.

The developers created five risk levels but did not explain how they did so or what the levels meant. It appears that they did so by dividing the sample into equal fifths without regard to whether those cutoffs represented significant differences in risk. The 2019 failure rates in the table above show that there aren’t significant differences in the risk of violation between several of the various risk levels. Reporting violation rates stratified by risk categories makes it difficult to compare violation rates to pre-PTRA timeframes because prior to the PTRA, performance on pretrial release was tracked by type of offense.

The PTRA was developed on the basis of information about defendants who were released pending trial. It did not analyze detained defendants because it could not. It is being used to assess the prospects of defendants who are making their initial appearances and facing decisions about detention or release. It is being used to recommend the release of defendants who otherwise would be detained pursuant to a regime that did not include the PTRA. In this sense, the initial study presumed that released defendants were representative of detained defendants. It advocated that detained defendants be treated the same as released defendants. But released defendants are not representative of detained defendants, and the sample that the developers used to develop the PTRA was not a representative or random sample of detained defendants.

All of this begs the question of what level of risk is acceptable, who should determine it, and how should they do so. The federal criminal justice system in general and the federal bail statute in particular contemplate that an impartial judicial officer should make that decision on an individualized basis, according to the factors specified in section 3142. While a valid risk assessment tool may be able to quantify the risk that a particular defendant will violate based on group characteristics, some of those characteristics may be legally problematic, and no risk assessment tool can decide what quantum of risk is tolerable.

Finally, the profile of federal defendants arrested and facing detention or release has changed since the PTRA was developed. There was a 22.5 percent increase in the number of federal defendants from 2001 to 2015, and the table below highlights differences in the total number of defendants and a selection of charged offenses over this period of time.

These are significant changes in the profile of the population of federal defendants facing pretrial detention or release since the PTRA was developed, and it does not appear that the PTRA has been recalibrated to account for them.

For example, between 2001 and 2015, there was a significant increase in the number of child exploitation defendants. In 2001, 831 suspects were arrested for federal, nonviolent sex offenses. In 2015, the number had increased to 3,114 suspects, 3.74 times as many. Those defendants often own a home, have a degree, have stable employment, have no prior criminal history, and have no history of drug or alcohol abuse. They are likely to score low on the PTRA. Yet they pose a significant risk of reoffending, and society has been gaining appreciation for the magnitude of the harm that their type of offense causes its victims. It is true that 18 U.S.C. § 3142(e)(3)(E) creates a rebuttable presumption of flight and dangerousness if a judicial officer finds probable cause to believe a defendant has committed a number of specified child exploitation offenses. However, that does not alleviate the concern that an out-of-date PTRA may not accurately assess the risk posed by current defendants.

The PTRA is information that has been affecting judicial decisions regarding release or detention for years. Especially now that pretrial detention practices are getting increased scrutiny, it is time that it received more critical attention. In particular, its development, application, validity, accuracy, and reliability should be examined by someone other than its developers and proponents. PTS should disclose its use in every case so that the parties can litigate whether it is faithful to section 3142, whether it is consistent with an individualized determination of a defendant’s liberty interest, whether it relies on suspect considerations, and whether it is sufficiently reliable for a judge to consider when deciding whether to release or detain a defendant pending trial.

An article in the Fall 2021 issue of Criminal Justice, “Shedding Light on the United States Pretrial Services Agency’s Pretrial Risk Assessment Tool,” incorrectly referred to Marie VanNostrand and Gena Keebler of Luminosity as “the developers” of the Pretrial Risk Assessment Tool (PTRA). The PTRA was developed by the Administrative Office of the US Courts - Probation and Pretrial Services Office (PPSO), not by VanNostrand or Keebler. Criminal Justice regrets the error.