RSI can help. We use our expertise in courts, ADR, program evaluation and research to provide the most reliable and relevant information to courts and those who work with them. RSI is the go-to evaluator for courts and ADR programs, with decades of experience of assessing programs throughout the country. Our research has influenced the court ADR field as well, by providing guidance to courts on essential topics and helping individual organizations to answer their specific questions.
While our activities have been focused on court ADR, we have also worked with bar associations, state executive branch departments, non-profits and others to conduct research and provide evaluation services on ADR questions and programs.
Quality evaluation is intrinsic to the way RSI approaches court ADR. While we are passionate about the value of ADR to courts, parties and society more broadly, we are just as passionate about courts doing ADR well. We understand that every court ADR setting is different, and resources for evaluation are limited. That's why our approach to evaluation services is collaborative and tailored. We work with courts and other stakeholders to identify program goals and determine the key indicators to measure. We customize the evaluation plan to fit with the program’s needs and its financial and staff capacity.
Our evaluation system design work and evaluations have been conducted for a number of case types and areas of dispute resolution, both with courts and with other entities. These include:
RSI uses two different approaches to help courts learn whether their programs work: 1) working with courts to develop the tools needed for them to conduct their own evaluations and 2) conducting independent program evaluations for them. We tailor both approaches to individual program needs.
We provide courts with customized tools that allow them to evaluate their own programs. These tools can be combined to create a complete evaluation system or developed separately.
For those who want an entire evaluation system, we provide the following:
We also provide the above as a la carte services for courts that need only particular tools for their evaluations, such as help determining how to collect data in their case management system, or the design of surveys and reports.
Evaluation Capacity Building through Tools
RSI helped a state agency that facilitates the resolution of disputes within municipalities and communities to build its capacity to evaluate its efforts. For this, we developed a self-assessment system for the agency. This included a series of online surveys for dispute resolution and group facilitation participants, email text to send with the link to the surveys, a self-reflection tool for the facilitator and an instruction manual for staff that stepped them through the self-evaluation process.
Data Collection Tools
When the Administrative Office of the Illinois Courts wanted to collect more data from family mediation programs around the state, they asked RSI to assist in developing a participant survey and a quarterly report form for courts to inform the AOIC about the mediations that resulted from court referrals.
RSI has more than 20 years of experience conducting program evaluations for courts that want an independent, external assessment of their ADR programs. When we develop an evaluation plan, we not only call on that experience, but also use what we have learned through our long history of designing and administering ADR programs to help us better understand what to measure for each program, how to measure it and what to do with that information. This has provided courts we serve with a more complete and nuanced picture of how their programs are functioning and what they can do to improve.
Our evaluations have ranged from large, multi-year assessments of all aspects of a program to targeted analysis of particular program processes or outcomes. In addition, we provide analysis and reporting on previously collected data.
Evaluation of Online Dispute Resolution Programs
RSI is conducting evaluations of ODR (online dispute resolution) programs in small claims courts in Hawaii and Texas in partnership with the University of California Davis and The Pew Charitable Trusts. This evaluation examines the impact of ODR on access to justice with an eye toward determining whether certain groups within society tend to experience the benefits or costs associated with ODR more than others. An initial report is expected in mid-2021.
Multi-Site Evaluation of Foreclosure Mediation Programs
RSI conducted an evaluation of eight foreclosure mediation programs in Illinois. We assessed each program individually on a number of performance indicators and all were compared with one another to determine whether program design had an effect on program performance. We then made recommendations for improvement for each program and developed overall program design recommendations for any court developing a foreclosure mediation program. The evaluation of the eight programs built upon an initial evaluation of the first six programs to launch. That evaluation pointed courts to areas for improvement, including lowering barriers for homeowners to enter the program and providing pre-mediation counseling services to help homeowners understand their options and complete their loan modification packets. Courts that changed their processes in response to recommendation significantly improved their performance.
Comprehensive Evaluation of a Child Protection Mediation Program
For the Superior Court of Washington, DC, RSI conducted an in-depth analysis of the Child Protection Mediation Program’s processes and outcomes. Using a combination of focus groups, surveys and interviews, along with program and case data, RSI identified the program’s strengths and weaknesses and provided recommendations for improving the program’s processes. These recommendations touched upon almost every aspect of the program, such as extending the deadline for mediation, streamlining communications prior to mediation, requiring parents and their lawyers to arrive earlier to mediation than other participants, and developing a fill-in-the-blank form for mediators to capture points of agreement and action steps to give to the participants before they leave the mediation.
Analysis of Participant Experience
A court in southern Illinois wanted to know how parents were reacting to its pilot family mediation program. It turned to RSI to develop survey forms and to analyze and report on the participants’ responses. Our report let the court know that parents had favorable opinions of the process.
Program Usage Analysis
A concern in the US District Court for the Northern District of Illinois that the Lanham Act mediation program for intellectual property cases was not being used led to a request that RSI determine what was happening. We designed a survey for intellectual property lawyers to find out whether they used mediation for their cases and how they perceived mediation as a means to resolve intellectual property cases. We then analyzed the responses and reported to the court that mediation was being used more than had been widely believed.
RSI’s evaluation system design often begins with a question: What do you want the ADR program to accomplish? While seemingly simple, articulating an answer to that question forms the foundation for the evaluation. That’s why we work closely with courts and stakeholders to articulate program goals and determine information needs.
We then work with the court to design the process for collecting data that will allow us to assess whether the goals are being met. Our evaluation designs have focused on minimizing cost and staff time through strategic use of technology. We have found that the more we can simplify data collection processes, the more likely data will be collected and used.
Development of Data Collection Instruments
Evaluations are not all the same and can require different methods and instruments to capture the required information. After consulting with the court, we develop the tools, such as surveys, interview protocols, spreadsheets or observation protocols. Once the tools are developed, we ask that the court review them to be sure that we have used appropriate language and have not included data that it isn’t possible to capture.
Data collection starts with a pilot period in which the data collection processes are tested. Once any kinks are worked out, the data collection period begins. During this time, RSI regularly checks the data to be sure that there are no problems, and makes adjustments if need be. The court’s responsibility at this time is to provide the data.
What Happens Once the Data Is Collected
After data collection is complete, we start the analysis. In this analysis, we examine the data through the lens of how interconnected issues may affect our conclusions, such as the manner in which cases reach mediation, the presence of lawyers or time allotted to mediate.
The Product of an RSI Evaluation
Just as at the beginning, RSI tailors the evaluation report to the needs of the court. All our reports include actionable recommendations that courts can use to build on their programs’ successes and to address their programs’ weaknesses.
We know that by and large, people are too busy to read long reports. Therefore, although we do provide full-length reports for our large projects, we also determine the best method to communicate evaluation findings to judges, program staff and stakeholders. These have included short summaries of the evaluation, infographics and oral presentations.
In contrast to program evaluation, which focuses on how a particular program (or programs) is functioning, research looks to answer broader questions about ADR. RSI has conducted independent research to help others answer their questions, such as:
In addition to answering individual questions, we have pulled together research and information on particular topics so that it can all be found in one place. This makes it more accessible and useful for judges, court administrators, neutrals and our research colleagues, allowing our work to ripple through the courts and the ADR field. Examples include:
Dispute system design for courts could be most simplistically described as figuring out how to move a particular group of cases efficiently and fairly from court, through an ADR process and back to court. A key aspect of access to justice is similarly described as providing the most effective legal information, advice or representation to poor and low-income disputants. But what if ADR could change the way that poor and low-income disputants access justice? RSI tackled this question in our study Accessing Justice through Mediation: Pathways for Poor and Low-Income Disputants. We surveyed stakeholders and researched the legal and ADR landscape in Illinois to determine the supports and barriers to using ADR to increase access to justice in the state. Then we designed a blueprint for a system whereby poor and low-income disputants could best achieve resolutions to their legal problems.
One of our most important contributions to the field of court mediation is Mediation Efficacy Studies, our study of dozens of studies that explored whether court mediation programs really save time and money and provide participant satisfaction. This is an assumption that drives many court ADR programs, but is it true?
The dynamics of intimate partner violence (IPV) pose a risk to the safety and self-determination of parties to a mediation. As a field, ADR practitioners have acknowledged that screening for IPV prior to mediation is critical, but do mediators have the resources and support to identify IPV and modify their sessions accordingly? To answer this question, RSI investigated the gap between "best practices" when it comes to IPV and the reality of what mediators are actually doing, and explored the opportunity for technology to address these deficits. By analyzing the current landscape of screening protocols, surveying practitioners and convening a group of experts in the fields of IPV survivor advocacy and mediation, we concluded that mediators, particularly ones who are new to the field or mediate only intermittently, would benefit from a tool that provides a guided screening interview to assist them in deciding whether a mediation should proceed, and if so, which adaptations to the mediation would be beneficial. To have the greatest impact, this tool would need to be available free of cost and require minimal training.
RSI was asked by a state bar association to develop a survey for judges regarding their experiences with and assessments of mediation in their jurisdictions, and then to analyze the results and provide recommendations for future steps to support the use of mediation in the state. The survey yielded surprising insights about the perceived needs for mediation in the state and led to recommendations for addressing mediation needs that the state bar had not anticipated. The state bar expected that a lack of interest or understanding of mediation would be the main barrier to its use. Instead, we found that many judges were interested in more mediation being conducted for self-represented litigants, but said they lacked the resources to provide this service.
Our research on foreclosure mediation at the outset of the foreclosure crisis provided the ADR field with reliable information for designing foreclosure dispute resolution processes across the country. These include a state-by-state examination of the goals and objectives of foreclosure dispute resolution programs, program models used around the United States and methods used for funding existing programs.
RSI works independently and with partners to provide research and evaluation services to a variety of courts, state agencies and non-profits.
Courts and State Agencies
Administrative Office of the Illinois Courts
Illinois Judicial Circuit Courts: 1st, 2nd, 6th, 11th, 16th, 17th, 19th, 20th, 21st and Cook County
Illinois Office of the Attorney General
Ohio Department of Education
Superior Court of the District of Columbia
Supreme Court of Ohio Dispute Resolution Section
U.S. District Court for the Northern District of Illinois
Non-Profits and Universities
American Bar Association Section of Dispute Resolution
Center for Conflict Resolution (Chicago)
Dispute Resolution Institute
National Association for Community Mediation
Northern Illinois University
University of California, Davis
University of Illinois
Family Interpersonal Resilience and Safety Transformation (FIRST) Fund
Illinois Equal Justice Foundation