A Mental Health Professional Weighs in on the Ja Report

The enclosed “Review of the Davis Ja Report” was written by Dee Roth, MA, a Board Member of the Pacific Institute for Health Innovation (PIHI).  Ms. Roth was Chief of Program Evaluation and Research for the Ohio Department of Mental Health for 36 years.  She has been the principal investigator on two National Institute of Mental Health (NIMH) grants, and has extensive mental health experience at the national level, including with NIMH and with the Substance Abuse and Mental Health Services Administration (SAMHSA).

Ms. Roth’s review of the Ja Report reflects PIHI’s commitment to promote clinical quality management approaches to ensure that mental health services provided are effective.

Among other insights, Ms. Roth notes:

  1. The Ja Report’s goal to “determine the level of community expertise available to promote better discharges and linkages” does not appear to have been evaluated.
  2. In well-done, credible evaluations, it is critical to state clearly what the central question being investigated is.  She notes the Ja Report confuses the central question it put forth, because the Ja Report didn’t at least measure services provided in the community (or at LHH) against a set of established quality standards, and didn’t evaluate the effectiveness of services provided.  The Ja Report didn’t measure the quality or quantity of behavioral health care services at LHH or in the community.
  3. Though the Ja Report claimed it had used a “random sample” in reviewing LHH patient charts, Roth notes this is incorrect, because the Study Group was 29.4% black and 44.0% white, but that the chart review sample was 46% black and 28% white, so the chart review group cannot have been regarded as either a random, or a representative, sample of the overall behavioral health patient group.  Instead, she suggests that race could be a strong interactive variable around discharges. [Although Roth didn’t discuss this, one has to consider whether Ja’s overall recommendations may apply to some demographic clusters, but not to the needs of other demographic clusters.]
  4. That though the Ja Report focuses extensively on so-called “grounded theory methodology” (a technique used to analyze qualitative data) Ja used to interpret information gleaned from 41 interviewees, there are indications Ja’s grounded theory methodology wasn’t followed.  [Roth did not comment on how this single problem may negate many of the Ja Report’s findings.]   

    Roth sees a more serious problem being that the “Key Questions for Interviews” on page 54 in Ja’s Report may have severely biased what was heard from interviewees, and how interviewers coded and interpreted the grounded theory data.  The “Key Questions” lead-in asserts at least four times that some question whether LHH staff will tell the truth, honestly, introducing a severe bias that interviewers may have approached their task assuming (or being instructed) to doubt with pre-conceptions the veracity of information provided by LHH staff chosen to be interviewed. [Note:  Many people are deeply concerned by the Ja Report’s possible implication that LHH staff would deliberately “lie” (not be “truthful”) to Ja’s interviewers.]
  5. That while percentages of LHH residents with “severe mental health illnesses” Ja presented were technically correct, evaluators experienced with the policy use of findings would have calculated the percentages differently, providing a clearer, more accurate picture.  A careful read of Roth’s review compared to Ja’s Report shows Ja moved the denominator from 12 to 9, misreporting that five (55.6%) of those with a severe mental illness did not have documentation of psychiatric evaluation or service in their charts, inflating that only 41.7% did not have such documentation.  
    Ja also claimed (by reducing the denominator) that 44.4% of patients with a severe mental health diagnosis had received psychiatric treatment during the study period, when, in fact, only 33.3% needed such treatment.  Roth was too kind by not commenting on why Ja inflated both indicators by double digits, possibly to claim an inflated need to hire additional behavioral staff.
  6. That the system-level group being formed in response to Ja’s recommendations needs to be made a broadly-representative group, and that it focus on Laguna Honda’s currently-mandated responsibilities and mission.  The system-level group that was actually appointed is meeting outside of the public view (its meetings are not open to the public), and hasn’t released any of its deliberations.
  7. That Ja Report’s recommendation that “higher salaried physicians be replaced by registered nursing staff, social workers and psychologists,” came completely out of the blue.  Roth sees no evidence in the data presented in the Ja Report that would support such a premature conclusion made even before policy and operations groups sat down to figure out appropriate treatment needs of patients.  Roth notes this very specific recommendation “about replacing doctors seems to echo politics, not data.”

But Roth doesn’t specifically address a key admission in the Ja Report:  Ja acknowledged that the five patients who had no documentation of psychiatric evaluation were actually being treated for psychiatric conditions by their primary care MDs.  Doesn’t that portend that cutting LHH’s primary care MDs might then inadvertently cut psychiatric care to a significant number of patients, in addition to cutting the medical care that they need?

If Ja did not stop to consider that cutting MDs might have an unintended consequence of cutting psychiatric care at the same time, what kind of nonsense is his recommendation to eliminate MDs at Laguna Honda Hospital, where 100% of its patients need medical care?  How could he have overlooked such a glaring inconsistency?  (Or did Ja just assume nobody would notice the unintended consequence?)



Copyright (c) 2009 by Committee to Save LHH.  All rights reserved.  This work may not be reposted anywhere on the Web, or reprinted in any print media, without express written permission.  E-mail the
Committee to Save LHH.