Methodology

Qualitative insight into the industry's top vendors through 18 key performance indicators

Black Book collects ballot results on 18 performance areas of operational excellence to rank vendors by software, systems, products, equipment and outsourced service lines. Client users submit ballots per vendor per function which are juxtaposed with client size, functions outsourced, software installed, systems employed, services rendered, industry/vertical(s) affected, country of service origin and overall service and/or product line domain.

The gathered data is subjected immediately to an internal and external audit to verify completeness and accuracy and to make sure the respondent is valid while ensuring the anonymity of the client company is maintained. During the audit, each data set is reviewed by a Black Book executive and at least two other external reviewers. In this way, Black Book report clients are able to clearly see how a vendor is truly performing. The 18 criteria on operational excellence are subdivided by the client's industry, market size, geography and software category/function outsourced and reported accordingly.

Understanding the statistical confidence of Black Book data

Statistical confidence for each performance rating is based upon the number of organizations scoring the outsourcing service. Black Book identifies data confidence by one of several means:

  • Top-10-ranked vendors and advisors must have a minimum of five unique clients represented. Broad categories require a minimum of 20 unique client ballots. Data that are asterisked (*) represent a sample size below required limits and are intended to be used for tracking purposes only, not ranking purposes. Performance data for an asterisked vendor's services can vary widely until a larger sample size is achieved. The margin of error can be very large and the reader is responsible for considering the possible current and future variation (margin of error) in the Black Book performance score reported.
  • Vendors, associates, consultants and advisors with over 20 unique client votes are eligible for top 10 rankings and are assured to have highest confidence and lowest variation. Confidence increases as more organizations report on their outsourcing vendor. Data reported in this form are shown with a 95% confidence level (within a margin of 0.25, 0.20 or 0.15, respectively).
  • Raw numbers include the quantity of completed surveys and the number of unique organizations contributing the data for the survey pool of interest.

2017 BLACK BOOKTM KEY PERFORMANCE INDICATORS

HEALTHCARE INFORMATION SYSTEMS TECHNOLOGY COMPANIES & SERVICES FIRMS

Strategic Alignment of Vendor Offerings to Healthcare System Client's Mission & Goals
Innovation & Optimization
Training
Client relationships and cultural fit
Trust, Accountability, Ethics and Transparency
Breadth of offerings, varied client settings, delivery excellence across all user types
3Deployment and EHR implementation
Customization
Integration and interfaces, interoperability and connectivity
Scalability, client adaptability, flexible pricing
Vendor staff expertise, compensation and employee performance
Reliability
Brand image and marketing communications
Marginal value adds
Vendors financial viability and managerial stability
Data security and backup services
Support and customer care
Best of breed technology and process improvement developments