Black Book Methodology

Qualitative insight into the industry’s top vendors through 18 key performance indicators

Black Book collects ballot results on 18 performance areas of operational excellence to rank vendors by software, systems, products, equipment and outsourced service lines. Client users submit ballots per vendor per function which are juxtaposed with client size, functions outsourced, software installed, systems employed, services rendered, industry/vertical(s) affected, country of service origin and overall service and/or product line domain.

The gathered data is subjected immediately to an internal and external audit to verify completeness and accuracy and to make sure the respondent is valid while ensuring the anonymity of the client company is maintained. During the audit, each data set is reviewed by a Black Book executive and at least two other external reviewers. In this way, Black Book report clients are able to clearly see how a vendor is truly performing. The 18 criteria on operational excellence are subdivided by the client’s industry, market size, geography and software category/function outsourced and reported accordingly.

Understanding the statistical confidence of Black Book data

Statistical confidence for each performance rating is based upon the number of organizations scoring the outsourcing service. Black Book identifies data confidence by one of several means:

  • Top-10-ranked vendors and advisors must have a minimum of five unique clients represented. Broad categories require a minimum of 20 unique client ballots. Data that are asterisked (*) represent a sample size below required limits and are intended to be used for tracking purposes only, not ranking purposes. Performance data for an asterisked vendor’s services can vary widely until a larger sample size is achieved. The margin of error can be very large and the reader is responsible for considering the possible current and future variation (margin of error) in the Black Book performance score reported.
  • Vendors, associates, consultants and advisors with over 20 unique client votes are eligible for top 10 rankings and are assured to have highest confidence and lowest variation. Confidence increases as more organizations report on their outsourcing vendor. Data reported in this form are shown with a 95% confidence level (within a margin of 0.25, 0.20 or 0.15, respectively).
  • Raw numbers include the quantity of completed surveys and the number of unique organizations contributing the data for the survey pool of interest.

Black Book’s customary Key Performance Indicators

Note: Some software products are evaluated on modified KPIs including Electronic Health Records, Health Information Exchanges and Health Data Security & Privacy.

Vendor overall preference/specific vertical industry recommendations

Vendor achieves the overall preference for expertise, appreciation and insight within the client’s respective vertical. Organizational structure meets or exceeds the needs of stakeholders and customers in this industry. The client is overwhelmingly likely to recommend the vendor for an engagement for the same software lines or service initiatives.


Customers are also continuing to push the envelope for further enhancements to which the vendor is responsive. Clients also believe that their vendors’ technology is helping them compete more effectively, generate larger revenues and profits and cut their overhead in ways that were difficult or impossible to accomplish before innovative outsourced services were introduced.


Vendor provides significant and meaningful training opportunities for employees and client staff. Vendor strives to develop technology staff, customer service and consultant employees in particular.

Client relationships

Vendor honors customer relationships highly. The relationship with the vendor elevates the institution’s reputation. Improving customer efficiency and effectiveness is a priority of the supplier. Governance of engagement is neither complex for buyer nor does it require vendor management attention regularly. There is no regular transparency or quality issue. Account management liaisons affect relations positively.

Trust, accountability, transperancy and ethics

Trust in enterprise reputation is crucial to the outsourced services client as well as prospects. Client possesses an understanding that their outsourcing organization has the people, processes and resources to effectively deliver the desired business results, based on its industry reputation and past performance.

Breadth of offerings, client types, delivery excellence

Industry recognized horizontal functionality and vertical industry applications to manage bundled managed services. Vendor routinely drives operational performance improvements and results in the areas they affect. Multiple offerings allow vendor to meet the progressive needs of the client. The delivery of services meets or exceeds standards of outcomes.

Deployment and outsourcing implementation

Client maintains expedient deployments on all initiatives. Vendor solutions eliminate excessive buyer supervision over vendor implementations. There is a minimized need for consultant management of deployments due to highly astute vendor staff. Services vendor overcomes implementation obstacles and challenges effectively. Technical, organizational and cultural implementation obstacles are handled professionally and timely. Implementation time exceeds expectations.


Software products and process services are customized to meet the unique needs of specific client purpose, processes and models. Little resistance to changing performance measurements clients’ needs vary.

Integration and interfaces

Managed services vendor supports interfaces so information can be shared between necessary applications. Solutions are easily integrated to existing backend systems. Seamless interfaces to legacy applications are performed as required for optimal functioning. Human integration and interface activities are administered precisely. Cross-industry and corporate services expertise is evidenced by execution and orchestration of multiple systems and processes.

Scalability, client adaptability, flexible pricing

Vendor provides flexible product and services pricing, allowing client to choose and pay for the precise functionality and services needed. Invested in significant infrastructure and have the ability to provide services to enterprise organizations. Technology and business process products and services meet the changing and varied needs of the customer. Vendor adapts to the client needs with appreciation and elasticity.

Compensation and employee performance

Vendor is focused on building and developing a strong employee team of producers. Employees act like owners/leaders. Company is moving towards leveraged pay at all levels. Implications of the “contractor mentality” are minimized. Vendor is using effective tools to tie performance metrics to compensation policy and compensating top leaders. Human resources- related criteria are scored from the client perspective on this indicator.


Vendor meets agreed terms as evidenced by routine, acceptable service level reporting and industry expectations. Depth and breadth of applications/solutions are acceptable in meeting client needs. Online reliability is maximized and outages/downtimes are minimized. Solid software product and outsourcing service capacities are demonstrated consistently. Service levels are consistently met as agreed. Services and support response is maximized by vendor team.

Brand image and marketing communications

The vendor’s sales and marketing processes accurately and appropriately represent service deliverables. Image is consistent with top service rankings. Sales presentations and proposals delivered upon and corporate integrity/honesty in marketing and business development are highly valued. Company image and integrity are values upheld top-down consistently. Vendor maintains high image and reputation so that customers of products, systems and services openly reference their vendor’s representatives as client team members with high level of relevant client communications.

Marginal value adds

Vendor’s cost savings are realized as generally estimated and not over positioned or over/underestimated in ways that effect major client satisfaction or costs. Savings expectations such as decreased service handling and support staff are realized. Buyer’s managed service support costs are reduced significantly. Vendor offers value-adds as a cross- industry outsourcing partner in cost savings management initiatives and creative programs through bundled delivery design. Vendor provides true business transformation opportunities to buyer and opportunities for top line contributions.

Viability and Managerial stability

Vendor’s viability, employee turnover, financial stability and/or cultural mismatches do not threaten relationship. This outsourcing vendor takes extraordinary efforts to avoid relationship problems among service providers and the client as well. Senior management and board exemplify strong leadership principals to steward resources appropriate that impact buyers.

Data security and backup services

In order to provide a secure and constantly dependable service offering for product development entities, a vendor has to provide the highest level of security and data back-up services. In some cases, you may find their service in these two areas superior to the security and back-up system in your own firm/corporation.

Support and customer care

Vendor provides an adequate amount of onsite administration and support to clients. There exists a formal outsourcing relationship and account management program that meets client needs among suppliers for the customer. Vendor evidences successful management strategies or services. Media and clients reference this vendor as a service leader and top producer correctly based on evidenced support programming. Customer services and relationship satisfaction is manifested through significant flagship clients as well as smaller and newest customers similarly. Customer satisfaction is manifested as a vendor priority without compromise.

Best of breed technology and process improvement

Related technology services and product management are considered best in breed. Vendor technology elevates customers via capabilities, equipment, processes, deliverables, professional staff, leadership, quality assurance and innovative initiatives. Technology services are delivered at or above current/former in-house service levels. Quality assurance and process improvement is integrated into all initiatives. Vendor leadership drives continuous services improvements to point of client service.