Steps in the Product Evaluation Process
The CDC's "Workbook for Designing, Implementing and Evaluating a Sharps Injury Prevention Program" offers instruction in the key steps in the product evaluation process for sharps devices that can also be applied to other supplies:
1. Organize a Product Selection and Evaluation Team
Healthcare organizations should designate a team to guide processes for the selection, evaluation, and implementation of engineered sharps injury prevention devices. Many institutions already have product evaluation committees that may be used for this purpose; others may want to assign this responsibility to a subcommittee of the prevention planning team. To ensure a successful outcome:
■ Assign responsibility for coordinating the process
■ Obtain input from persons with expertise in or perspectives on certain areas (such as from front-line workers)
■ Maintain ties to the prevention planning team
2. Set Priorities for Product Consideration
The team collectively determines which device types to consider.
3. Gather Information on Use of the Conventional Device
Before considering new products for evaluation, healthcare organizations must obtain information on use of the conventional device that it is replacing. Possible sources of information are purchasing and requisition requests. A survey of departments and nursing units might help identify additional issues. Key information to obtain from clinical areas includes:
■ Frequency of use and purchase volume of the conventional devices;
■ Most commonly used sizes;
■ Purpose(s) for which the device is used;
■ Other products the device is used with that might pose compatibility concerns;
■ Unique clinical needs that should be considered; and
■ Clinical expectations for device performance.
If the answers to these questions reveal areas with unique needs, representatives from these areas should be added as ad hoc members of the team.
4. Establish Criteria for Product Selection and Identify Other Issues for Consideration
Product selection is based on two types of criteria:
■ Design criteria that specify the physical attributes of a device, including required features for clinical needs and desired characteristics of the safety feature
■ Performance criteria that specify how well a device functions for its intended patient care and safety purposes
5. Obtain Information on Available Products
Potential sources of information on products with include:
■ Materials management staff who have information on product vendors and manufacturers and are also familiar with the service reliability of manufacturers’ representatives
■ Colleagues in other facilities who can share information on their experiences in evaluating, implementing, or rejecting certain products and devices
■ Websites with lists of manufacturers and products
6. Obtain Samples of Devices Under Consideration
Arrangements should be made to contact manufacturers or vendors to obtain samples of products for consideration. Once obtained, look at the devices based on the design and performance criteria and other issues that are important. Consider inviting manufacturers’ representatives to present information about their products to the team. Questions for the representatives might include:
■ Can the device be supplied in sufficient quantities to support institutional needs?
■ Is it available in all required sizes?
■ What type of training and technical support (e.g., on-site in-service training, teaching materials) will the company provide?
■ Will the company provide free products for a trial evaluation?
Discuss any technical questions related to the product. Based on these discussions, the team should narrow its choices to one or two products for an in-use evaluation.
7. Develop a Product Evaluation Survey Form
The form used to survey healthcare personnel who evaluate the trial device must collect information necessary to make informed decisions for final product selection. Teams should try to use readily available forms. This promotes standardization of the evaluation criteria and enhances the ability to compare responses among different healthcare organizations. If manufacturer-provided forms are used, they should be carefully screened to eliminate potential bias.
8. Develop a Product Evaluation Plan
Developing a product evaluation plan requires several additional steps, but it is necessary to ensure that the form obtains the desired information and documents the process.
■ Select clinical areas for evaluation. The evaluation does not need to be performed institution-wide, but should include representatives from areas with unique needs. Whenever possible, include both new and experienced staff.
■ Determine the duration of the evaluation. There is no formula for how long to pilot test a product, although two to four weeks is often suggested. Factors to consider include the frequency of device use and the learning curve, i.e., the length of time it takes to become comfortable using a product. It is important to balance staff interest in the product and the need for sufficient product experience. If more than one device is evaluated as the replacement for a conventional device, use the same populations and trial duration for each product. Make a defined decision on when to abort an evaluation because of unforeseen problems with a device.
■ Plan for staff training. Healthcare personnel participating in an evaluation must understand how to use the new device properly and what impact, if any, the integration of a safety feature will have on clinical use or technique. Training should be tailored to the audience needs and should include discussion of why the change is being proposed, how the evaluation will proceed, and what is expected of participants. It is important to provide information on the criteria used to evaluate clinical performance and to answer any questions about the interpretation of these criteria.
A team approach, using in-house staff and device manufacturer’s representatives, is one effective way to provide training. In-house staff know how products are used in a facility, including any unique applications, but manufacturer’s representatives understand the design and use of the safety feature. Give trainees an opportunity to handle the device and ask questions about its use, as well as an opportunity to simulate use of the device during patient care, in order to help reinforce proper use. Also consider those who might not be able to attend the training (e.g., staff on leave, new students, per diem staff) and how to implement catch-up training. One possibility is to identify persons in departments or on nursing units to serve as resources on the devices.
■ Determine how products will be distributed for the evaluation. Whenever possible, remove the conventional device from areas where the evaluation will take place and replace it with the device under study (128). This approach eliminates a choice of product alternatives and promotes use of the device undergoing evaluation. If the device undergoing evaluation does not meet all needs (e.g., all sizes are not available; the study device can be used for only one purpose and the conventional device has multiple purposes), it may be necessary to maintain a stock of the conventional product along with the product under study. In such instances, provide and reinforce information on the appropriate and inappropriate use of the conventional device. Precede and coordinate staff training with any switch in devices.
■ Determine when and how end-user feedback will be obtained. Obtain feedback on device performance in two stages. The first stage is informal and occurs shortly after the onset of pilot testing. Members of the evaluation team should visit clinical areas where the device is being piloted and engage in discussions about the device in order to get some preliminary indication of its acceptability for clinical use. These interactions can also reveal problems that might require terminating the evaluation early or providing additional training.
The second stage involves distribution of the product evaluation forms. To avoid recall bias, this should be done as soon as possible after the evaluation period is completed. An active process, such as distributing surveys during unit meetings, may be more reliable than a passive process, where forms are left in the clinical area and filled out at random, and also prevents staff from completing multiple evaluation forms for the same product.
9. Tabulate and Analyze the Evaluation Results
Compile data from the survey forms. Depending on the number of staff involved and survey forms completed, this can be done either by hand or by use of a computerized database. It is useful to score each question in addition to the overall response, particularly if evaluating two or more devices (e.g., hypodermic syringe/needle); responses to each question can be used to compare devices. In addition, categorize individual comments so they provide a better picture of the clinical experience with the device.
Consider calculating response rates by occupation and clinical area and analyzing data by these variables, if the volume of responses permits. This can help identify differences in opinion that may be influenced by variations in clinical needs.
Several factors can have a positive or negative influence on the outcome of a product evaluation. These include:
■ Staff experience with and preference for the conventional device;
■ Attitudes toward involvement in the product evaluation process;
■ Influence of opinion leaders;
■ Staff opinion of product evaluation team members and manufacturers representatives;
■ Perceived need for devices with safety features; and
■ Patient concerns.
It is possible that one or more of these factors may be influencing opinions if the response of certain groups of personnel to the product change is different from what was expected or differs from other groups in the organization. Meet with these groups to understand their issues; it might provide new insights for the evaluation team.
10. Select and Implement the Preferred Product
The evaluation team should make a product selection based on user feedback and other considerations established by the selection team. Model the process for implementing the selected device after the pilot evaluation process, and coordinate training with product replacement. It may be necessary to implement a product change over several weeks, moving by unit within the hospital.
The team should also consider a back-up plan in case the selected device is recalled or production is unable to meet current demands. Questions to ask include:
■ Should a less-preferred product be introduced as a replacement?
■ Should the conventional device be returned to stock?
■ If the conventional device is still being used for other purposes, should the stock be increased to meet current needs?
11. Perform Post-implementation Monitoring
Once a new device is implemented, assess continued satisfaction with the product through follow-up monitoring and respond to those issues not identified or considered during the evaluation period. In addition, some facilities may wish to assess post-implementation compliance with use of the safety feature. Each product selection team will need to consider the most effective and efficient way to perform post-implementation monitoring.