Biomedical Research

Abstract

Objective: To develop an integrated measurement system for competence assessment of nurses.

Methods: Extensive literature review was performed to acquire data on relevant competence frameworks, study methods, and background information. Initial draft was developed as a result of utilizing theoretical analysis approach for data analysis, comparison, and synthesis. Delphi method was adopted to screen and identify core competencies for completing this assessment system. Content validity of the assessment scale was tested by evaluation from a sub-group of Delphi experts, and a questionnaire survey was conducted among nurses to determine its reliability.

Results: This assessment system for core competencies of nurses identified 9 main domains of core competencies (the first-level domains) for nurses, which were divided into a total of 31 constituent subcompetencies (the second-level domains). Items included in the first-level domains and their corresponding weights were as follows: clinical nursing competency (W=0.1271), professional quality (W=0.1250), level of knowledge (W=0.1197), communication competency (W=0.1188), physiopsychological constitution (W=0.1165), management competency (W=0.1027), teaching competency (W=0.0983), scientific research competency (W=0.0972), and development work (W=0.0951). Participating experts rated the importance of the second-level domains by composite index, with the three most important items of sense of responsibility, first-aid skills and specialized nursing skills, while teaching organization, patient family management and research awareness were rated as the lowest importance. Validity examination of the competency assessment scale resulted in the Item-level Content Validity Index (I-CVI) of 0.92-1.00, and the Scale-level Continent Validity Index (S-CVI) of 0.94, with the inter-rate agreement of 0.94. Weighted Kappa coefficient was 0.77 ± 0.14 for test-retest reliability of this assessment scale.

Conclusions: This assessment system for core competencies of nurses provides honest, objective and comprehensive evaluation for ICU nurses’ core competencies.

Keywords

Nurse, Core competency, Assessment scale, Delphi evaluation

Introduction

Core competencies of nurses, also known as core competitiveness of nurses, integrate knowledge, skills and attitudes required for nurses to achieve effective or superior performance and outcomes [1]. Lenburg [2] defines nurses’ core competencies as assessment and intervention skills, communication skills, critical thinking skills, human caring and relationship skills, management skills, leadership skills, teaching skills, knowledge integration skills. Strasser [3] thinks the core competencies for nurses include problem-solving ability (the consultation and health assessment skills), ability to manage common disease conditions, effective communication and counselling skills, ability to apply rational drug use, identifying when and where to refer, ability to access and use information, demonstrating a caring and confident approach, and providing general clinic management. Scribante et al. [4] deem that nurses’ competencies are mainly professional competence, cognitive competence, interpersonal skills, and critical care patterns of interaction. Pascual et al. [5] point out that specialized nursing knowledge and experience are two important competency factors for nurses. Dun et al. [6] group competency standards in specialist critical care nursing into 6 domains: professional practice, reflective practice, enabling, clinical problem solving, teamwork, and leadership. Aari et al. [7] divide the nurses’ clinical competence into nursing care, clinical guidelines, nursing interventions, and identify professional competence with ethical activity; decisionmaking; development work, and collaboration. Leonard et al. [8] has conducted a study on nurses’ cultural competence. Curran et al. [9] believe informatics competencies are vital for nurse practitioners. Ford et al. [10] have established Skills Fair Workshop to assess competencies in critical care nursing. Australian College of Critical Care Nurses (ACCCN) has developed competency standards for specialist level critical care nurses, and Fisher MJ et al. [11] have examined its construct validity. Compared with the abundant research on nurses’ competencies in other countries, we have much less input in China, especially with a gap in developing the competence assessment scale. A valid and accurate competency assessment tool can help to enhance nurses’ holistic competence and facilitate development in intensive care nursing; it can also serve to guide nursing education and training, provide references for staff deployment and post assignment among nurses and assist nurses in developing self-awareness of their professional competency through feedback, and promote their professional development [12,13]. This study aims to define the core competencies standards for nurses through Delphi process, and develop a comprehensive assessment scale appropriate and applicable for nurses.

Materials and Methods

Determination of competency domains

A project team was established, comprised of experts representing nursing management, nursing education, and nursing practice. The team initiated interviews and semi-structural questionnaire surveys, looked up various documentation, including Training Program for Nurses (issued by Ministry of Health of China), Guidelines for Establishment and Management (Trial) (issued by Ministry of Health of China). They also extensively searched and reviewed literatures capturing studies on nurses’ competencies. After thorough data analysis, discussion and modification, competency domains sensitive enough to appropriately define the core competencies of nurses were determined, and initial draft of these competency standards was formulated.

Delphi expert panel

Delphi experts were all from different nursing settings in Grade III, Class A hospitals (the highest level hospitals according to the 3-tier, 9-level hospital system in China) across China. Experts were selected in line with the following principals: 1 involving representative and authoritative specialties; 2 representative of different regions in China; 3 representative of different levels in nursing; 4 covering both teaching and non-teaching hospitals; 5. capturing both general and special hospitals. Inclusion criteria were: 1 engaged in clinical practice, administration, education, or research settings of nursing; 2 possessing associate degree or above; 3 showing initiative for this study and willingness to participate the Delphi procedure; 4 able to ensure response in all of the three Delphi rounds; 5 meeting the requirements for working years in nursing: 10 years or above for those with associate degree, 5 years or above for bachelor’s degree, 3 years or above for master’s degree and 2 years or above for doctor’s degree. There were also 3 criteria to exclude experts who were inappropriate for this study: 1. engaged in non-critical care nursing; 2. academic qualification below associate degree; 3 not willing to participate in the consultation. Questionnaires were sent to the experts via e-mail in each round and those who made valid response became participating experts in the next round. Questionnaire contents of the 3-round Delphi consultation are listed in Table 1.

Table 1: Consultation contents in the three Delphi rounds.

Expert evaluation for content validity of assessment scale

According to the feedback from 3 Delphi consultation rounds, the initial draft of core competency standards was revised, and transformed into the holistic assessment scale for nurses' core competencies, with added background information on participant experts. A group of 12 representative experts from Delphi panel evaluated the content validity of this assessment scale.

Reliability of assessment scale

For the reliability evaluation, 60 nurses, selected from 3 Grade III Class A hospitals in 18 provinces across China by convenience sampling method, went through a test-retest procedure with this scale for their core competence assessment.

Statistical analysis

All data analysis was conducted via SPSS 16.0 (SPSS Inc., Chicago, IL) and EXCEL 2007. Weight and composite index were utilized to reflect the importance of each competency domain; Kendall’s coefficient of concordance (W) was applied for assessing agreement among experts; content validity of this competency assessment scale was evaluated based on the content validity index and inter-rater agreement; weighted Kappa coefficient demonstrated the test-rest reliability.

Results

Delphi panel

The final draft was completed, identifying 9 first-level competency domains and 31 second-level competency domains with related descriptions and criteria. Delphi panel consisted of 34 experts in ICU nursing from 30 Grade III, Class A hospitals located in 18 provinces or cities, including both teaching and non-teaching hospitals, general and specialized hospitals from the east, west, south, north, and middle of China. Responses to questionnaire in each Delphi round are indicated in Table 2.

Table 2: Responses in the three Delphi rounds.

Background of Delphi experts

Experts’ ages, total working years, working years in nursing were 41.6 ± 8.5 years old, 21.9 ± 9.2 years and 13.1 ± 6.3 years respectively. Their academic degrees varied as associate degree (16.1%), bachelor’s degree (67.7%), master’s degree (12.9%), and doctor’s degree (3.2%). Their positions included director of nursing department (16.1%), superintendent of nursing (6.5%), department (or service) director of nursing (16.1%), and nurse manager (61.3%). Table 3 demonstrates different settings of ICU nursing engaged by Delphi experts.

Table 3: Delphi experts’ nursing settings (n=31).

Expert authority coefficient

Experts’ level of expertise, level of familiarity with the consultation contents, and their basis for judgment were evaluated through a calculation and transformation into weighted scores, which were 0.77, 0.88, and 0.84 respectively. The expert authority coefficient, calculated based on the above scores, was 0.83.

Experts’ evaluation on importance of competency domains

Composite index was adopted to reflect the importance rating of the first-level and second-level competency domains by the experts. A higher composite index suggests a higher degree of importance, and vice versa. According to the consultation result, the 3 domains of the highest importance in the first level were clinical nursing competency, professional quality and level of knowledge, while the 3 of lowest importance were teaching competency, scientific research competency and development work. As for the second-level domains, the three most important were sense of responsibility, first-aid skills, and specialized nursing skills, while the three of the lowest importance were teaching organization, patient family management, and research awareness. Kendall’s coefficient of concordance (W) was calculated to assess the agreement among Delphi experts in evaluating the competency domains, as illustrated in Table 4. After 3 Delphi rounds, the core competency domains (first-level and second-level) for ICU nurses were finally defined, as shown in Table 5 with their corresponding weights.

Table 4: Kendall’s coefficient of concordance (W) for experts’ views, and significance test (n=31).

Table 5: Nurses’ core competency domains and their corresponding weights.

Discussion

Expert initiative coefficient indicates how concerned and cooperative experts are during a study [14], which can be reflected by the questionnaire response rate. A response rate of 70% is generally acknowledged to be sufficient for a study [15]. In our study, the Delphi experts showed constant initiative, earnestly reading and completing questionnaire, providing valuable comments and suggestions. Response rates in the 3 Delphi rounds were all above 85%, and respondents providing comments and suggestions in each round were 41.9%, 46.4% and 20.0% of the total returned questionnaires, respectively. Experts’ active participation contributed to enhance the accuracy and credibility of our study.

In Delphi study, expert source plays a role in affecting the integrity, credibility and accuracy of consultation results. Fully aware of the unbalanced development in different regions of China, our project team, in order to avoid bias, tried to capture experts representative of the ICU nursing level in different regions across China. As a result, the Delphi panel was set up with experts from 30 Grade III, Class A hospitals in 18 provinces and cities, covering the eastern, western, southern, northern and middle regions, and representing the highest nursing level in each region.

The Delphi experts were all engaged in different settings of nursing, capturing clinical practice, education, administration and research, which contributed to be vital components in our core competency domains for nurses. Expertise in theses settings can help to ensure that the competency domains and related criteria are objective, accurate and feasible.

A professional position of nurse manger or above was possessed by all experts. The nurse manager is equipped with, as required by their responsibilities, proficient knowledge, experience, and skills in the settings of clinical practice, education, administration and research. In view of this, they are supposed to offer valuable suggestions and comments.

The Delphi consultation experts represented three age groups: the old, middle-aged, and young, with 42% below 40 years old, 48% between 40-50 years old, and 10% above 50 years old. The old-aged experts usually manifest greater experience and higher level of proficiency in clinical nursing, education, administration and research, while the middle-aged and young experts show advantages of being more active in thinking and more open-minded towards state-of-the-art. By combining views from experts of all these 3 age-groups, we would achieve more persuasive results necessary for an authoritative, developmental and prospective study.

The experts had a period of 13.10 ± 6.26 years working in nursing, and some of them had been working in since this department was established. The entire period of nursing is closely related with experts’ familiarity with consultation contents: a longer engagement suggests a higher level of familiarity, leading to more valuable suggestions, and more accurate and credible results.

Experts’ authority is defined by their level of expertise, level of familiarity with the consultation contents, and their basis for judgment. The professional title serves as an important factor to indicate the level of expertise. Experts in our Delphi consultation demonstrated an overall high level of professional titles, with 58% having senior titles, 32% of intermediate titles, and 10% of junior titles. They also showed a generally high level of familiarity with the consultation, with 45% being selfrated as “very familiar with the contents”, 49% as “familiar with the contents”, and 6% as “having knowledge about the contents”. The basis for judgment mainly derives from practical experience, theoretical analysis and data reference. In our study, based on the level of expertise, level of familiarity, and basis for judgment, 31 experts’ authority coefficients were calculated to be 0.700~0.933, with the average of 0.83. According to relevant studies [16-18], it is generally acknowledged that authority coefficient ≥ 0.7 suggests a sufficient credibility. In this view, our experts’ suggestions and consultation results can be evaluated as credible.

Degree of agreement among experts was reflected through degree of variation, approval rating and Kendall’s coefficient of concordance. Degree of variation measures dispersion of experts’ views, with a higher variation degree indicating a lower consistency. Approval rating is the ratio of the experts providing approval opinions to the total participants. A Higher approval rating suggests a higher degree of agreement. For our study, the project team, taking the practical situation into consideration, set the approval rating of 80% as the minimum to ensure the significant agreement among Delphi experts. Kendall’s coefficient of concordance (W) was used to indicate the overall agreement among all experts to all the evaluated data. A smaller hypothesis test P value suggests a higher consistency and concordance among expert’s opinions, thus more credible results [19].

In the first Delphi round, approval ratings for different items defined as the first-level domains varied from 87.1% to 100%, with 100% for professional quality, physio-psychological constitution, scientific research competency, and development work, 96.8% for clinical nursing competency and teaching competency, 93.5% for management competency and communication competency, and 87.1% for level of knowledge. In the second round, an approval rating of 100% was received for all the first-level domains except for the scientific research competency, which got an approval rating of 96%. After 2 rounds of consultation, experts’ approval rating for the structure of the first-level domains was ≥ 96%, indicating a high level of agreement among experts. Therefore, the first-level domains can comprehensively define nurses’ core competencies.

Importance of each first-level domains was demonstrated based on an assigned value, with an average of 2.87~3.839 among experts and the degree of variation was 9.7%~23.3%. With reference to relevant research data [17], the project team has deemed that experts had a high agreement in rating importance of the first-level domains, which means a high level of consensus.

After 3 Delphi rounds, experts’ approval ratings for the second-level domains were ≥ 92%, indicating a high level of agreement on the second-level domains in reflecting the contents of the first-level competencies.

In the first round of consultation, the average importance of the second-level domains, reflected by an assigned value, was 3.032~3.935 among experts, with the variation degree of 6.3%~26.2%. Referring to relevant research data [17], the project team has deemed that experts had a high agreement in rating importance of the second-level domains, which means a high level of consensus.

In the second consultation round, value was assigned to reflect experts’ approval degree on the contents of each second-level domains, with the average for each varying from 1.76 to 2.0, and the variation degree of 0%~37.7% (average 20.1%). For the domain construct, the average approval degree was 1.84~2.0 and degree of variation was 0%~25.7% (average 11.36%). This illustrated a high level of agreement on the content and construct of the second-level domains. After the third round, 80%-100% (average 92.72%) experts deemed the contents of the second-level domains appropriate, and 84%-100% (average 93.00%) felt the same with the construct. After 2 rounds of Delphi process (round 2 and round 3), experts tended to show higher agreement on the second-level domains’ content and construct, suggesting that the contents and construct can accurately reflect ICU nurses’ core competencies.

Kendall’s coefficient of concordance (W) for the first-level domains was 0.4, and the hypothesis test P value was<0.0001, indicating a consensus of experts views’ and a credible result. For the second-level domains, Kendall’s coefficient of concordance (W) was 0.26, and the hypothesis test P value was<0.0001, indicating a consensus of experts views and a credible result.

According to relevant data [20], I-CVI>0.78, S-CVI>0.90, inter-rater agreement ≥ 0.70 are supposed to indicate a content-valid assessment scale. In our study, a group of 11 experts, selected from Delphi panel, evaluated the relevance between contents and purposes of the assessment scale, and a statistical analysis resulted in I-CVI of 0.92~1.00, S-CVI of 0.99, and inter-rater agreement of 0.94. Evaluation results show the assessment scale is valid to genuinely reflect ICU nurses’ core competencies.

Weighted Kappa coefficient (Kw) was adopted to evaluate the reliability of this assessment scale. It is generally acknowledged that a weighted Kappa coefficient<0.40 suggests low consistency, while 0.40~0.75 for moderate-to-high consistency, and>0.75 for very high consistency. In our study, Kw was<0.40 for one item of the assessment scale, 0.40~0.75 for 5 items, and>0.75 for 25 items, with the average Kw of 0.77. It is safe to say that the assessment scale has high test-retest reliability.

References

1. Mclagan PA. Competencies: the next generation. Training Dev 1997; 51: 40-47.
2. Lenburg CB. The framework, concepts and methods of the competency outcomes and performance assessment (COPA) model. Online J Issue Nurs 1999; 30: 2.
3. Strasser S, London L, Kortenbout E. Developing a competence framework and evaluation tool for primary care nursing in South Africa. Educ Health (Abingdon) 2005; 18: 133-144.
4. Scribante J, Muller ME, Lipman J. A guideline for competency of the critical care nurse. Am J Crit Care 1996; 5: 217-226.
5. Pascual J, Bueno P, Cuenca M, Asiain MC, Marín B. Certification of professional competence in critical care. Opinion survey. Enferm Intensiva 1998; 9: 16-20.
6. Dunn SV, Lawson D, Robertson S. The development of competency standards for specialist critical care nurses. J Adv Nurs 2000; 31: 339-346.
7. Aäri RL, Tarja S, Helena LK. Competence in intensive and critical care nursing: a literature review. Intensive Crit Care Nurs 2008; 24: 78-89.
8. Leonard BJ, Plotnikoff GA. Awareness: The heart of cultural competence. AACN Clinical Issues: Advanced Practice in Acute and Critical Care. Compl Altern Therap 2000; 11: 51-59.
9. Curran CR. Informatics competencies for nurse practitioners. AACN Clinical Issues: Advanced Practice in Acute and Critical Care. Nurs Inform Found Nurs Professionalism 2003; 14: 320-330.
10. Ford LA, Wickham VA, Colver C. Developing a skills fair workshop: enhancing competency performance. Dim Crit Care Nurs 1992; 11: 340-346.
11. Fisher MJ, Marshall AP, Kendrick TS. Competency standards for critical care nurses: do they measure up? Aust J Adv Nurs 2005; 22: 32-39.
12. Yan RQ, Shen N. Research and development of nurses core competencies. Chin Nurs Res 2004; 18: 201-203.
13. You LL. Why and how to develop the core abilities of the nurse. Nurs Pract Res 2007; 4: 67-68.
14. Hennessy D, Hicks C. The ideal attributes of chief nurses in Europe: a Delphi study. J Adv Nurs 2003; 43: 441-448.
15. Yang L, Yan ZM, Chen L. Development of food hygiene standards and the assessment system. J Prevent Med Inform 2006; 22: 25-28.
16. Zhou MH, Li YP. A Delphi study to construct an integrated assessment system for hygienic cities. Chin J Publ H Manag 2001; 17: 260-263.
17. Guan XQ, Li RX, Liu YC. Research and application of assessment for education of medical postgraduates. Beijing Milit Med Sci Press 2000; 12.
18. Collier J, MacKinlay D, Phillips D. Norm values for the Generic Childrens Quality of Life Measure (GCQ) from a large school-based sample. Qual Life Res 2000; 9: 617-623.
19. Cheng C, Liu YZ, Wang RD. The test for Kendalls coefficient of concordance W by SPSS. J Taishan Med Coll 2010; 31: 487-490.
20. Liu K. Test of content validity. J Nurs Train 2010; 25: 37-39.