Benchmarking the performance of women scientists

Summary¹

In 2005 NACI commissioned the Centre for Research on Science and Technology (CREST) at Stellenbosch University to design a monitoring and evaluation framework to benchmark the performance of women in the National System of Innovation (NSI). CREST developed a dynamic monitoring and evaluation framework and proposed four different application scenarios. In each case the application scenario (or intended audience and use) will determine the level of information gathering, the cluster of indicators to be used and the frequency of data collection.

The four application scenarios are:

• System monitoring: Reporting the Minister (annually);
• Sector monitoring: To inform policies and interventions (three-year cycle);
• International benchmarking: For international comparisons (three-year cycle);
• System review: For comprehensive review of the system (every six years).

For each of these four application scenarios, CREST recommends a different combination of constructs and indicators.

The proposed framework uses data available in the system, but also makes recommendations for the collection of additional data that are not readily accessible or not available at all. The cost implications must be considered in the context of the application scenario.

The authors demonstrate that it will be complex and expensive to source the required data sets for the comprehensive monitoring and evaluation framework. The data comes from many different organisations and much of it is not readily available. In many cases, obtaining data will require special requests, negotiations with relevant agencies or specially commissioned surveys.

The nine evaluation and monitoring constructs, each associated with specific indicators and policy concerns, are:

1. SET potential (leakages in the pipeline, distribution across broad study fields, size and potential of SET and R&D pool);
2. SET labour force (SET human resource capacity, horizontal distribution across SET occupations, absorption of SET graduates);
3. R&D workforce (R&D human resource capacity, horizontal distribution across sectors, absorption of SET graduates);
4. Fairness and success in funding (access to funding, distribution of funds, funding amounts);
5. Rank and employment (vertical distribution within sectors, permanent appointments and promotions);
6. Scientific agenda setting (distribution of executive and senior managers across sectors; representation on scientific boards and councils);
7. Scientific recognition (recognition by peers, distribution of reviewers for scientific journals and funding agencies, membership profiles of science academies, citation ratings);
8. Scientific output (authorships and publications);
9. Scientific collaboration and networking (co-authorships, collaborative research projects, conferences and sabbaticals).

The development of the framework was informed by an extensive literature review, as well as by a series of consultative workshops. During the process of refining the indicators, CREST created opportunities for stakeholders to comment.

¹ The comprehensive report entitled: "A monitoring and evaluation framework to benchmark the performance of women in the NSI", published December 2005, is available from the National Advisory Council on Innovation.