Process evaluation and quality assessment in family planning services in Rosario, Argentina

Process evaluation and quality assessment
in family planning services in Rosario, Argentina

Marcelo R. Raffagnini
Hospital Dr. Roque Saenz Peña
Secretaria de Salud Pública de la Municipalidad de la Ciudad de Rosario
Rosario, Argentina

See also presentation

Abstract

Background: Reproductive Health Counseling, a new strategy in our Family Planning Services, was carried out in the last years, in Rosario, Argentina, mainly focusing on a group of women defined to be at “high risk”. The aim of this study is to analyze the effect of this intervention by measuring the use of the Responsible Procreation Programme, and to assess the user's satisfaction. Methods: we will conduct a multilevel regression method to assess the impact of the strategy. Results: at the end of the study we expect to answer this question: “do communities with a stronger family planning program presence, tend to have higher contraceptive prevalence rates and/or lower fertility rates than those with lesser program presence?”

Background

During the last years, in Rosario, Argentina, a new strategy in Family Planning Services: “counselling in reproductive health” was carried out (1). A group of women was identified, including: post partum women with four or more children, or with medical conditions increasing the risk of adverse pregnancy outcomes, post partum women with HIV/AIDS, adolescents and post abortion cases.

Given the widely recognized importance of quality of care in the provision of family planning and sexual and reproductive health services, there is a large need to develop simple means of evaluating quality of care. Of particular interest are approaches that take into account women's satisfaction.

According to the literature (2, 3, 4, 5, 6, 7, 13, 14, 15), there are three preferred methods for impact assessment in family planning (8):

Experimental Studies
Quasi-experiments
Observational Studies

Experimental Studies (Randomized Trials)

It is widely accepted among epidemiologists that randomized controlled trials are the “gold standard” for an unbiased measure of a program or intervention. In such experiment, study subjects or groups are assigned to “treatment” and “control” groups randomly; that is, once the subjects or groups of subjects to be studied have been chosen, some are assigned to the treatment group and some to the control (or comparison) group at random. Given a sufficiently large sample size, randomization enhances the chances that the treatment and control groups are equivalent with respect to all factors other than exposure to the program being evaluated. This conveys an enormous advantage over other methods of measuring program impact. (9) Randomized experiments have limited application in the evaluation of the Rosario Responsible Procreation Program (RPP), especially in the measurement of “Counselling” intervention, because such strategy is actually in use in different services with varying strength, generating a contamination between possible groups or clusters.

Non-experimental Studies

It refers to a group of experimental research designs in which study subjects or groups of subjects are not randomly assigned (10). The most commonly used quasi-experimental designs, “constructed control designs,” follow the same logic and involve the comparison of treatment and control subjects or groups of subjects as in randomized experiments. A design that has the widest applicability in assessing the impact of family planning programs is the pretest-post-test, non-equivalent control group design. For example, subjects or population subgroups that are as similar as possible to the treatment group with respect to economic status, geographic location, ethnicity, and other characteristics might be chosen on purpose to serve as comparison groups. Alternatively, a population subgroup that is similar to the treatment population may be identified and a random sample chosen to serve as a comparison group.

Observational Studies

Impact assessments based upon multilevel regression methods fall under the general heading of non-experimental or observational studies (i.e. studies without control groups). Because the approach is non-experimental, the treatments vary from area to area as a result of a decision making process that is beyond the control of the researcher. The criteria underlying the program location or resource allocation decisions may or may not be known to the researcher.

The basic idea is to try to demonstrate a statistical relationship between measures of family planning program activities and selected outcome measures (e.g., contraceptive prevalence, total fertility rate, unintended pregnancies, discontinuation) using services and districts as the unit of analysis, while holding constant the effects of non-program factors such as age, urbanization, female education and labor force participation, ethnicity, etc. The effects of non-program factors on the outcomes of interest are controlled through the application of regression methods. The approach attempts to answer the question “ do communities with a stronger family planning program presence tend to have higher contraceptive prevalence rates and/or lower fertility rates than those with lesser program presence?»

Objectives

General aim

To evaluate the effects of a new component included into the Responsible Procreation Program (RPP) directed towards the promotion, prevention and assistance in reproductive health on a selected group of women at high risk, in terms of their behaviors regarding the acceptance of the program, as well as their socio-cultural and migratory characteristics, conceptions and practices related to sexuality and reproductive health and to assure and to improve the provision of contraceptive methods for all women in the programme during a period of eighteen months.

Specific aims

To analyze the effects of “Reproductive Health Counseling” as a new strategy to improve the inclusion into the program of a selected group of women at “high risk”, carried out at two Municipal Maternity’s Hospitals, in the gynecology services of Hospitals Sáenz Peña, Carrasco, Alberdi and Vilela, Martin Maternity and in Primary Care Centers.
To analyze the behaviors of a selected group of women defined at high risk, included into the RPP, regarding the choice of contraceptive methods, place of assistance, appointment of the citations and permanency in program for a year, depending on their socio-cultural and migratory risk characteristics, and conceptions and practices related to sexuality and reproductive health, as well as the level of the satisfaction with the service provided, and reasons for drop-out.
To provide contraceptive methods to all women into program during a period of eighteen months.

Methods

We will conduct a process evaluation to asses the results of the intervention “Counselling in Reproductive Health” in an extended modality that will be defined by the RPP and projected to the Primary Care Centers, which have not incorporated this strategy yet.

Extended modality means a more active and systematic participation of Health Promoters, intensifying direct contacts with the women at high risk. This implies a follow-up from the first general interview of promotion of the RPP, with successive contacts in order to identify the reasons for these critical situations: not included into the program and the possible drop-out , as well as the reinforcement and support and advise from Health Promoters. Furthermore, we will measure client satisfaction throughout a year of follow up.

A. Study population

The study population comprises women living in Rosario, attending municipal health centers, without insurance coverage, with high risk pregnancies; teenagers (less than 20-year-old), women with 4 children or more, women who recently underwent abortion, and women tested positive for HIV/AIDS.

Eligibility Criteria:

a) Women attending Municipal Maternities (Martin and Roque Saenz Peña) for delivery or abortion during the study period, in order to intensify preventive actions to avoid unintended pregnancies multiparous or with recent abortion, HIV/AIDS or with diseases that complicate pregnancy or puerperal period

Available information for 2002 (attached table 1) indicates that the mean monthly number of women at risk was of 262, 177 in the Martin Maternity and 85 Roque Sáenz Peña, contributing each one, respectively, 67.6 % and 32.4 % of the monthly cases of risk.

We estimated a sample size of 300 women, that can be recruited during a period of 35 to 40 days:200 cases in Martin Maternity, (with an average of 6 high -risk pregnant women delivering per day), and 100 in the Roque Saenz Peña Hospital (with an average of 3 high -risk pregnant women delivering per day).

b) In promoting primary prevention, teenagers are another target group (less than 20 years old).

The sample will consist of 450 teenagers, which will be contacted immediately after the family planning consultation. In every hospital the recruitment will be done in successive consultation days, without interruption, until till the total of 30 cases is reached.

There will be 10 municipal Primary Health Care Centers among the 30 presently associated with the RPP, two by district, based on the Municipal Epidemiology System’s information. Selected centers (attached table 2) correspond to two different situations in every district: the first centre with the large recruitment, and the second one (with a similar volume of annual consultations), with a small number of recruitment.

In a similar way to the adopted mechanism in hospitals, the teenagers will be contacted in the successive days of the consultation, without interruption, till a total number of 30 cases by centre (overall total of 300).

B. Methodology

Selected will be cases after delivery or abortion (300 women) or after consultation for contraception (450 teenagers). These women compose the cohorts to be followed-up over a year, using the method of "tracers" as Kessner describes.(12)
At the moment of patient selection, and for study purposes, Health Promoters will carry out the first interview using structured interview forms.
This first contact will have the following aims:

To inquire about knowledge concerning sexual and reproductive health, prevention of sexually transmitted diseases, about RPP and available services, possible options in relation with contraceptive methods, etc.
To identify information sources to improve the above mentioned knowledge (services, professionals, etc.)
To investigate their believe and practices related to sexuality and reproductive health, and about socio-cultural and migratory characteristics.
To report the characteristics of the RPP, services and options that it offers and to orientate respect places of more accessible assistance to every case.
To investigate the reasons (advantages or barriers) for access to services and assistance to seek care.

After the indication of a contraceptive method, first observation is planned two months by the first interview. Such action will involve the review of patients' records, to check the women incorporation to the program. Successive visits will be according to women's choice at first interview (e.g. home visits or at the service, phone call or e-mail) in the following cases:

Women who have not returned to RPP after two months from the first contact with care providers.
Women who had chosen a contraceptive method and did not come for medical control or for renewal of prescription

Special forms will be used to obtain information about the reasons for not using the program, drop-out or not attending for control visits. In these successive sessions, promoters will reinforce their Counseling activities supporting the RPP.

A basic protocol will be implemented to systematically collect the information of all the Services.

Such protocol will allow to certify women inclusion into RPP and to coordinate the fieldwork (domiciliary questionnaires), moreover, it allows obtaining general information of all “high risk” women that use the services (comparison base to measure the effects of Counseling strategy).

The lack of precedents that allow to estimate the number of necessary interviews, leads to establishing a similar number to that of the women into cohorts (750), giving the possibility to repeat two or more interviews if necessary.

On having finished the year of follow-up, a last interview is foreseen to the women who remained into program, in services and after consultation. This will allow registering user visions and opinions from experience assembled contacting health services, accessibility conditions and level satisfaction level, also by a constructed instrument to such purpose.

A basic record system by service and district, will be developed, including contraceptive method, and personal data of women, to delivery contraception to all others women into program.

C. Organization

A work group will be organized to coordinate the project, incorporating a member of the RPP management team by district and researchers who will lead the study, and a designated field coordinator.

This group will have the following responsibilities:

To develop the basic protocol for systematic data collection and to support the study development.
To implement the new record system at all municipal gynecology services and Primary Health Care Centers, with the corresponding training of professionals involved.
To elaborate the questionnaires instruments corresponding to first and subsequent interviews with the selected women.
To specify procedures and responsibilities linked to the initial recruitment of study cohorts; searching, organization and information supply, during interviews, and their accomplishment in established times, etc.
To qualify and to supervise Health Promoters actions, establishing counseling and interviews procedure guidelines, to assure the information reliability.

D. Analysis plan

Two analytical lines are defined:

The first is to understand women's behavior in relation with the RPP, why they remained or left the program and their satisfaction with the received care during the study period.
The other line is the measurement of the effects of the “Counselling” strategy, where the reference will be the general information produced by services about recruitment and program attendance by women in the month before the study is initiated.
In both lines, a multivariate analysis is foreseen, assessing women's risk conditions, adopted methods, socio-

cultural and migratory precedents, conceptions and practices related to sexuality and reproductive health, and taking as classification factors the services women selected (e.g. gynecology department, primary health care centers with high or low recruitment).
In relation to the third aim, a descriptive analysis will be done by district and centre for the amount of inputs distributed by contraceptive method and woman's age.

Results

At the end of the study we expect to answer this question: “do communities with a stronger family planning program presence, tend to have higher contraceptive prevalence rates and/or lower fertility rates than those with lesser program presence?”

References

Secretaría de Salud Pública. Municipalidad de Rosario. Plan de Salud Integral de la Mujer. (Subprograma Programa de Procreacion responsible. Primer documento de Consenso. Año 2001.
Young Mi Kimitri Putjuk, Endang Basuki and Adrienne Kols Increasing patient participation in reproductive health consultations: an evaluation of "Smart Patient" coaching in Indonesia Patient Education and Counseling Volume 50, Issue 2 , June 2003, Pages 113-122
Baraitser P, Fettiplace R, Dolan F, Massil H, Cowley S. Quality, mainstream services with proactive and targeted outreach: a model of contraceptive service provision for young people. J Fam Plann Reprod Health Care. 2002 Apr;28(2):90-4. [PubMed]
Williams T, Schutt-Ainé J, Cuca I.: Measuring Family Planning Service Quality Through Client Satisfaction Exit Interviews. International Family Planning Perspectives, 2000, 26(2):63–71
McAllister KF, Elliott L, Thomson DA, Bigrigg A, Mackie CM. Evaluation of a young person's sexual health service in a commercial setting. J Fam Plann Reprod Health Care. 2002 Oct;28(4):203-6. [PubMed]
Soliman MH. Impact of antenatal counseling on couples' knowledge and practice of contraception in Mansoura, Egypt. East Mediterr Health J. 1999 Sep;5(5):1002-13. [PubMed]
Bessinger R, Bertrand JT. Monitoring Quality of Care in Family Planning Programes. A Comparison of observations And Client Exit Interviews. International Family Planning Perspectives. 2001, 27 (2): 63-70.
Rossi, PH, Freeman H. Evaluation: A Systematic Approach. Newbury Park, CA: Sage Publications. 1993
Bertrand JT, Magnani RJ, Ruttenberg, N. Evaluating Family Planning Programs. Set 1996.
Rothman K, Greeland S. Modern Epidemiology. 5:67-90.
Secretaría de Salud Pública – Sistema Municipal de Epidemiología. Documento de Trabajo sobre actividades y resultados del PPR hasta el año 2000.
Kessner DM, Kalk CE, Singer J. Assessing health quality--the case for tracers. N Engl J Med. 1973 Jan 25;288(4):189-94. [PubMed]
Xu B. Counseling and service-based family planning: Luwan, Shanghai. China Popul Today. 1999 Oct;16(4-5):9-11.[PubMed]
Bruce J. Fundamental elements of the quality of care: a simple framework. Stud Fam Plann. 1990 Mar-Apr;21(2):61-91. [PubMed]
Hemminki E, Sihvo S, Koponen P, Kosunen E. Quality of contraceptive services in Finland. Qual Health Care. 1997 Jun;6(2):62-8. [PubMed]

Tables

Table 1: No of deliveries/abortions per health care centre/per month (average)

Monthly average	Maternity Martin	Maternity hospital Dr. R. Sáenz Peña	Total
Women attending the hospital for delivery/abortion	345	153	498
Subtotal ( No of women eligible )	177	85	262
- adolescents( < 20 y)	95	43	138
- parity > 4	57	26	83
- with hypertension, pre-eclampsia, eclampsia, cardiac disease, diabetes	20	13	33
- fetal abnormalities/late abortions	3	2	5
- with HIV/ AIDS	2	1	3

Table 2: Selected Health Care Centers

District	Health Care Centre	Participation in the RPP (% of women attending of the total No of women in the district)
West	D. Staffieri	54,5%
West	Rosselló	4,4%
North	C. Casas	42,3%
North	1º de mayo	6,2%
South	Sur	55,6%
South	Luis Pasteur	0,8%
Northwest	R. Coulin	30,7%
Northwest	J. H. Dunant	3,4%
Southwest	Champagnat	34,5%
Southwest	San Vicente de Paul	6,5%