|
|
SEPT Summer Project
Evaluation Report
July - October, 2000
Principal Investigators
Usha George, Ph.D. Wes Shera, Ph.D.
Research Team
Madalena Silva Joanne Daciuk Mary Steen
Table of Contents
Introduction * Background * The Planning of the SEPT Summer Project *
Structure of the SEPT Summer Project *
Methodology of the Evaluation *
Findings * Nature of Services *
Functioning of the SEPT Summer Project *
Tracking Form Data Analysis *
Recommendations * Concluding Notes * SEPT SUMMER PROJECT- PLANNING GUIDE *
This report outlines the findings of the independent evaluation of the SEPT Summer Project. The purpose of this report is to provide constructive feedback on the functioning of the SEPT Summer Project over its six-week time-frame, in order to enhance the effectiveness of the SEPT Summer Project. The SEPT Summer Project is an innovative project designed to assist newcomers with their settlement needs. The SEPT Summer Project is a continuation of the SEPT Project initiated in September 1999. It is funded by the Ontario Administration of Settlement and Integration Services (OASIS), Citizenship and Immigration Canada (CIC). The aim of the SEPT Summer Project was to facilitate newcomers’ access to settlement services. SEPT workers would link newcomers to community-based services by working collaboratively with staff from service providers such as libraries, community health centres, Parks and Recreation and a limited number of schools with summer programs. According to CIC, the activities that were to be undertaken by the clusters during the SEPT Summer Project were as follows:
The Planning of the SEPT Summer Project There is no general consensus as to when the decision to have a SEPT Summer Project was finalized. In the minutes of the Steering Committee meeting of January 1999, project renewal was mentioned, but no explicit discussion of the SEPT Summer Project was recorded in the Steering Committee minutes until May 2000. In May 2000, OASIS issued an invitation for proposals outlining the criteria for participating in the SEPT Summer Project. The turn-around period for lead agencies to complete proposals was four weeks. The proposals were to be received by OASIS on June 23, 2000. One cluster decided not to take part in the SEPT Summer Project. The SEPT Summer Project officially started service delivery on July 3, 2000, and ran until August 18, 2000. Structure of the SEPT Summer Project The SEPT Summer Project had an administrative structure similar to that of the regular school year. Cluster management, the Steering Committee and the Evaluation Committee continued to function. The Liaison Team was dissolved. The search to hire for the position of SEPT Coordinator was initiated, although the search was not completed. Other changes continue to be considered as a result of the Toronto District School Board (TDSB) restructuring. As of June 2000, the TDSB has four administrative areas with four area offices. Each area office has an area manager, who will be the contact for SEPT Project in each area. The administrative areas are the North Area (North York), the West Area (Etobicoke and York), the South Area (Toronto and East York), and the East Area (Scarborough). For a complete review of the structure and overall management of the SEPT Project, please refer to the Final Evaluation Report.
The evaluation of the SEPT Summer Project was an extension of the evaluation that took place during the school year of 1999-2000. The evaluation of the SEPT Summer Project commenced on July 3, 2000. The purpose of this evaluation was to recommend changes to the operationalization of the service delivery model, or to the model itself, which would result in more efficient and effective service delivery to newcomers. This evaluation was carried out by a research team from the Centre for Applied Social Research, Faculty of Social Work, University of Toronto. The principal investigator was Usha George, Ph.D, and the co-principal investigator was Wes Shera, Ph.D. Joanne Daciuk was the data analyst and the project coordinator was Madalena Silva. Mary Steen worked as a research assistant for the evaluation of the SEPT Summer Project. A stakeholder service model of project evaluation was used. Therefore, ongoing consultation was provided by the Evaluation Team to the management of the SEPT Summer Project. One of the major strengths of this model was that it allowed the Evaluation Team to maintain the rapport already established with the stakeholders, and thus, be informed of sensitive issues that might not have been brought to the attention of reviewers using another evaluation model.
Semi-structured interview schedules were used for all interviews. A number of interview schedules were developed for the different stakeholder groups. To review all interview schedules, please consult Appendix One of this report.
Data Collection for the SEPT Summer Project The data collection methods used for this report are summarized in the following table.
Table 1: Summary of Data Collection for the SEPT Summer Project
Seventeen key informants were invited to participate in the interview process. They included contact people for the SEPT Summer Project in libraries, Parks and Recreation sites, community agencies, and partner agencies in which the SEPT workers were stationed. Eleven key informants were interviewed. Six key informants decided not to participate in the interviews due to their limited contact with the SEPT workers. Some key informants stated that they did not have an understanding of the mandate and services of the SEPT Summer Project, and that they had only agreed to provide facilities for the summer. Three of the six key informants who did not wish to be interviewed reported that they had never seen SEPT workers at their locations. The coordinators of the seven clusters participating in the SEPT Summer Project were interviewed for this report. Individual interviews were also held with eighteen SEPT workers. Three summer school principals were interviewed. Thirteen on-site visits were conducted in a variety of locations, including libraries, Parks and Recreation sites, community agencies, and schools in which summer school took place. The Evaluation Team also observed five group information sessions. Document analysis for the evaluation of the SEPT Summer Project included, but was not limited to the following:
Data analysis was conducted by examining common themes that emerged from the interviews, site visit annotations, and observations of the group information sessions. All administrative documents concerning the project and monthly statistics of service delivery were also examined. It should be noted that very little information pertaining to Cluster Four was provided to the Evaluation Team.
Limitations of the Methodology The Evaluation Team recognises that there were certain limitations to this research. One major limitation was the time-frame in which the project was conducted. Another limitation was the summer holidays of SEPT staff and of key informants made it difficult to interview some staff and informants. Some clusters and other stakeholders provided the Evaluation Team with late or incomplete documentation, which hampered data analysis.
The following section reports the findings of the evaluation. This section is divided into four sub-sections: nature of services, functioning of the SEPT Summer Project, day-to-day implementation of the SEPT Summer Project, and tracking form analysis. Since these sections are interrelated, there may be some overlap between them.
Services Provided by the SEPT Summer Project
The SEPT Summer Project offered a wide range of services. Many of the services of the SEPT Project were integrated into the SEPT Summer Project. During the summer, a large portion of the SEPT workers’ time was dedicated to outreach and general promotion of SEPT. To this end, a variety of methods were used, including information booths in local malls, distribution of flyers in residential areas, and presentations to various community agencies.
Services Delivered in Community Agencies and Institutions The collaboration between the SEPT Summer Project and community agencies/institutions indirectly prompted SEPT workers to promote the services of these agencies to newcomers. The analysis of the data shows that a variety of activities took place at the local libraries, such as information and orientation sessions on library services, translation of library flyers, and assistance to clients registering for library cards. One-on-one information and referral services were also provided in some libraries. Through these activities, SEPT workers created a role for themselves within the library environment. Due to the late start of SEPT Summer Project, many SEPT personnel reported that there were problems in working with Parks and Recreation. The registration for summer programs had already taken place, and most programs were completely booked by the time the SEPT Summer Project began. SEPT workers could only inform clients about the registration process for subsequent sessions. Some one-on-one services were also provided in these settings as well. Many SEPT workers were not satisfied with their work in Parks and Recreation locations. SEPT workers generally reported feeling accepted in community health centres and they were satisfied with the facilities available to them in such centres. The SEPT workers were involved in activities, such as support groups and outreach. They also saw clients on a one-to-one basis. They felt that their services complemented those of the community health centres well. Outreach, information and referral services were provided in shopping malls in some clusters. Some group sessions were also held in shopping malls. Generally, workers preferred that the malls be used for a limited amount of outreach and group sessions. Some of the SEPT workers interviewed did not wish to work for long periods of time in malls because they felt that the atmosphere was not conducive to building rapport with clients. Still, they could meet a high volume of clients in some locations, which was excellent for outreach. These locations were most satisfactory when the mall provided other social services or when SEPT workers were given a highly visible information booth. Apart from the general comments in the preceding paragraphs, the small size of the sample made it difficult to identify any tendencies with regard to which kinds of agencies/institutions worked best with the SEPT Summer Project. Generally, there were large differences between branches of the same agency/institution in terms of the acceptance of SEPT workers. Not surprisingly, SEPT workers felt more comfortable and valued in those locations that were most open to newcomer clients. In some community agencies/institutions, it was felt that the SEPT workers merely required access to an office for the summer, and that there would be no relationship or contact between the agency/institution staff and the SEPT workers. Further, not enough planning took place to come to a common understanding of what SEPT workers would do during the summer in order to maximise the efficiency of services delivered to clients.
Services in Schools with Summer Programs The time-frame for summer school is very limited, making it difficult to establish a relationship with the summer school administration. Some summer school principals felt that SEPT workers should only work in summer schools with ESL programs or very high concentrations of newcomers. The SEPT workers who were located in schools during the summer generally did not experience the same kind of disorientation that they did when working in other kinds of locations. The SEPT workers often used this time to follow-up with existing clients. Groups Sessions and Site Visits A variety of group sessions took place during the summer on topics such as library catalogues, Internet training, summer employment, parenting, and housing. Although the sessions were generally informative and helpful, sometimes connections were not made between the topics of the session and newcomer settlement services. Further, when the sessions were introduced to participants, there was sometimes no reference to the SEPT mandate nor to any assistance that SEPT workers could provide. Often more than one SEPT workers attended the group sessions. Sometimes it was appropriate, as there were clients who spoke different languages at the sessions. However, it was observed that in one case, there were SEPT workers who spoke the same language at the group session. This seemed to be an inefficient use of time. The Evaluation Team observed that logistics, such as location, the day of the week and the scheduled times for sessions were sometimes inappropriate. These details have a tremendous impact on the number of participants who are able to attend. During the Evaluation Team site visits, a recurring theme was the inadequacy of work facilities. Some of the offices in which SEPT workers were stationed were difficult to access or the workers were sitting in an open area with the general public. Many SEPT workers had no access to necessary office equipment or supplies at the locations at which they were placed. The Evaluation Team observed that were relatively few posters announcing the presence of SEPT workers in the various locations. The Evaluation Team did not observe any SEPT workers’ schedules posted to inform the agencies, staff and, potential clients as to the availability of the SEPT workers. This issue was also raised by key informants who said that they were sometimes unsure of the hours of the SEPT workers.
Functioning of the SEPT Summer Project
Collaboration with Community Agencies and Institutions SEPT workers were located at four main locations for the Summer Project: libraries, community health centres, Parks and Recreation facilities and in a limited number of schools. The collaboration between the SEPT Project and these agencies provided SEPT workers with the opportunity to promote settlement services and the services of these agencies to newcomers. The libraries appeared to be generally receptive to the SEPT workers stationed at various branches across the city. However, this relationship was not without challenges; some branches worked better with the SEPT Summer Project than others. As well, workspace and access to office equipment were issues in several libraries. Fewer workers were stationed at Parks and Recreation facilities. The relationship with this organization did not function as well on the local level as it did with other community agencies/institutions. Although, Parks and Recreation was able to provide facilities for the SEPT workers, there was no direct link between the services that SEPT workers provided and the services of Parks and Recreation. All interviewed agreed that the collaboration between SEPT and Parks and Recreation needs a great deal of enhancement. The collaboration with community health centres in which workers were actually placed usually functioned well. The mandates of SEPT and the community health centres generally seemed to complement each other. Some of the services provided by SEPT workers focused on the mandate and services of the agencies/institutions in which they were stationed, such as the translation of flyers and information sessions on the agencies/institutions in which they were placed. It is difficult to determine whether this is an appropriate use of SEPT workers’ time. Such services may be viewed as providing information and orientation to newcomers. However, one can also argue that these services are outside the SEPT mandate and should be provided by the agencies/institutions themselves. Further, this demonstrates the need to clarify the roles and expectations for SEPT workers during the Summer Project.
Day-to-Day Implementation of the SEPT Summer Project SEPT workers’ schedules varied greatly according to cluster. In some clusters, workers were stationed at four or more locations a week. In others, SEPT workers were only situated in three or less locations. Several clusters scheduled regular community outreach. SEPT workers’ holidays also complicated the schedules. The workers who were placed in fewer locations generally felt more positively about the SEPT Summer Project. Many of the workers who were located in four or more locations a week complained about travelling time and/or reported that they did not feel that they became oriented to all of their locations. Some of these SEPT workers said that they felt the SEPT Summer Project would be improved if they were able to stay longer in just a few locations.
SEPT Workers Facilities and Working Conditions Although some SEPT workers had well-equipped offices, the majority of the SEPT workers lacked a private meeting area, access to a telephone or access to a computer during the SEPT Summer Project. Some SEPT workers had to use public pay phones to make work-related calls. The majority of the SEPT workers had to go to their lead agencies or take their work home because of the lack of computers. As well, relatively few SEPT workers had a secure place to keep their new files outside of the lead agency. All of these factors impaired the efficiency, and added to the stress levels of the SEPT workers. The reception that the SEPT workers received greatly influenced their working conditions in each location. When SEPT workers were welcomed and oriented to a new location, they tended to have a better impression of the location and a more positive work experience. Because of the short lead-time of this project, quick orientation to new locations improved the efficiency of SEPT workers. Most SEPT workers found it fairly easy to become adjusted to summer schools. Orientation and adjustment to the other locations greatly depended on the connection that the SEPT workers felt to the staff in each of these locations. It is difficult to state any general tendencies regarding which agencies worked best with the SEPT Summer Project because of the short time-frame and small sample.
Supervision of SEPT workers was similar to the supervision during the school year, except that it generally occurred less frequently. All of the clusters had staff meetings with their workers, only they tended to have the meetings less often. Most clusters only had a few staff meetings during the summer. The coordinators reported that they tried to visit their workers at the different locations throughout the summer. Some also maintained phone contact with their workers, as they did during the year. Sometimes supervision was complicated by the fact that most staff, including coordinators, took their holidays during the summer. In some clusters, the Evaluation Team felt that certain crucial points missed supervisors’ attentions. These items will be outlined in the following section.
Critical Issues in Implementation The Evaluation Team was surprised to find that three of the seventeen key informants who were contacted decided not to be interviewed because they had never seen the SEPT workers who had been assigned to their locations. In two cases, the key informants specifically said that they knew that SEPT workers had never come to their agency/institution. These key informants said that they had not received an explanation about this issue. Some key informants and the Evaluation Team noted that, in at least two cases, clients were left waiting for hours for SEPT workers. Members of the Evaluation Team were also kept waiting for long periods of time in some clusters. As well, appointments were made or changed with very little notice. Information from some clusters was either late or incomplete. Even the short planning time for the SEPT Summer Project fails to adequately explain these particular lapses. It should be noted, however, that these problems were not reported in all of the clusters.
In order to track service delivery, both the SEPT Service Tracking Form and the Group Service Tracking Form were used during the Summer Project.
The Evaluation Team wishes to give several caveats regarding the following data. There were inconsistencies in the instructions given to SEPT workers in different clusters regarding location codes. As well, clusters were provided with codes that could not be read by the teleform data scanning program, resulting in a major review of the location codes to determine their accuracy. The Evaluation Team must also mention the many delays that were experienced in accessing this data. In general, the data was not managed as well as it had been during the school year. Differences between the SEPT Project during the school year and the SEPT Summer Project are noted as general trends. A detailed statistical analysis of the differences between the projects was not undertaken because the two projects are extremely different in terms of the delivery of the services.
Analysis of Data from SEPT Service Tracking Forms A SEPT Service Tracking Form was completed by the SEPT workers for services provided to clients. The tracking form gathered information about client profiles and service delivery, including language of the interview, whether an interpreter was present, referral source, settlement service area, type of settlement services needed, referrals to other agencies, and time spent on delivering services. The terms "newcomer" and "client" are used interchangeably in this section. The SEPT workers completed a total of 1,159 tracking forms for newcomers receiving services in the SEPT Summer Project. This included 919 families who received settlement services in seventy locations, organized in seven geographic clusters throughout the City of Toronto. Table 2 shows the profile of the individual clients served in the SEPT Summer Project. Over half (58%) of the clients served were parents or guardians, 20% were students and 26% of the clients were someone other than a parent/guardian or a student. In the majority of forms tracked (72%), SEPT workers met with only one client.
Table 2: Profile of Clients
Table 3 shows the distribution of clients served in the months of July and August, the clients served by cluster, and the location of the service.
Table 3: SEPT Client Service Distribution
* "Other" includes malls, community centres/Parks and Recreation facilities, community health centres, churches, mosques, "family residence", cultural associations, senior centres, community agencies, and other locations.
Table 4: Location of Service by Cluster
Table 5 presents the clients’ immigration characteristics, including region of last permanent residence, year of arrival in Canada, and current immigration status. The majority of the newcomers (42%) were from South Asian countries, followed by newcomers from Middle Eastern, Southeast Asian, African, and European countries. Half (51%) of the newcomers arrived in Canada since 1999, and the majority (72%) were landed immigrants.
Table 5: Immigration Characteristics
Table 6 shows the distribution of the primary languages spoken by the newcomers served by SEPT. Please note that Bosnian, Croatian, Fukinese, Hebrew, and Hungarian were not reported during the summer months, although these languages were reported during the school year.
Table 6: Primary Language Spoken
In 67% of the interviews with newcomers, the language used in the interview was a language other than English. Usually, interviews were conducted in the primary language of the client. Only 1% of the interviews were conducted with an interpreter present.
Table 7: Language of Interview and Presence of Interpreters
The SEPT Service Tracking Form gathered information about the sources of referrals to the SEPT workers. The referral could be from more than one source. Eighty-seven per cent of the referrals were client self-referrals to the SEPT workers. Only 9% of referrals were from school staff and 7% were made by a community agency. Table 8 shows the settlement service area pertaining to the clients’ visit. The majority of the clients sought help with educational issues, followed by employment, language, health issues, financial issues, and housing.
Table 8: Settlement Service Area
The types of settlement services provided to the clients are listed in Table 9. Provision of information was the most frequently reported settlement service (86%). Approximately three-quarters of the clients required a referral (73%). Support, orientation, and consultative services each constituted approximately 20% of the services provided.
Table 9: Type of Settlement Service
The SEPT Service Tracking Form collected information about referrals to other service providers, such as school services, community agencies and settlement programs. Table 10 shows the referrals to other services or programs. Fourteen per cent of the clients were not referred to any other service or agency. The mean number of referrals to any other services was 1.8 (std. dev=1.43, median =2). The majority of referrals were made to a community agency (78%), followed by referrals to settlement programs (31%). Only 9% of the clients were referred to school services during the summer months.
Table 10: Client Referred to Other Services or Programs
Finally, the SEPT Service Tracking Form collected information about the total time spent delivering services. Table 11 shows that the majority of services delivered by SEPT workers took 30 minutes or longer.
Table 11: Total Time Spent Delivering Service
Results of Cross-Tabular Analysis A series of cross-tabular analyses was performed to determine if there were relationships between variables, such as the referral source to the SEPT workers, the settlement service areas, the types of settlement services and the referrals to other settlement services among parent/guardians, students, new versus repeat clients, year of arrival in Canada, immigration status, language of the interview, language category of the newcomer, school term, school level, and cluster. As in the SEPT Project during the school year, we consider this data to be statistically significant at the .05 level by a conventional and appropriate statistical test (e.g. Chi-square or T-test), and when measures of the strength of the relationship (e.g. Cramer’s V, or Phi [f ]) showed non-trivial values (e.g. relationship that is significant at least at the .05 level, with the strength of the relationship greater than or equal to Cramer’s V=.20). (Witte, R. & Witte, J. Statistics, Holt, Rinehart and Winston, 1980). Very few statistically significant relationships were found in the cross-tabular analysis. The summer data showed patterns similar to those of the school year data. The following are the statistically significant results of the analysis. Repeat clients were more likely to be self-referrals than first-time clients (c 2 =45.82, df=1, p£ .000, f =.201). New clients were more likely to seek information services (c 2 =49.56, df=1, p£ .000, f =.209) than repeat clients, and follow-up services were more frequently sought by repeat clients (c 2 =102.20, df=1, p£ .000, f =.300). Newcomer clients other than parents or students were more likely to seek consultation services (c 2 =94.37 df=1, p£ .000, f =.285) and were more frequently referred a SEPT Lead Agency (c 2 =48.07, df=1, p£ .000, f =.204). There were differences in the information and referral services delivered depending on the language category of the client. South Asians (94%), Europeans (91%), other language groups (85%), and people from the Middle East (83%) were more likely to seek information services than South East Asian (70%) (c 2 =67.67, df=4, p£ .000, Cramer’s V =.242). People from the Middle East (87%) were more likely to seek referral services than South Asians (73%), Southeast Asians (68%), Europeans (66%), and other language groups (59%) (c 2 =48.56, df=4, p£ .000, Cramer’s V =.205). The time spent delivering service also varied by language group. People from the Middle East (75%) were more likely to receive service for over 30 minutes than South Asians (65%), Europeans (60%), other language groups (51%), and Southeast Asians (44%) (c 2 =51.50, df=4, p£ .000, Cramer’s V =.213). Tables 12 and 13 shows the settlement service area and the type of settlement service delivered by cluster. There were significant differences found between clusters in clients seeking education and immigration settlement service. Also, significant differences were found in the consultative, information, interpretation, orientation, referral, support, and follow-up services delivered by cluster.
Table 12: Settlement Service Area by Cluster
Bold represents a statistically significant relationship; Italics represents a tendency; and Normal Font represents no statistically significant relationship a (c 2 =39.97, df=6, p£ .000, Cramer’s V =.186)b (c 2 =58.96, df=6, p£ .000, Cramer’s V =.226)c (c 2 23.70, df=6, p£ .000, Cramer’s V =.143)d NSe (c 2 =32.17, df=6, p£ .000, Cramer’s V =.167)f (c 2 =20.59, df=6, p£ .000, Cramer’s V =.133)g (c 2 =19.14, df=6, p£ .000, Cramer’s V =.128)h (c 2 =48.79, df=6, p£ .000, Cramer’s V =.205)i NSj (c 2 =33.55, df=6, p£ .000, Cramer’s V =.170)k NSl (c 2 =22.94, df=6, p£ .000, Cramer’s V =.141)m NS
Table 13: Type of Settlement Service by Cluster
Bold represents a statistically significant relationship; Italics represents a tendency; and Normal Font represents no statistically significant relationship a (c 2 =89.23, df=6, p£ .000, Cramer’s V =.277) b NSc (c 2 =69.86, df=6, p£ .000, Cramer’s V =.246)d (c 2 =87.242, df=6, p£ .000, Cramer’s V =.274)e NSf (c 2 =55.01, df=6, p£ .000, Cramer’s V =.218)g NSh (c 2 =98.52, df=6, p£ .000, Cramer’s V =.292)i (c 2 =62.56, df=6, p£ .000, Cramer’s V =.232)j NSk (c 2 =84.32, df=6, p£ .000, Cramer’s V =.270)
Summary of Trends in SEPT Service Tracking Form Data Analysis There were some differences between the kinds of clients who were seen by SEPT workers during the summer compared to clients seen in the school year. There were almost four times as many new clients as there were during the school year, and new clients actually outnumbered repeat clients. This may indicate that there was more outreach to new clients during the SEPT Summer Project. As well, clients in the "other" category (clients who were neither parents nor students) increased. The percentage of visits by individual clients to the SEPT workers increased compared to visits of two or more people. There were not many differences in the ethnic backgrounds of the clients served during the SEPT Summer Project compared to the clients served during the school year. The only difference was that almost twice as many South Asian clients were served, and far fewer European clients visited SEPT workers during the summer. The information and referral services delivered varied according to the language category of the clients. South Asians, Europeans, other language groups, and people from the Middle East were more likely to seek information services than were Southeast Asians. People from the Middle East were more likely to seek referral services than were South Asians, Southeast Asians, Europeans and other language groups. Almost four times as many clients had arrived in Canada in the year 2000 than during the school year. However, the immigration status of the clients was almost identical to the status during the school year. The frequency of visits to SEPT workers by clients seeking employment services almost doubled during the summer, as did the number of clients seeking recreational services. Service in housing issues more than doubled and service in language issues also increased. The provision of information was still the most frequent service type. Assistance was provided with forms and documentation more frequently during the summer. The frequency of mediation services dramatically decreased during the summer. The amount of support, interpretation and problem-solving services also decreased in the summer months. Repeat clients were more likely to be self-referrals than first-time clients. New clients were more likely to seek information services than repeat clients, and follow-up services were more frequently sought by repeat clients. Newcomer clients in the "other" category were more likely to seek consultation services and were more frequently referred to a SEPT Lead Agency. In comparison to the school year, the frequency of referrals to community agencies almost doubled, and the frequency of referrals to settlement programs, primarily to LINC and job search programs, more than tripled. Referrals to libraries increased five-fold in the SEPT Summer Project, and the frequency of referrals to community centres more than doubled. The length of time for each interview increased during the summer. The length of time of the interviews varied according to language group. People from the Middle East were more likely to receive service for over 30 minutes than South Asians, Europeans, other language groups, and Southeast Asians. There were significant differences found between clusters in the number of clients seeking assistance regarding education and immigration issues. Also, significant differences were found in consultation, information, interpretation, orientation, referral, support, and follow-up services delivered by clusters.
The analysis of data from the group sessions was extremely difficult. Many of the problems with the Group Service Tracking Form data mentioned in the final evaluation report were still in evidence during the SEPT Summer Project. Probably the largest problem was the lack of pre-established topic categories for the group sessions. As a result, this part of the data was full of potential biases. As well, the data was not grouped by cluster, which made cross-cluster comparisons nearly impossible. The Evaluation Team has noted that some group sessions that took place during the summer did not appear in the data set. Indeed, data from at least one cluster seems to be missing, but it is difficult to verify that belief because of the lack of grouping by cluster. For some of the sessions that the Evaluation Team did attend, the number of participants reported was higher than the number of participants that the Evaluation Team observed.
The Evaluation Team divided the 114 group sessions into broad categories for clarity. Group sessions occurred in the following categories:
SEPT Outreach Sessions: These included sessions on the mandate and services of SEPT. They were given to students, parents, and other members of the community.
Settlement Information Sessions: These included information sessions on the topics of housing, employment, immigration, parenting, recreation and summer activities, health, and other settlement issues.
Other: Sessions grouped into the "other" category included those with unspecified topics, field trips, recreational activities, community fairs, and other topics that did not fit into any other category.
Library Information Sessions: These information sessions included sessions on library programs, catalogues, and Internet use in the libraries.
Support Groups: This category included ongoing support groups.
Student Information Sessions: These included sessions on safety and street-proofing, specifically designed for students.
Table 14: Topics of Group Sessions
The Evaluation Team proposes the following recommendations in order to improve the SEPT Summer Project. The recommendations are not ordered according to importance. Recommendation One: The planning of the SEPT Summer Project should begin much earlier in the year. The Evaluation Team suggests that formal planning begin in January. We further recommend that letters of understanding should be signed by the SEPT Project and community agencies/institutions participating in the SEPT Summer Project. These letters should be signed no later than May 2001. These letters of understanding should delineate the responsibilities of each signing party, such as the responsibility to provide office equipment. A common understanding of the role of the SEPT workers in each setting should be agreed upon, and this information should be disseminated to each local branch of the agency/institution. Recommendation Two: Project-wide planning of the SEPT Summer Project should include cluster coordinators to prevent duplication of planning efforts. Recommendation Three: Coordinators and cluster management should be primarily responsible for the placement of SEPT workers in the SEPT Summer Project. Recommendation Four: SEPT workers should work in 3 or less locations each week if they are working full-time. Part-time workers should not work in more than 2 locations. SEPT workers should not be stationed at more than one location during the day. Recommendation Five: Outreach should be carried out in pairs, especially if apartment buildings are part of the outreach plan. Recommendation Six: Publicity and promotion of the SEPT Summer Project should begin by May 2001. Recommendation Seven: Group session topics should be connected to settlement service areas and the SEPT Project should be introduced during each session. Recommendation Eight: Coordinators should provide detailed information to agencies/institutions participating in the SEPT Summer Project about SEPT and inform these community partners if there is to be any major change in the schedule of the SEPT workers. Recommendation Nine: SEPT workers’ schedules should be provided to each agency/institution participating in the SEPT Summer Project. SEPT workers should call their supervisors and their contacts in the various agencies/institutions if they are going to be late, absent or if there is a change in their schedule. Recommendation Ten: The Evaluation Team would recommend that time be allocated for data cleaning of the tracking form data in the future. Recommendation Eleven: It is essential to come to an agreement as to what the roles of the SEPT workers are for the Summer Project. The Evaluation Team recommends that and an ad hoc committee of the Steering Committee be struck to clarify the roles and expectations of the SEPT workers for the Summer Project. The rationale for the SEPT Summer Project and the effort that went into it are certainly admirable. However, the Evaluation Team found that it was sometimes difficult to assess whether problems in the project resulted from simple lack of planning and experience or from more pervasive problems. The SEPT Summer Project performed outreach into the community and provided needed information and referrals for newcomer families. Generally, careful planning, especially with regard to the appropriate placement of SEPT workers, and forging both project-wide and local collaboration with other community service providers should ameliorate many of the initial problems. However, it is still to be seen whether a summer project is the best use of the project’s time and money. Due to the short time frame of this project, the Evaluation Team is not in a position to make definitive conclusions as to which agencies/institutions are suitable for collaboration during the SEPT Summer Project.
SEPT SUMMER PROJECT- PLANNING GUIDE
This guide is based on the findings and recommendations of the SEPT summer evaluation. It is intended for the future planning and implementation of the SEPT project for summer. The following are specific tasks for Steering Committee, Cluster Co-ordinators and SEPT Workers.
At the level of Steering Committee
At the Cluster Level
At the SEPT worker Level
Given the nature of the project, measurement of outcome (measurable change in quality of life achieved by a client between entry into and exit from the program) is not feasible. Therefore we need to pay more attention to the outputs (how much of available service a client actually receives and whether the client receives the full compliment of services specified in the program design). Two ways in which output can be measured are: units of service (data through the tracking forms) and in terms of quality. The quality dimensions can be translated into client satisfaction surveys. There are 14 generally recognized quality dimensions (Martin, 1993 in Kettner, Moroney, and Martin 1999). These are as follows:
Dimension Definition Accessibility The service is easy to access e.g. Signs in place, TTC accessible
Assurance The staff are friendly, polite and knowledgeable Communication Clients are kept informed in language they can understand about the service and any changes
Competence Staff possess the requisite knowledge and skills to provide the service
Conformity The service meets standards
Deficiency Any quality characteristic not otherwise identified that adversely affects customer satisfaction. e.g. available only once a week.
Durability The result does not dissipate quickly- services have a lasting effect
Empathy Staff demonstrate an understanding of and provide individualized attention to client
Humaneness The service is provided in a manner that that protect the dignity and self-worth of the client
Performance The service does what it is supposed to do
Reliability The ability to provide the service in a dependable and consistent manner with minimal variation over time or between client
Responsiveness The timeliness of employees in providing services
Security The service is provided in a safe setting and is free from risk and danger
Tangibles The professional appearance of facilities, equipment, personnel, and published materials
Source: Kettner, P.M; Moroney, R.M; and Martin, L.L. (1999). Designing and Managing programs: An Effectiveness-Based Approach (second edition) Thousand Oaks: Sage Publications. p.128
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||