|
|
Home
| Databases
| WorldLII
| Search
| Feedback
Victoria University Law and Justice Journal |
OUTCOME MEASURES FOR EVALUATING THE EFFECTIVENESS OF JUVENILE JUSTICE PROGRAMS
CAROLINE SPIRANOVIC,[∗] HELEN COCKBURN,[∗∗] LORANA BARTELS[∗∗∗] AND ROBERTA JULIAN[∗∗∗∗]
Within the ‘what works’ literature, recidivism is typically embraced as the sole or primary outcome measure of success for offender intervention programs. Often, no account is taken of other important measures for evaluating program success. As such, our understanding of what works is based largely on programs that have demonstrated effectiveness with respect to reduced recidivism rates. Focusing specifically on tertiary prevention approaches for juvenile offenders, this article argues that there are significant limitations in using rates of recidivism as the primary outcome measure of program success. Firstly, this article explores the importance generally of incorporating a comprehensive evaluative framework into program design. Secondly, the limitations of relying upon recidivism as the sole or primary outcome measure in program evaluation are outlined. Thirdly, this article briefly describes the Risk-Needs-Responsivity (RNR) model and the Good Lives Model (GLM) as examples of models that can be used to inform the selection of appropriate outcome measures for program evaluation. This article then offers three examples of recent outcome evaluation studies which sought to determine the effectiveness of post-sentencing tertiary intervention programs for juvenile offenders using a broad range of indicators of success. Finally, the article suggests alternative outcome measures that might be usefully incorporated in future program design, as well as the monitoring and evaluation of existing programs.
The term ‘evaluation’ covers a range of distinct processes and activities. Evaluations can occur during any stage in the life of a program. As Manning[1] noted, various types of evaluations may be conducted either on their own or in combination to examine different aspects of a program’s success. These different forms of evaluation include:
• context evaluation, which determines the suitability of a program for its intended population (ie, how the program is likely to function in a particular socio-political, physical and economic climate);
• formative evaluation, which determines the needs a program should fulfil and includes evaluation of preliminary results and possibly pilot testing;
• process evaluation, which determines whether the program was implemented in accordance with its design;
• program performance evaluation, which is similar to process evaluation, as this occurs at the decision-making stage and is concerned with assessing the proper implementation of policies and whether these policies are achieving desired goals;
• impact evaluation, which focuses on the positive and negative effects of the program, but is typically concerned with immediate and short-term impacts; and
• outcome evaluation, which assesses program results in the short- and long-term, such as impact on recidivism rates.
Evaluation is an important aspect of program design, particularly where programs are publicly funded. Evaluations demonstrate the value of a program and sustain ‘return on investment’ arguments to justify continued funding. They are a method of quality assurance, highlighting areas of weakness to program staff, encouraging accountability and suggesting targeted responses to improve performance. The importance of embedding evaluation within program design also extends beyond the parameters of the individual program. Evaluation results from past programs provide an evidence base for future policy decisions.[2] Ideally, program evaluation should include all of the forms described above, but the focus of this paper is on outcome evaluations.
With respect to program outcome evaluations, a distinction is often drawn between studies of efficacy and studies of effectiveness. Tightly controlled studies employing experimental designs such as randomised control trials (RCTs) with strict eligibility criteria are examples of efficacy studies. Studies that have less experimental controls and eligibility criteria and are conducted under real-world conditions are generally referred to as effectiveness studies.[3] As this paper will be focusing on examples of program outcome evaluations that employ studies of effectiveness, the term effectiveness, as opposed to efficacy, will be used where relevant throughout.
The importance of evaluation is recognised in key national documents that guide the delivery of juvenile justice support programs in Australia, such as the Australasian Juvenile Justice Standards,[4] the National Framework for Protecting Australia’s Children 2009–2020[5] and the National Indigenous Law and Justice Framework 2009–2015.[6]
The majority of outcome evaluations rely on recidivism rates as the outcome variable by which the effectiveness of the support program is gauged. As Cunneen and Luke[7] noted:
Recidivism studies are a common way of measuring the effectiveness of various criminal justice programs and interventions, and re-offending is a major overall performance indicator for the criminal justice system. Departmental goals are variously set to encapsulate reductions in re-offending and, particularly for juvenile justice services, reducing re-offending may seem to constitute their raison d’être. State audit commissions and the national Productivity Commission hold reductions in re-offending as a primary measure of effectiveness.
They went on to state, however, that ‘[l]ess thought is given to understanding the limitations of the measure that is being used’.[8]
Although it may seem reasonable on face value to rely on recidivism rates as the primary outcome variable and measure of program effectiveness, recidivism as an outcome measure is problematic.[9] Firstly, the term ‘recidivism’ does not denote a single consistent definition of re-offending. Cunneen and Luke[10] noted that the literature classifies encounters with the criminal justice system at many different stages as instances of re-offending. These include police rearrests, apprehensions or cautions, court referrals, court appearances, reconvictions and returns to custody. Each of the contextually dependent definitions has limitations as an accurate and useful measure of re-offending.[11] For instance, returns to custody may overestimate rates of recidivism by including technical breaches that would not be considered genuine cases of recidivism in many instances.
Furthermore, some studies may simply focus on whether there have been any instances of recidivism, rather than considering more significant measures, such as the frequency and severity of re-offending or the time lapse between release and first re-offence. Ideally, as noted by Cunneen and Luke,[12] as well as Richards,[13] multiple measures of recidivism should be adopted that include not only information regarding duration, frequency and severity of any re-offending but also rely upon multiple data sources (eg, arrests and convictions) rather than a single data source (eg, arrests).
In addition to the limitations associated with defining what counts as recidivism, Richards[14] outlined a number of difficulties associated with reliance on this outcome measure as the main proxy for the effectiveness of criminal justice systems generally. Many of the points she makes also apply to the use of recidivism as an outcome measure to assess the effectiveness of interventions for juvenile offenders more specifically. The major difficulties identified by Richards may be summarised as follows:
• Official records of re-offending are only a proxy measure for actual re-offending, as detection rates and clearance rates will influence estimated recidivism rates based on recorded offences in any given period.
• Recidivism rates are influenced by the length of follow-up period, such that longer periods of follow-up are associated with higher rates of recidivism.
• Short-term follow-up periods are particularly problematic when gauging the effects of juvenile justice programs on recidivism, given that offending peaks during adolescence and, as such, juvenile offenders are at an increased risk of re-offending during adolescence regardless of any interventions received.
• There are often unrealistic expectations concerning the level of reduction in recidivism rates that ought to be achieved by an intervention. Given the increased risk for offending during adolescence, a ‘normal’ rate of recidivism for the juvenile offender population of concern should be used as a benchmark to gauge degree of improvement.
• A sole focus on recidivism as the outcome measure for effectiveness underrates the importance of other outcomes, such as improvements in health and wellbeing, as well as education and/or employment achievements, which may in turn have longer term effects on crime and recidivism.
These caveats, associated with reliance on recidivism rates as the primary outcome measure, highlight the importance of employing long follow-up periods wherever possible and of utilising a range of outcome measures, rather than solely relying upon recidivism rates. However, the need to use long follow-up periods may represent a significant challenge in many practical situations where there are often substantial political and financial pressures to obtain results over a relatively short period of time. The extent to which it is appropriate to use relatively short follow-up periods in such situations will depend on a number of factors, such as the intended target group, the data sources that will be used, and the expected effects of an intervention.[15] With respect to the target group, it may be reasonable to use shorter follow-up periods where juvenile offenders are concerned, given that juveniles are more likely on average to re-offend sooner than adult offenders.[16] As noted by Richards,[17] however, it is important that expectations regarding the magnitude of effect of an intervention are realistic, given that adolescence is the peak time for offending. In terms of the data sources used, Payne[18] noted that shorter follow-up periods such as 12 months may be acceptable for police arrest data. However, longer follow-up periods will be required for courts and correctional agencies data, given the often lengthy period of delay that occurs before cases progress to these latter stages of the criminal justice system. Finally, with respect to the anticipated effects of an intervention, it may be reasonable to rely upon short follow-up periods where the effects of an intervention are expected to be relatively short-lived.
The foregoing discussion highlights some of the limitations and methodological challenges associated with using recidivism outcomes to measure program effectiveness. The discussion has highlighted the importance of using a range of outcome measures rather than relying solely on recidivism rates. The following section of this paper therefore discusses models that can be used to guide selection of appropriate outcome measures for juvenile justice interventions, as well as justice interventions more broadly. It is acknowledged that there may also be challenges in adopting other measures, particularly where additional measures may necessitate the use of qualitative rather than quantitative methods. Research in criminology and criminal justice has largely been dominated by quantitative methods, possibly due to the prevailing but diminishing misperception that qualitative methods are scientifically inferior,[19] subjective[20] and ‘inherently unrepresentative and ... unable to be generalised to populations’.[21] It has been argued that these concerns largely reflect misunderstandings about qualitative methods and the application of standards of rigour to qualitative methods that are more appropriate to quantitative methods.[22] Qualitative methods have their own standards of rigour which, when followed, ensure trustworthiness of the findings obtained.[23] The application of quantitative methods is not a guarantee in any case that the study findings are sound as, in common with qualitative methods, there are a number of potential limitations of quantitative methods that researchers must address when conducting program outcome evaluations.[24] In any event, Richards and Bartels[25] have suggested that
it is worth considering how important the principles of ‘generalisability’ and ‘representativeness’ really are, and whether qualitative research may have something to offer that quantitative research is unable to achieve, namely, adding a richness to the data that cannot be obtained by looking at numbers alone.
Researchers are increasingly using mixed methods research designs that incorporate both quantitative and qualitative methods, in order to overcome the limitations of relying on a single method and to capitalise on the unique advantages offered by both. Research incorporating both quantitative and qualitative methods is able to produce results that may be ‘generalised to the wider population’ and also ‘rich in meaning’.[26]
The overarching evidence-based model that guides current interventions and programs for offenders is the Risk-Needs-Responsivity (‘RNR’) model. This model has had considerable impact on correctional agencies, including juvenile justice agencies, in many Western developed nations,[27] including various jurisdictions across Australia.[28] The RNR model, first proposed by Andrews, Bonta and Hoge in 1990, is a psychological approach to working with offenders that recognises the important role played by both individual psychological factors and social and contextual factors in criminal behaviour. [29]
As outlined by Andrews and Bonta,[30] the three major principles of the RNR model are as follows:
• Risk principle – this determines who should be treated and proposes that high intensity services should be reserved for the high-risk offenders and low-intensity services should be delivered to low-risk offenders. Determination of offender risk should be based on a reliable and valid assessment of risk, preferably a structured actuarial risk tool that incorporates both static and dynamic variables.
• Need principle – this determines what should be treated and proposes that treatments should target criminogenic needs. Criminogenic needs are psychological and psychosocial variables that facilitate engagement in criminal behaviour but are amenable to change (eg, pro-criminal attitudes, antisocial personality, pro-criminal associates, education, employment achievements, family/marital relationships, substance abuse, lack of pro-social pursuits).
• Responsivity principle – this determines how to conduct treatment and proposes that the style and mode of treatment must match the offender’s learning style and ability. In particular, treatments should be tailored to match strengths, ability, motivation, personality, and demographic characteristics such as gender, ethnicity, and age.
Applying the RNR model to what we know about juvenile offenders, it is clear that specific interventions are needed to address the unique risks, needs and responsivity levels of young offenders. With respect to responsivity, it is well recognised that young offenders differ from adults in their decision-making capabilities, thinking styles, vulnerabilities and neuro-biological capabilities.[31] It also appears that there are some unique criminogenic needs specific to young offenders. A 2001 meta-analysis conducted by Cottle, Lee and Heilbrun[32] of 23 studies totalling more than 15000 juveniles revealed five broad categories of variables associated with risk for re-offending in juvenile offenders. The strongest broad predictor of recidivism was offence history. However, the other four broad categories of variables were dynamic in nature and indicative of criminogenic needs. These dynamic variables were:
• family and social factors (eg, significant family problems, ineffective use of leisure time, delinquent peers);
• educational factors (achievement test scores and educational performance);
• substance use history; and
• clinical/psychological difficulties (eg, non-severe mental health problems and conduct problems).
This discussion of the dominant model in the design of correctional programs is important in the context of this article because it suggests other factors that may be useful in evaluating program effectiveness. Although recidivism rates are typically used as the outcome measure for evaluations of program performance, this article argues that incorporating other measures in evaluation studies may be more appropriate and meaningful than using recidivism as a standalone measure of success.
It would be prudent at this point to emphasise that although the RNR model is currently the dominant model guiding criminal justice interventions, it is not the only model. A complementary model that is gaining increasing support and recognition in offender rehabilitation is the Good Lives Model (‘GLM’).[33] In applying the GLM to risk management, Ward[34] proposes that offender risk is best managed and reduced by focusing on the internal and external conditions necessary to achieve an individual’s ultimate goals/aims in life (ie, human goods). To be clear then, the focus of rehabilitation is on the individual’s ultimate end goals (desired human goods) as opposed to reducing risk through addressing criminogenic needs. Ward and colleagues argue that all humans are goal-driven and are motivated to achieve primary human goods, that is, desired states, characteristics and experiences that ultimately enhance wellbeing.[35] Ward[36] identified nine broad and basic human goods: life (eg, good health), knowledge, mastery in areas such as occupational and leisure/play pursuits, agency (ie, autonomy and self-directedness), inner peace (ie, an internal state free of stress and negative emotions), friendship (ie, intimate/romantic, familial and other friendship based relationships) and community, spirituality (ie, a sense of meaning and purpose in life), happiness and creativity. The level of importance placed on each of these human goods by the individual reflects their personal identity, that is, the type of person they would like to be and the type of life they would like to live.[37]
It has been proposed that the GLM is a more positive and strengths-based approach to offender rehabilitation and is more flexible and individualised than the RNR approach to offender treatment. It has also been argued that the GLM will ultimately be more appealing to offenders and more likely to lead to positive change as it highlights for offenders alternative ways of achieving desired human goods in more socially acceptable and fulfilling ways. In contrast, the RNR model focuses on risks and does not identify alternative ways of achieving human goods. Thus, Ward suggests that the RNR approach to risk management needs to be embedded within the GLM. This would entail formulating a good life plan that incorporates strategies to achieve desired primary human goods in socially acceptable ways. These strategies include seeking out internal and external conditions necessary to achieve secondary goods (ie, stepping stones to achieving primary goods such as joining a sports team to pursue primary goods of friendship and mastery of play) as well as addressing internal and external barriers (ie, criminogenic needs and risks). Hence, criminogenic needs are viewed as obstacles that block the attainment of primary human goods.[38]
In the context of this article then, the importance of the GLM in program outcome evaluation is in directing evaluators to a broader and more inclusive range of indicators of program success, such as positive achievements with respect to secondary goods, in addition to decreases in criminogenic needs and a reduction in recidivism. The GLM is not at odds with the RNR model in many respects but while both approaches may identify criminogenic needs, the GLM focuses on positive or protective factors. From the GLM perspective, positive steps a young person may take to improve their life, such as educational/vocational achievements, joining a sports team, obtaining a legitimate paying job, securing independent accommodation, and building a new network of friends from work and sporting groups (ie, secondary goods), may also arguably be indicators of program success.
The next section of the paper outlines alternative outcome measures by drawing on three examples of recent outcome evaluation studies which sought to determine the effectiveness of post-sentencing tertiary intervention programs for juvenile offenders. What is common to these outcome evaluation studies is the consideration of additional outcome measures besides recidivism which include criminogenic needs identified by Cottle, Lee and Heilbrun[39], as well as positive steps (ie, secondary goods) towards obtaining primary human goods as conceptualised by Ward and colleagues.[40]
Post-Release Support Programs (‘PRSPs’) for juvenile offenders who have been released from detention may be administered at any stage from immediately post-sentence through to detention and post-release.[41] These programs typically involve supervision and reporting, as well as interventions that may include community-based therapy, education and employment programs. Due to the variation in the nature of these programs, it is difficult to determine whether PRSPs are effective at reducing juvenile recidivism.[42] Nonetheless, it is clear that there is support for the effectiveness of certain common components of PRSPs, and insofar as PRSPs make use of these approaches, it can be said that they are informed by evidence-based approaches.
To date, there has been little by way of investment into PRSPs for young offenders in Australia and, as a result, the research literature pertaining to the effectiveness of such programs is scarce. In 2011, the ACT Human Rights Commission[43] presented a comprehensive report that outlined the need for greater investment in PRSPs for young offenders in Australia. Upon reviewing the available literature, the Commission suggested that PRSPs:
• need to begin prior to release;
• must include aftercare services and support, including outreach services; and
• should support young people to maintain relationships on the outside whilst incarcerated.
The New South Wales (‘NSW’) PRSP commenced in 2002 under the auspices of the Department of Juvenile Justice. It is a structured, voluntary 12-week intervention which focuses on addressing the economic, social and welfare needs of young offenders. The primary purpose of the program is to facilitate the re-integration of juvenile offenders in their communities following release from detention. Its stated goal is to reduce the incidence of recidivism.
Cunneen and Luke[44] reported on their findings of an evaluation study of the NSW PRSP, using a quasi-experimental matched-pairs design for juvenile offenders for the program period 2002–2005. At the time of the evaluation, there were 10 PRSPs operating throughout NSW. During the evaluation, it was observed that the PRSP providers would typically begin working with young offenders in the weeks preceding their release from detention. Although it was intended that the program be conducted over a 12-week period, the duration of the program was extended for a further 12 weeks in most cases.
The following discussion focuses on the qualitative methods used and the findings from qualitative analyses, particularly in relation to outcomes other than recidivism. Qualitative interviews were conducted with Department of Juvenile Justice staff and PRSP providers, as well as young offenders who had participated in the program. More than three quarters (78%) of the young offenders interviewed stated that the program had helped them to stop re-offending. Participants with positive responses to the program stated that, in meeting key outcome areas, the PRSP had reduced the likelihood of re-offending.
The key outcome areas addressed by the program were:
• securing finances (eg, Centrelink payments or paid employment), accommodation (eg, securing independent accommodation or negotiating accommodation with family members);
• education and/or employment (eg, enrolling in either formal or non-formal education programs, enrolling in vocational training programs or applying for jobs); and
• meeting legal obligations (eg, obtaining legal representation and assistance, appearing in court, reporting to probation/parole officers).
The participants felt that the program helped them to refrain from further offending through addressing their financial, accommodation, education/training, employment and legal needs. They in particular noted the important case management and co-ordination role taken by PRSP workers, who typically liaised with educational institutions, potential employers, Centrelink, property owners, legal authorities and legal services, as well as health professionals to ensure that the various needs and obligations of the young person concerned were being met. In many instances, the clients commented on the fact that PSRP workers not only typically arranged these appointments and meetings, but also assisted them further by reminding them of these appointments and physically driving them to these appointments where necessary. The emotional support offered by PRSP workers was also highly valued by the young offenders interviewed.
Despite these positive findings based on qualitative measures, the findings with respect to quantitative measures of recidivism rates were inconclusive. Accordingly, Cunneen and Luke concluded that research such as this ‘reconfirm[s] the need to consider a range of outcome measures when determining program impacts, and which include in particular, young people’s views’.[45]
This section focuses on the U-Turn pilot conducted in Tasmania.[46] U-Turn is an intervention for young people aged 15–20 years who either have a history of or are at risk of being involved in motor vehicle theft. U-Turn is guided by the National Motor Vehicle Theft Reduction Council’s (‘NMVTRC’) best practice model for intervening in the lives of young repeat offenders involved in motor vehicle theft.[47] The core component of U-Turn is a structured 10-week automotive training course that provides participants with skills in the areas of car maintenance and bodywork. Other components include:
• personal development;
• employment and further education;
• recreational activities;
• literacy and numeracy;
• road safety;
• case management; and
• post-course support.
U-Turn was piloted in Tasmania over a two-year period beginning in 2003. Under contract to Tasmania Police, Mission Australia provided the program during the pilot period. The evaluation period for U-Turn commenced in May 2003 and was completed in early 2005. During this time, eight courses were offered and 52 young people graduated from the program.
The local evaluation of the Tasmanian pilot focused on assessing outcomes pertaining to the following three key objectives for U-Turn:
• managing the program efficiently and effectively in line with Total Quality Management principles and best practice;
• bringing about a shift in the lives of recidivist young offenders and other program participants through behavioural change and life skills; and
• preventing recidivists from re-offending. [48]
A multi-method approach was employed to assess the extent of achievement in these areas, including literature reviews, interviews with participants and significant others, surveys of stakeholders, process-oriented interviews with project staff, on-site observations, and quantitative analysis of recidivism data and U-Turn service database records.[49]
Part of the evaluation focused on outcomes other than recidivism, including behavioural change and life skills (ie, the second key objective of U-Turn). In relation to this objective, the results of interviews with participants, significant others and program staff demonstrated that the program was having a positive outcome on the majority of the young participants.
It is perhaps not unexpected that positive changes were observed in the areas of workplace training and skills. However, all those interviewed or otherwise surveyed also observed that U-Turn was resulting in broader changes and improvements. These changes were occurring in areas such as anti-social behaviour, personal skills, self-esteem, social skills and self- and inter-personal awareness. Program participants, in particular, cited reductions in drug and alcohol abuse, loitering and other types of anti-social behaviour, changes in their attitudes towards car theft, changes in motivation and attitudes towards life more generally, better driving practices and improved personal relationships (for example, U-Turn staff acted as positive male role models and program participants reported spending less time with peers who were a negative social influence).
Positive outcomes such as these are not captured by the relatively un-nuanced approach of measuring recidivism rates alone. They highlight the importance of assessing alternative outcome measures and obtaining feedback from a range of stakeholders, including young people who have participated in the intervention. As Julian[50] stated:
Measuring the ‘success’ of programs such as U-Turn is a difficult task. One of the key learnings from this evaluation...was the need to acknowledge the value of qualitative data in providing evidence of success that may fall short of the ultimate desired outcome (such as a reduction in recidivism). In this case, the qualitative data provided rich evidence of a shift in the lives of the recidivist young offenders and other program participants through behavioural change and life skills.
It should be noted that preliminary findings from the quantitative analysis of official recidivism data (ie, re-arrests and re-convictions) indicated that the program was also having a positive impact with regards to reducing the frequency and severity of recidivism for the 52 participants who completed the full 10-week automotive training course.[51] More recent data indicate that the program ‘has made a significant difference to the lives of more than 450 young people and their families’.[52] The 46th cohort of participants graduated in 2013, with ‘the majority of participants who had committed multiple offences prior to the course reduc[ing] their offending significantly’, while previously high-rate recidivists reduced their offending by 31%.[53] Regrettably, notwithstanding these positive outcomes in relation to both the ‘usual’ recidivism basis and ‘alternative’ measures of effectiveness, the Tasmanian Government recently decided to discontinue funding for the program from June 2015, and instead focus on an older cohort.[54]
Panyappi is a culturally appropriate crime prevention program that targets young Indigenous people who have already come into contact with the criminal justice system. An evaluation was conducted from January to May 2004, when the program had been running for 12 months. An Indigenous cultural advisor was engaged during the process to ensure it was conducted in a way that was respectful of cultural protocols and practices.[55] Panyappi aims to bring about far-reaching positive change above and beyond engagement in crime. The aims of the project are to:
• intervene in pathways of offending behaviour and bring about a positive shift in each young person’s attitude toward offending and in their behaviour;
• decrease each young participant’s contact with the juvenile justice system and/or agencies associated with this system;
• promote self-discovery and self-determination by young people participating in the program, their family and wider community; and
• work collaboratively with all agencies that have mutual responsibility for resolving the young person’s difficulties.
The first and third aims highlight once again the importance of assessing alternative outcomes. To explore achievements against the stated aims, the evaluation involved a combination of both qualitative and quantitative data. The qualitative data came from guided interviews and focus groups with the young people in the program, as well as their families, Panyappi program staff and other key stakeholders. Quantitative data were derived from an analysis of program statistics and client demographics.
In relation to the first aim, qualitative data suggested that the program was successful in changing young people’s attitudes towards offending in the short-term. As the author of the evaluation report noted, ‘[a]long with family members, [mentors and program collaborators] also indicated that young people’s attitudes had started to shift as they realised they could succeed in other areas and had other options they could take apart from offending’.[56]
The following quote from a Panyappi mentor exemplifies the sorts of changes in broader attitudes that occurred in young people involved in the program:
She wants to do as much as she can while I’m with her to put her on a different track ... I think [it’s] the fact that there is someone out there that is interested in what they are doing and just wanting to be there and help them and participate in what they want to do, so they feel valued and important.[57]
In relation to the third aim of the program, discussions with clients and other key stakeholders indicated that the program was having a positive impact on self-belief and personal and cultural identity. In addition, the qualitative data revealed that Panyappi was having a positive effect on other important areas, including educational achievement and re-engagement, family relationships and levels of stress within the family. This is therefore another example of an evaluation uncovering outcomes of a juvenile justice support program which can be said to be positive in themselves, irrespective of their implications for rates of recidivism.
With respect to the aim of reducing recidivism, follow-up data on rates of re-offending were obtained for 15 young people who were accepted into the program from March 2003 onwards and these participants were followed up until May 2004.[58] Recidivism was defined as offences that resulted in any of the following: convictions, custodial or non-custodial orders, referrals to family conferences, and/or formal cautions by police. Reductions in recidivism, as well as other qualitative improvements in relationships were observed over the duration of the evaluation period. In particular, 12 of the 15 young people who participated in Panyappi reduced their rate of offending by 25% or more while involved in the program. Five young people (33%) did not commit any new offences in the follow-up period. These reductions in offending were considered to be substantial improvements given the high rates of offending for many program participants prior to their involvement in Panyappi.
In gauging feedback from clients and other stakeholders, these case studies reflect best practice standards developed by the Australasian Juvenile Justice Administrators[59] with respect to outcome evaluations of juvenile justice support programs. Moreover, the studies clearly and consistently highlight the importance of assessing a range of outcomes, not solely changes in recidivism rates, when determining the effectiveness of a program. The nature of the alternative outcomes measured will naturally vary depending on the overall purpose of the program. For instance, promoting self-discovery and self-determination is a highly appropriate and important outcome for the Panyappi program given that it targets young Indigenous people. It is also clearly appropriate to gauge achievements in areas relating to finances, accommodation and legal needs in the case of the PRSP.
However, these differences aside, many of the alternative outcomes gauged in the three evaluation studies reviewed in this article clearly relate to criminogenic risk factors. Criminogenic risk factors are associated with risk for recidivism and thus logically ought to be addressed in any tertiary crime prevention program for young offenders. We note that improvements in relation to education participation, substance use and so on are often seen as intermediate outcomes on the path towards the ultimate goal of reduced recidivism. However, we suggest that such improvements should be seen as goals in themselves. Furthermore, many of the program outcomes from the three examples reviewed, including achievements with respect to employment, education, and relationships, could be conceptualised as achievements pertaining to secondary goods under the GLM[60] and thus would constitute important changes that would enable the young people concerned to live good lives and, in so doing, reduce the risk of recidivism.
Based on the major criminogenic risk factors identified by Andrews and Bonta,[61] as well as the criminogenic risk factors for recidivism in juvenile offenders identified by Cottle, Lee and Heilbrun,[62] the following variables in particular are proposed as additional outcome measures for juvenile justice support programs:
1. family and social factors (eg, family functioning, leisure/recreation activities that are pro-social, associating with delinquent peers);
2. social achievement (educational and employment);
3. substance abuse;
4. clinical/psychological difficulties (eg, mental health difficulties and conduct problems); and
5. pro-criminal attitudes (eg, thoughts, beliefs and values supportive of criminal behaviour).
Although the three evaluation studies discussed here gauged outcomes relating to family and social factors, social achievement and pro-criminal attitudes, there has been relatively little emphasis on substance abuse and other clinical/psychological difficulties. Furthermore, all three of the evaluation studies adopted qualitative methods to gauge outcomes in these areas. It would be highly desirable to explore the possibility of supplementing qualitative outcome measures with quantitative measures of program outcomes in these areas. One possibility with respect to quantitative measures of these factors would be to employ a structured risk assessment tool for predicting recidivism in juveniles, such as the Australian adaptation of the Youth Level of Service/Case Management Inventory (YLS/CMI-AA).[63] The advantage of using the YLS/CMI-AA is that the instrument provides an empirically validated measure of these key domains. In particular, the YLS/CMI-AA contains items measuring the following eight domains: prior and current offences, family and living circumstances, education/employment, peer relations, substance abuse, leisure/recreation, personality/behaviour, and attitudes/ orientation.[64] This suggestion of using a structured assessment such as the YLS/CMI-AA to gauge program success with respect to criminogenic needs is supported by a recent study which found that the percentage change in scores for seven of the 10 domains on Andrews and Bonta’s Level of Service Inventory-Revised (LSI-R) significantly predicted rearrest.[65] In making the case for the inclusion of a broader range of program outcomes, we do not suggest that there is no place for quantitative assessments of reductions in recidivism. Instead, we are calling for a more comprehensive, sensitive and nuanced consideration of ‘what works’ with young offenders.
In 2012, Save the Children Australia commissioned a review of the research literature on high-quality juvenile justice programs that reduce recidivism and also assist young offenders to re-engage with community life, education and employment. This paper was drawn from that review. The views expressed here are those of the authors and do not necessarily reflect the views of Save the Children Australia.
[∗] Caroline Spiranovic, Honorary Research Fellow, Faculty of Law, University of Western Australia and School of Population Health, Faculty of Medicine, Dentistry and Health Sciences, University of Western Australia; Research Fellow, Law School, University of Tasmania..
[∗∗] Helen Cockburn, Lecturer, Law School, University of Tasmania.
[∗∗∗] Lorana Bartels, Associate Professor, School of Law & Justice, University of Canberra. Honorary Associate Professor, Faculty of Law, University of Tasmania.
[∗∗∗∗] Roberta Julian, Associate Professor, Director Tasmanian Institute of Law Enforcement Studies, School of Social Sciences, University of Tasmania.
[1] Matthew Manning, ‘Establishing an Evidence Base – Program Evaluation’ in Anna Stewart, Troy Allard and Susan Dennison (eds), Evidence-based Policy and Practice in Youth Justice (Federation Press, 2011) 169.
[2] Joseph Wholey, Harry Hatry and Kathryn Newcomer, Handbook of Practical Program Evaluation (Jossey-Bass, 3rd ed, 2010).
[3] David Royse, Bruce Thyer and Deborah Padgett, Program Evaluation: An Introduction to an Evidence-Based Approach (Cengage Learning, 6th ed, 2015) 267.
[4] Australasian Juvenile Justice Administrators, Juvenile Justice Standards (2009).
[5] Department of Families, Housing, Community Services and Indigenous Affairs (Cth), Protecting Children is Everyone's Business: National Framework for Protecting Australia's Children 2009–2020 (2009).
[6] Standing Committee of Attorneys-General Working Group on Indigenous Justice (Cth), National Indigenous Law and Justice Framework (2010).
[7] Chris Cunneen and Garth Luke, ‘Recidivism and the Effectiveness of Criminal Justice Interventions: Juvenile Offenders and Post Release Support (2007) 19 Current Issues in Criminal Justice 197, 197.
[8] Ibid.
[9] Ibid. See also Jason Payne, ‘Recidivism in Australia: Findings and Future Research’ (Research and Public Policy Series No 80, Australian Institute of Criminology, 2007); Kelly Richards, ‘Measuring Juvenile Recidivism in Australia’ (Technical and Background Paper No 44, Australian Institute of Criminology, 2011).
[10] Cunneen and Luke, above n 7, 200.
[11] Payne, above n 9, Table 1, 34-37; Richards, above n 9, 6-7.
[12] Cunneen and Luke, above n 7.
[13] Richards, above n 9.
[14] Ibid.
[15] Payne, above n 9, 46-47.
[16] Ibid.
[17] Richards, above n 9.
[18] Payne, above n 9.
[19] Heith Copes and Mitchell Miller, ‘Preface’ in Heith Copes and Mitchell Miller (eds) The Routledge Handbook of Qualitative Criminology (Taylor and Francis, 2015).
[20] Susan Morrow ‘Quality and Trustworthiness in Qualitative Research in Counseling Psychology’ (2005) 52 Journal of Counseling Psychology 254.
[21] Kelly Richards and Lorana Bartels, ‘The Story Behind the Stories: Qualitative Criminology Research in Australia’ in Lorana Bartels and Kelly Richards (eds), Qualitative Criminology: Stories From the Field (Hawkins Press, 2011) 1, citing Richard Tewksebury, ‘Qualitative Versus Quantitative Methods: Understanding Why Qualitative Methods Are Superior for Criminology and Criminal Justice’ (2009) 1 Journal of Theoretical and Philosophical Criminology 38.
[22] Copes and Miller, above n 19; Morrow, above n 20.
[23] Morrow, above n 20.
[24] Gannaro Vito and George Higgins, Practical Program Evaluation for Criminal Justice (Routledge, 2015) 83–85.
[25] Richards and Bartels, above n 21, 1.
[26] Maggie Walter, ‘The Nature of Social Science Research’ in Maggie Walter (ed), Social Research Methods (Oxford University Press, 3rd ed, 2013) 3, 21.
[27] For discussion, see Don Andrews and James Bonta, ‘Rehabilitating Criminal Justice Policy and Practice’ (2010) 16 Psychology, Public Policy and Law 39.
[28] Andrew Day, Kevin Howells and Debra Rickwood ‘Current Trends in Rehabilitation of Juvenile Offenders’ [2004] (284) Trends and Issues in Crime and Criminal Justice 1.
[29] See Andrews and Bonta, above n 27, for discussion.
[30] Ibid.
[31] Chi Meng Chu and James Ogloff, ‘Sentencing of Adolescent Offenders in Victoria: A Review of Empirical Evidence and Practice’ (2011) 19 Psychiatry, Psychology and Law 325.
[32] Cindy Cottle, Ria Lee and Kirk Heilbrun, ‘The Prediction of Criminal Recidivism in Juveniles: A Meta-analysis’ (2001) 28 Criminal Justice and Behavior 367.
[33] Tony Ward, ‘The Management of Risk and the Design of Good Lives’ (2002) 37 Australian Psychologist 172–79; Tony Ward, Pamela Yates and Gwenda Willis, ‘The Good Lives Model and the Risk Need Responsivity Model: A Critical Response to Andrews, Bonta and Wormith (2011)’ (2012) 39 Criminal Justice and Behavior 94–110.
[34] Ward, above n 33.
[35] Ward, above n 33; Ward, Yates and Willis, above n 33.
[36] Ward, above n 33, 174.
[37] Ward, above n 33; Ward, Yates and Willis, above n 33.
[38] Ward, above n 33; Ward, Yates and Willis, above n 33.
[39] Cottle, Lee and Heilbrun, above n 32.
[40] Ward, above n 33; Ward, Yates and Willis, above n 33.
[41] Peter Murphy, Anthony McGinness and Tom McDermott, Review of Effective Practice in Juvenile Justice: Report for the Minister for Juvenile Justice (Noetic Solutions Pty Ltd, 2010).
[42] Ibid.
[43] Human Rights Commission (ACT), The ACT Youth Justice System 2011: A Report to the ACT Legislative Assembly ( 2011).
[44] Cunneen and Luke, above n 7.
[45] Ibid 209.
[46] Aynsley Kellow, Roberta Julian and Megan Alessandrini, Young Recidivist Car Theft Offender Program (U-Turn): Local Evaluation – Tasmania (Tasmanian Institute of Law Enforcement Studies, 2005) ; See also Roberta Julian, ‘Crossing Boundaries, Developing Trust: Qualitative Criminological Research Across Cultures and Disciplines’ in Lorana Bartels and Kelly Richards (eds), Qualitative Criminology: Stories From the Field (Hawkins Press, 2011) 116.
[47] See National Motor Vehicle Theft Reduction Council, Website <www.carsafe.com.au> for further discussion.
[48] Kellow, Julian and Alessandrini, above n 46, 2, citing Ann Sharley and Associates, Best Practice Model and Business Plan for a Young Recidivist Car Theft Offender Program (NMVTRC, 2002).
[49] Kellow, Julian and Alessandrini, .above n 46.
[50] Julian, above n 46, 124.
[51] Kellow, Julian and Alessandrini, above n 46.
[52] NMVTRC, ‘U-Turn Tasmania Update’ (2013) 49 Theft Torque 3, 3.
[53] Ibid.
[54] ABC News, Tasmanian Government Denies Axing Young Offenders Program as Mission Australia Fears Some Teens Will Fall Through the Cracks, (30 January 2015) <http://www.abc.net.au/news/2015-01-30/tasmanian-government-denies-axing-young-offenders-program/6057618> .
[55] Kathleen Stacey, Panyappi Indigenous Youth Mentoring Program: External Evaluation Report (South Australian Department of Human Services, 2004).
[56] Ibid 6.
[57] Ibid 42.
[58] Ibid.
[59] Australasian Juvenile Justice Administrators, above n 4.
[60] Ward, above n 33; Ward, Yates and Willis, above n 33.
[61] Andrews and Bonta, above n 27, 46.
[62] Cottle, Lee and Heilbrun, above n 32.
[63] Robert Hoge and Don Andrews, Australian Adaptation of the Youth Level of Service/Case Management Inventory (Multi-Health Systems, 1995), cited in Andrew McGrath and Anthony Thompson, ‘The Relative Predictive Validity of the Static and Dynamic Domain Scores in Risk-need Assessment of Juvenile Offenders’ (2012) 39 Criminal Justice and Behavior 250. See also Rachel Upperton and Anthony Thompson, ‘Predicting Juvenile Offender Recidivism: Risk-need Assessment and Juvenile Justice Officers’ (2007) 14 Psychiatry, Psychology and Law 138.
[64] Upperton and Thompson, above n 63.
[65] Ryan Labrecque, et al, ‘The Importance of Reassessment: How Changes in the LSI-R Risk Score Can Improve the Prediction of Recidivism’ (2014) 53 Journal of Offender Rehabilitation 116–128.
AustLII:
Copyright Policy
|
Disclaimers
|
Privacy Policy
|
Feedback
URL: http://www.austlii.edu.au/au/journals/VicULawJJl/2015/4.html