employs formative, process and summative evaluation methods in its
annual National Accreditation Process.
- Formative evaluation is employed when a
state or local program is “forming” as a means of
shaping the program to the JAG Model
and organizational best practices.
Formative evaluation is used in the establishment of the state
organization and local programs.
- Process evaluation is conducted annually
by JAG but on a continuous basis at the state and local level
to describe and measure if the JAG Model has been properly implemented.
The process evaluation focuses on the following questions:
- Who are the students selected to receive
the services provided by a JAG Model program?
- What services are provided as well as:
when, how, frequency, and in what settings?
- What interactions are occurring between
program staff and stakeholders?
- What are the management strategies and
- What are the costs of a JAG Model program?
- How satisfied are stakeholders with the
program?students, employers, parents, administrators, etc.?
- How effective is the JAG staff in implementing
the JAG Model?
- How are staff members trained?
- What are the roles and responsibilities?
Is staff satisfied with their role?
- What are the program costs? Has there
been a change in funding? What about future funding?
- Summative evaluation is conducted based
upon the measurable outcomes tracked using the Electronic National
Data Management System (e-NDMS). Among the most important summative
data captured by student, school, site, and state, we are able
to answer the following questions at the close of the 12-month
Outcomes for Graduates:
- What is the graduation rate?
- What is the employment rate?
- What is the positive outcome rate?
- What is the full-time jobs rate?
- What is the full-time placement rate?
- What is the average wage paid to employed
- What is the rate of graduates attending a
Program Outcomes for Non-Seniors:
- What improvement was achieved in daily attendance?
- What improvement was achieved in grade point
- What improvement was achieved in passing
- What improvement was achieved in reducing
- What improvement was achieved in enhancing
their image with others in the school?
- What improvement was achieved in improving
basic skills, i.e. math and reading?
- What improvement was achieved in participation
in the Career Association?
- What improvement was achieved in the staying
in school status?
The primary purpose of the National Accreditation
Process is to provide third-party feedback to state and local affiliates
as to how well they have implemented the JAG Model and operated
an effective statewide organization.
State affiliates may request third-party feedback
to identify major or minor deficiencies in implementing the JAG
Model. JAG views the National Accreditation Process as the first
step in a program improvement initiative. State or local oversight
bodies can request the following services:
Review - This review is conducted in the initial program
year of a new state affiliate. The review is designed to provide
feedback to the oversight body, management team, and staff without
impacting the accreditation status of the state affiliate. The
Mid-Year Review is usually followed by additional technical assistance
to overcome deficiencies.
Site Review - A targeted review is for a designated site
or schools. The review is usually requested by the Management
Team as a means of improving performance in designated schools
prior to a comprehensive site review. The site reviewer(s) will
conduct an exit interview to explain the deficiencies that must
be addressed before the National Accreditation Process is implemented.
Site Review - This is the traditional site review in which
the state organization and a sample of local programs/schools
will be reviewed in addition to a thorough review of the data
contained in the Electronic National Data Management System. The
Comprehensive Site Review is the initial step in the National
Accreditation Process that concludes with the awarding of standard,
provisional or probationary accreditation. JAG will work closely
with the oversight body and/or the Management Team to deliver
consulting and/or training to upgrade the state's accreditation
Review - The state affiliate's oversight body can request
a management review to provide feedback to the State Director
and/or Management Team to improve personal and programmatic performance.
JAG is willing to conduct a 360-degree assessment for one or more
members of the Management Team. The 360-degree feedback will be
used to develop a Professional Development Plan to guide the continuous
improvement of the targeted members of the Management Team.
The critical ingredients in raising JAG Model
Programs to a higher level of results is essential for any school-based
or publicly funded program. Being successful and staying successful
is much more challenging today than ever before. The six ingredients
for higher results are:
- Genuine focus on at-risk youth - The JAG
- Data-driven management (e-NDMS.) Use data
to reduce failures (defects.)
- Core processes that are required to deliver
JAG Model services and extraordinary outcomes! Continuous improvement!
- Proactive management! Eliminate “quick
- Boundaryless collaboration! Smash barriers
that block the flow of new ideas! Eliminate duplicity and waste!
- Drive for perfection! Tolerate failure!
DATA COLLECTION METHODS
State and local affiliates are evaluated on
the basis of their ability to operate an organization that is efficient
and effective in implementing JAG Model Program Applications. The
JAG Model components are the evaluation categories for which data
is captured from the following sources:
- On-site Interviews by the Site Review Team
- Electronic National Data Management
System - The Internet-based data system provides JAG the online
capability to monitor the following:
- Students selected for the program
- Services delivered to program participants
- Outcomes achieved by program participants
during the follow-up period
- Destination Validation Telephone Surveys?a
telephone survey is conducted with employers to determine their
satisfaction of the services delivered by Job Specialists
- State Program Handbooks, Policy Manuals,
Brochures, Videos, etc. JAG reviews these documents to evaluate
the extent to which the state and local affiliates “walk
the talk.” There must be alignment between program literature
and the services delivered.
- Participation in National Network Professional
Development Opportunities. JAG monitors those that participate
in national training sessions/events. The training sessions that
are delivered at national training sessions/events are based on
the evaluative feedback provided by the site reviewers and the
overall National Accreditation Report.
NATIONAL ACCREDITATION REPORT OUTLINE
Based on the data collected, the National Accreditation
Report documents deficiencies in implementing the JAG Model. JAG
is an organization in which affiliates and staff are held strictly
accountable for process, services and outcomes. The report serves
as an evaluation tool that includes the following:
- Overview - overall evaluation statements
- Affiliate Information - basic organizational
- Program Demographics - who is served
- Program Outcomes - what is achieved
- Outstanding Performance and Outcomes - what
exceeds the JAG Model
- Major Areas of Non-Compliance - what must
- Minor Areas of Non-Compliance - what should
- Model Enhancements - best practices to be
shared with the National Network
- Observations and Recommendations - prescription
to improve performance
- Accreditation Status - awarded based upon
measurable inputs and outcomes
- Action Plan - the deadline for submitting
an action plan to overcome deficiencies
The state affiliate is requested to submit
an Action Plan in response to the National Accreditation Report.
The Action Plan is reviewed to determine if deficiencies will be
overcome, process improved, and/or outcomes improved. If the Action
Plan is inadequate, JAG will schedule a face-to-face meeting with
the Management Team to develop an Action Plan that will result in
the intended areas of improvement. JAG monitors the implementation
of the Action Plan until the performance areas are improved. A variety
of services are delivered to state and local affiliates that are
unable to improve their process and/or outcomes including: on-site
technical assistance, visits to high performing states/locales,
participation in the National Training Seminar, telephone consultation,
to name a few.