test FAQs - Frequently Asked Questions
General FAQs
1. What is Online Toolbox?
The Afterschool Outcome Measures Online Toolbox is an online data system designed to assess program quality and student outcomes of skill development and positive behavior, using scales that have well-established reliability and validity. The Toolbox includes three types of surveys: student self-reports (for students in grades 3 to 12), program staff reports of student outcomes, and classroom teacher reports of student outcomes in the areas of skill development and positive behavior. STEM-related beliefs and attitudes are also available.
Participating programs have the option to select which surveys to administer (one, two or all three surveys) depending on the types of reports they are interested in receiving: student self-reports, program staff reports, and classroom teacher reports. Surveys are administered at two points so changes in student behavior can be measured. For afterschool programs, surveys are administered in Fall/Winter (Time 1) and Spring (Time 2). For summer programs, surveys are administered at the beginning and end of the summer program. Surveys are appropriate for students grades 3 through 12.
Participating programs have the option to select which surveys to administer (one, two or all three surveys) depending on the types of reports they are interested in receiving: student self-reports, program staff reports, and classroom teacher reports. Surveys are administered at two points so changes in student behavior can be measured. For afterschool programs, surveys are administered in Fall/Winter (Time 1) and Spring (Time 2). For summer programs, surveys are administered at the beginning and end of the summer program. Surveys are appropriate for students grades 3 through 12.
2. Why are these outcome measures important?
We know that afterschool and summer learning programs can have beneficial effects on students’ skills, behaviors, and attitudes. The Online Toolbox can be a valuable resource for programs interested in capturing these effects on students. In addition, the measures of program experiences reflect program quality. Programs can use their program report of program quality and student outcomes for their own program improvement as well as for reports to stakeholders and funders.
3. Who is eligible to use the Online Toolbox?
Any program can use the Online Toolbox for a service fee. Interested programs can inquire with us at [email protected], or contact Andrea Karsh directly at949-824-9225.
We are happy to answer any questions you have about the Online Toolbox and discuss your program’s specific needs. If desired we can then provide a cost estimate, with no obligation, based on the number of program sites and students per site, the number of survey respondents (student self-reports, staff reports and classroom reports) and level of analysis desired.
We are happy to answer any questions you have about the Online Toolbox and discuss your program’s specific needs. If desired we can then provide a cost estimate, with no obligation, based on the number of program sites and students per site, the number of survey respondents (student self-reports, staff reports and classroom reports) and level of analysis desired.
4. If our program already collects survey data why would we want to use the Online Toolbox?
Even if you collect survey data from students, or other respondents, the value of these measures are that they are research-based measures that have been extensively field tested and shown to be a valid and reliable means of measuring student outcomes.
5. How do I select which surveys to administer?
You should select those surveys that best meet your program evaluation goals and capacity. Programs have the option to choose from three types of surveys: (1) Student self-reports, (2) Staff reports on students, and/or (3) Classroom Teacher reports on students. The more types of surveys administered the more informed you will be on the effects of your program, because you will have scores from different sources.
Classroom teacher surveys may require principal approval. See the Sample Principal Letter on the Survey Instructions webpage: Afterschool Instructions
Classroom teacher surveys may require principal approval. See the Sample Principal Letter on the Survey Instructions webpage: Afterschool Instructions
6. Can I select a sample of students participating in our program to complete surveys?
We strongly recommend that you do not select a sample of students from each participating site to ensure representative data and valid results. If you are unable to select all students attending the program at a site, then you should select a group of students such as all the students in one or more grade levels, or all the students in one or more classes or clubs at the site. If you have further questions about sampling contact us at [email protected].
7. When do I administer the surveys?
A pre and post survey window is discussed for each contract. It is shorter for summer programs and longer for year-round afterschool programs.
8. How long does it take to complete all the surveys?
Extensive field tests, involving thousands of students in grades 3 to 12, have determined that the Online Toolbox can be used easily and reliably. We found that students typically take less than 10 minutes to complete their full self-report. Teachers and staff typically require less than 5 minutes to complete one student report. Sites can administer surveys to students throughout the established pre and post survey windows. Staff and teachers can complete reports on students in several sessions during the survey window.
9. Can I view the survey questions?
We do not distribute the full measures to the public. A description of the measures with sample questions for each scale is available under the Reports and Documentation Tab on our website, or click on links below:
Description of Online Toolbox Measures [Core/CDE] (PDF)
Description of Online Toolbox Measures [STEM] (PDF)
Technical summaries of the scales included in the Online Toolbox are available in the Technical Appendix of The Forum for Youth Investment: From Soft Skills to Hard Data (January, 2014) [click on live links to access the documents].
Description of Online Toolbox Measures [Core/CDE] (PDF)
Description of Online Toolbox Measures [STEM] (PDF)
Technical summaries of the scales included in the Online Toolbox are available in the Technical Appendix of The Forum for Youth Investment: From Soft Skills to Hard Data (January, 2014) [click on live links to access the documents].
10. How are survey results for my program reported?
Each program using the Online Toolbox will receive a summary of student scores for each participating program site in addition to the aggregate scores for their program. These reports will be confidential and given to each program director. Reports are not made public. Reports are accessed through the Program Portal using the username and passcode assigned to the program director.
11. If more than one site in my program is participating, will my reports be disaggregated by site?
Scores for individual sites and for the program as a whole are included in a single program report. Please note that a minimum number of surveys (n=10) are needed per site to report site results.
12. Do you report results for individual survey respondents?
No. The reports do not identify individual students or provide individual responses or scores to ensure confidentiality. Instead, programs receive reports of their mean scores for each scale by program and site. These reports enable programs to compare their students’ skills, behaviors, and program experiences over time, and to compare their program and site-level outcome scores to aggregate scores for all participating programs.
Technical FAQs
1. If our program does not have computer and Internet access can we still use the Online Toolbox?
The Online Toolbox requires the use of a computer device that connects to the Internet. If you do not have access to a computer or computer lab at your program site, you may consider the use of smartphones, notepads, or laptops to administer surveys.
2. How do I get my Username and Passcode?
A username and passcode is assigned to the Program Director. The Program Director receives an email that his or her login information has been assigned. If you are not the Program Director but need to carry out any step in the process, then you will need to request the login information directly from the person in your program who received it. We cannot send login information to any other email or provide it over the phone in order to ensure the confidentiality of student information. If the original Program Director is no longer at your program, please contact [email protected] to notify us of this change.
3. I have submitted my Participant Data file. When will I get the individual codes to administer surveys?
Survey codes are generated between 5-7 working days from the date of submission of a correctly completed Participant Data file. The Program Director receives an email notification that the survey codes are available to download.
4. How do I get codes for new students, staff or teachers when I have already received codes?
Please add the names and other required information for those new respondents at the bottom of the most recent Survey Codes file(s) your program received and re-upload it through Program Portal using your program’s login information.