UNIVERSITY ASSESSMENT

DESIGNING INTERNAL DEPARTMENT RESEARCH PROCESSES
he outcomes assessment process looks like this:
Draw Sample at ACE	Data Collection System	Type of Evidence	Form of Data
Program Enrollment All first-time majors who enrolled in ACE department over the last five years.	Store Data within OIR Computer System	Number of students enrolled Demographic data of ACE student	Quantitative # % T-test F-test
Faculty Productivity All full-time and part-time faculty, graduate teaching assistants, and instructors who taught courses in ACE departments over the last three years	Store data within OIR Computer System Excel, SPSS, Lotus123, Adobe	Total number of credit hours taught by level of instructional faculty	-Quantitative -Qualitative
Draw Sample at ACE	General Education	Type of Outcome Discipline-specific	Noncognitive
Graduating Seniors: (a random sample) and all graduating ACE seniors	Science, engineering, logic, technology, history skills Writing skills Verbal / Oral Math/Computation Scientific /engineering/technological  outcomes:  Knowledge of field (cognitive and lab skills)	Depends on ACE Department	Civic responsibility Commitment to Excellence Moral development Intellectual integrity
All graduating ACE seniors		Scientific /engineering/technological  outcomes:  Knowledge of field (cognitive and lab skills)	Professional values and attitudes
ALL DATA STORED ON COMPUTER DISKS

UNIVERSITY ASSESSMENT

DATA ASSESSMENT STRATEGIES

INTERNAL DEPARTMENT ASSESSMENT STRATEGIES
During this stage of the academic program review, committee discussions take a long time.  A great deal of debate occurs when the discussion begins to focus on the indicators that will be used to assess the proposed outcome dimensions or constructs.  The table below shows the type of evidence, the assessment strategy used to collect the data, the indicators used as evidence, the estimated sample sizes, and the points of contact when the data would be collected.
Evidence	Assessment Strategy	Indicators	Sample Size	Point of Contact
Program Enrollment Faculty Productivity	Registrar's Records Office of Institutional Research	Number of students Enrolled Number of credit hours	All students All courses in the  departments at ACE	Admission Semester Census data
ACE General Education Outcomes
Science, engineering, logic, technology, history skills	Test/application to project	Scores on Test of Science, engineering, logic, technology, history skills	1,2	Month before Graduation
Writing skills	Test/applied project	Department Writing exam	1,2	Same
Oral presentation	Performance measure	Videotape or oral presentation of ACE senior project	1,2	Senior year
ACE Noncognitive Outcomes
Civil responsibility	Test/applied project	Checklist of ACE student Volunteer activities	1,2	Month before Graduation
Open-mindedness	Test/applied project	Locally developed survey questions	1,2	Same
Moral development	Inventory	Defining Issues Test	1,2	Same
Intellectual integrity	Test/applied project	Locally developed survey questions	1,2	Same
ACE Discipline-specific Outcomes
Knowledge of  departments at ACE of	Inventory	TABE	2	Month before graduation
Computer Lab  skills	Performance	Computer Lab output	3	Senior
Professional values and attitudes	Test/applied project	Survey questions developed by  departments at ACE faculty	2	Month before graduation
1  A random sample of #=n  non- departments at ACE graduating seniors
2  All graduating  departments at ACE majors (N= )
3  A random sample of 5-10  departments at ACE graduating seniors

UNIVERSITY ASSESSMENT

ACE DEPARTMENTAL ASSESSMENT PROJECTS

(DATA COLLECTION AND ANALYSIS STRATEGIES)

OIR DATA COLLECTION AND ANALYSIS OF ACE DEPARTMENTAL ASSESSMENT PROJECTS
Establish Departmental Research Process.  Isolate Goals of Program Review.  Initiate Program Review Process.  Once the program review is in place, the responsibility for the data collection and analysis is turned over to the Office of Institutional Research.  Locally developed instruments are pretested on small samples of students and revised.  Commercial instruments are purchased and administered during the last month before graduation. The department faculty identified key classes that all seniors are taking and declared an "assessment" day for the data collection effort.  Their cooperation also is instrumental in collecting evidence for a small sample of students who are doing portfolios or computer projects.  Institutional database files must be obtained.  The chart below summarizes the types of data collected and the analysis strategies used.
Variables	Data	Data Analysis	Summary Statistic
Program Enrollment Data	Headcount data by class	Descriptive Group comparison by Class level	Ns and %s Means and t-tests
Faculty Productivity	Number of credit hours	Descriptive Group comparison by Faculty level	Ns and %s Means and t-tests
ACE General Education Outcomes
Science, engineering, logic, technology, history skills	Scores on Test of Science, engineering, logic, technology, history skills	Descriptive Group comparison	Mean and S.D. Means, t-test
Writing skills	Test/applied project	Descriptive Group comparison	Percent passing Test of proportions
Departmental Writing exam	Graded exam score	Descriptive Group comparison	Percent passing Test of proportions
Oral presentation	Oral presentation of Senior project	Descriptive Group comparison	Percent passing Test of proportions
ACE Noncognitive Outcomes
Civic Responsibility	Checklist of student Volunteer activities	Descriptive Group comparison	Percent checked Test of proportions
Open-mindedness	Locally developed Survey items	Descriptive Group comparison	Percent checked Chi-square of item Responses
Moral development	Defining Issues Test	Descriptive Group comparison	Percent each stage Mean total score Test of means
Intellectual Integrity	Locally developed items	Descriptive Group comparison	Percent checked Chi-square of item responses
ACE Discipline-specific Outcomes
Science, engineering, logic, technology, history skills	College Board Achievement Test	Descriptive	Mean, S.D., and percent above a score of 600
Computer Lab skills	Computer output	Descriptive	None
Professional/community  values and Attitudes	Locally developed items by subject area faculty	Descriptive	Percent with Agree or Strongly Agree

UNIVERSITY ASSESSMENT

GENERAL ASSESSMENT MATRIX

	MEASURES OF ASSESSMENT PROCESS
ACE  AA, BA, MA, Ph.D., post Graduate Kmt SEBA Program	Cognitive Measures of ACE Students			Attitudinal Measures of ACE Students
Intended Outcomes	Local	ACE TEST	Licensure		Graduating ACE Student	ACE Alumni
Students Completing the Program will compare very favorably in their knowledge with those students completing a similar program nationally.
	DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS		DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS

UNIVERSITY ASSESSMENT

ASSESSMENT OUTCOMES

Program/Department:       Programs

Intended Outcome/Objective:  Students completing the AA, BA, MA, Ph.D., post Graduate Kmt SEBA program will compare favorably in their knowledge and skills with those students completing similar programs nationally.

Goal and Means of Assessment	What	When	Office Responsible	Type of Feedback	Use of Results
NATIONAL COMPARISION The average scores of the graduates of the AA, BA, MA, Ph.D., post Graduate Kmt SEBA program in  subject test (which they will be required to take shortly after graduation) will be at or near the 50th percentile compared to national results.
ACE STUDENT PERCEPTION 85% of the graduates of the  program will "Agree or "Strongly agree" with the statement "In my field I feel as well prepared as the majority of individuals nation wide who have completed a similar degree during the past year." COMPUTER LITERACY Use of Windows Vista-Based Programs
DATA TO BE STORED ON COMPUTER DISKS

UNIVERSITY ASSESSMENT

CHARACTERISTICS OF LOCALLY-DEVELOPED AND EXTERNALLY-DEVELOPED TESTS

Characteristics	Externally-developed tests	Locally-developed tests
Development time	None: Developed by company	Varies; depends on local testing practices, test development resources and expertise
Relationship between test and program objectives	Varies; test is tailored to broad-based needs	Close, tailored to local needs; adjusted as needed
Comparison groups	May include national, and regional norms; may include norms by gender, class level, college/ majors, institution type; infrequently updated	Created and maintained locally; generally no external norms; can be modified as needed
Costs	Usually high; materials & scoring costs may be reoccurring	Usually low; can be managed with limited reoccurring costs
Results	May be long delays; little choice in type of analyses	Can be immediate; local needs/ decisions drive analyses

UNIVERSITY ASSESSMENT

Enrollment Management Information Needs Matrix
	RECRUITMENT RETENTION
	Inquiry	Application	Enrollment	Persistence	Completion	Alumni
Performance Monitoring Indicators (PMIs)
Policy Research And Analysis
ALL DATA STORED ON COMPUTER DISKS

UNIVERSITY ASSESSMENT

FORMAT FOR IDENTIFYING EXISTING ASSESSMENT DATA AT ACE

For each study or data set, ACE Departments should identify the following:

What is it (brief description)?	Who has it (office/person)?	What population does it cover (e.g., freshmen, remedial students, etc.)?	When was it done (i.e., term, year, etc.)?	Special features or limitations
DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS

UNIVERSITY ASSESSMENT

GUIDE TO HANDLING DATA
Isolate Type of Data	Collect Data	Store Data at ACE	Retrieve Data
QUANTITATIVE DATA •·         Numbers Empirical facts Census Data Admissions Data IPEDS Data Numerical Reports OIR Fact Sheets Federal Statistics Financial Data	Paper files, Computer Data, Documents, Statistical Profiles Video/audio recorder HP Scanners Internet Fax Optical scan sheets Computer-assisted data Collection	Journals, logbooks, Interview transcriptions, or notes Magnetic tape, video tape, audio tape, laptops, CD, cassettes	Manual, Pentium V computers, text file retrieval, hypermedia, software programs: UNIVERS data files, SPSS 16.0, Excel, Lotus 1-2-3, Freelance Graphics, MS Publisher Vista, MS Word Manual or hypermedia Software retrieval: Transfer to CD Disk via HP CD-Maker.  Store CD's
QUALITATIVE DATA Words, Thoughts Verbal Expressions Actions Concerns Beliefs Opinions Pictures Graphic Illustrations	Paper files, reports, charts, manuals, tables, graphs Manual data entry via Pentium computers at ACE HP Scanners Internet Fax Optical scan sheets Computer-assisted data Collection	Rating/scoring sheets External Hard Drives and Memory sticks External Hard Drives and Memory sticks External Hard Drives and Memory sticks	Manual Database software structured query language statistical analysis (Lotus 1-2-3, Excel, and SPSS)  software Same Same

UNIVERSITY ASSESSMENT

PLACEMENT DATA

	READING	ENGLISH	MATH
# ACE Students Tested
# ACE Students Registered
# ACE Students tested needing developmental coursework
# ACE Students registered needing developmental coursework
# Number of ACE students taking developmental coursework

PLACEMENT DATA  (PERCENT)

Percent	Indicator
	Percent of ACE students tested needing at least one developmental course:
	Percent of ACE students registered taking at least one developmental course:
	Average of ACE students requiring at least one developmental course:

MEAN ACE PLACEMENT SCORES

ENGLISH
MATH
READING
DATA TO BE STORED ON COMPUTER DISKS

UNIVERSITY ASSESSMENT

Survey Implementation Guidelines

Survey completion time

Campus personnel will need approximately 10-15 minutes to complete a ACE Campus Survey.

Survey Handling

Please do not fold or staple the answer sheets.  They are computer-processed in ACE's Office of Institutional Research.

How to administer the survey

• 1. Administer the survey via campus mail, interview, interdepartmental mail, and computers in the future.

(Remember, for scoring purposes, the survey cannot be folded.)  If you choose this method to administer the survey, we suggest you return the inventory promptly, by a specific deadline, and that you plan for follow-up reminder mailings.  Also, you may want to reinforce deadlines with announcements during staff meetings or in staff newsletters.

2.   Contact department heads, directors, and deans     

• one-on-one and ask them to handle the distribution and collection of the surveys.
• Be sure to explain (and ask them to relay) why completing the survey is important.

• 3. Use a designated staff or faculty meeting.
• Surveys can be completed during the meeting or returned later.

How to select your sample ACE

If at all possible, most institutions will want to survey 100 percent of its

 faculty, staff, administrators, and board members. 

If that is not possible, ACE will want to survey at least 25-50 percent of its campus personnel.  The sample should represent each personnel segment's relative size. 

Also, you may want to over sample faculty and other personnel segments that interact with students on a regular basis.  Groups to consider surveying include:

• Faculty representing all majors, programs, and departments

• President and vice presidents

• Deans and directors

• Professional staff

• Board members

• Advising/counseling staff

• Student services staff

• Library staff

• Student records staff

• Business office staff

• Bookstore/student center staff
• Community
• Businesses
• Graduates
• Potential Employers
• Students and others

UNIVERSITY ASSESSMENT

FACULTY RESOURCE UTILIZATION MATRIX

Material	Activities	Costs
Overheads	Plan sequence Clearly write out or type masters; if typed, photocopy	Personnel costs associated with your time in planning the lesson/lecture Any costs associated with typing or photocopying Cost of overhead
Notes	Plan content Type masters Duplicate	Planning time Typing costs and time Duplication costs of time paper and ink
Workbooks	Plan content Type masters Duplicate	Planning time Typing cost and time Duplication costs of time, paper and ink
Video	Plan content and script Shoot Edit Prepare support materials such as notes to go with video	Planning time and writing and checking content If simple: time, cost of tape If complex: cost of crew talent and edit time As with print, time typing, layout and printing costs
Computer Simulation	Preparation Select application Develop script and enter materials Conduct Beta test  Develop Support materials	Plan the simulation Assemble the information Write the simulation in a way that can be translated to computer Develop the simulation Test and evaluate the simulation As with video, there may be costs of developing and producing support materials
STORE DATA ON COMPUTER DISKS IF POSSIBLE	STORE DATA ON COMPUTER DISKS IF POSSIBLE	STORE DATA ON COMPUTER DISKS IF POSSIBLE

UNIVERSITY ASSESSMENT

Systematic Tracking of ACE Graduate's Employment Status

Total Number of Graduates Number of respondents Percent responding Number of respondents employed Percent of respondent employed		Number of respondents attending college % of respondents attending college
Graduation Year Number of Graduates Number of respondents Percent responding Number of respondents employed Percent of respondents employed		Number of respondents attending college % of respondents attending college
Graduation Year Number of Graduates Number of respondents Percent responding Number of respondents employed Percent of respondents employed		Number of respondents attending college % of respondents attending college
Graduation Year Number of Graduates Number of respondents Percent responding Number of respondents employed Percent of respondents employed		Number of respondents attending college % of respondents attending college
Graduation Year Number of Graduates Number of respondents Percent responding Number of respondents employed Percent of respondents employed		Number of respondents attending college % of respondents attending college
Graduation Year Number of Graduates Number of respondents Percent responding Number of respondents employed Percent of respondents employed		Number of respondents attending college % of respondents attending college

UNIVERSITY ASSESSMENT

RECRUITMENT SYSTEM

ADMISSIONS OFFICE

RECRUITMENT MATRIX

	RECRUITMENT  INDICATORS
PROCESS	NUMBER	Performance Monitoring Indicators (PMIs)
		Website kits
		Website applicant
Inquiry		Number of mail and phone inquiries
Application		Number of applications received
		Number of acceptances offered
		Percent of applicants offered admission
		Number of accepted applicants enrolling
Enrollment		Percent of accepted applicants enrolling
		Full-time equivalent enrollment
		Student average courseload
		Number of full-and part-time first-time freshmen
		Number of new transfer students
		Number of exceptional and conditional admits
		Number, type, and amount of financial aid awards
		Share of local area residents/high school grads enrolling
		Racial/ethnic composition of entering students
		Mean/distribution of ACE scores when applicable
		High school GPA/rank distributions of entering students
		Predicted freshman academic performance
		Number and percent of entering students needing remediation
		Distribution of enrollment by college and program
		Distribution of enrollment by class location and time
DATA STORED ON COMPUTER DISK	DATA STORED ON COMPUTER DISK	DATA STORED ON COMPUTER DISK

UNIVERSITY ASSESSMENT

STUDENT TRACKING

Data Elements for Student Tracking System
Student attributes at Entry
Subject ID number (SSN)	Admissions/placement test scores
Date of birth	Prior college attended/transfer credits
Gender	Financial aid need/award type and level
Race/ethnicity	Reason for attending/goal at this institution
Native language	Program of study
High school attended/class rank and GPA	Resident/commuter status
Student Progress Term-by-Term
Remediation attempted and completed	Program of study
Credit hours attempted and earned this term	Cumulative credits attempted and earned at end term
Term grade point average	Cumulative GPA at end of term
Academic standing	Degree or certificate awarded
Follow-up Indicators, Graduates and Leavers
Transfer/graduate school	Relationship of job to college program
Credits accepted/lost in transfer	Annual salary
Transfer/graduate school program	Employer/industry and location
Cumulative credits earned/GPA at transfer school	Employer ratings of student job preparation
Degree awarded/program of degree	Alumni association/fund donor
Employment status
DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS

UNIVERSITY ASSESSMENT

GENERAL DEPARTMENT ASSESSMENT PROCESS OUTLINE DEVELOPMENTAL EDUCATION LIBERAL ARTS COMPUTER INFORMATION SYSTEMS OFFICE INFORMATIONS SYSTEMS BUSINESS ADMINISTRATION
Program Objective	Outcome Criteria	Assessment Measure	ACE Population
Cognitive Knowledge	Students will be able to demonstrate mastery of basic knowledge gained in ACE curriculum	Several standardized test items on existing exams, portfolio, other objective measures	All students enrolled in the particular ACE course
Student Perceptions	Students understands goals and objectives of ACE course	15-item, in-class survey	Sample of students enrolled in ACE course
Faculty Perceptions	Faculty agree that goals and objectives of ACE course are being met	Focused dialogue, video, cassettes, computer disks, printed reports (1) student performance on tests (2) department chair's evaluation of classroom instruction (3) student evaluation of faculty	Particular department faculty (, DE)
DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS

UNIVERSITY ASSESSMENT

SAMPLE RESEARCH PROJECT
Steps to Developing a ACE Student Perception Study
Step 1	Gather data on ACE student demographics, retention, and program participation from institutional records.
Step 2	Survey a sample of all ACE students (or survey all students, if the student body is small enough), assessing student satisfaction, perceptions about the climate of the institution, attitudes about the institutional mission and their educational experience.
Step 3	Interview ACE students from diverse socio-economic backgrounds to broadly assess their experiences on campus and their perceptions of the campus climate.
Step 4	Gather ACE alumni responses through a general survey or through focused discussions with a small sample.
Step 5 Step 6 Step 7 Step 8 Step 9 Step 10	Interview selected ACE faculty and staff concerning their experiences of the climate on the campus (valuable but optional depending on the need). Collect/Process Data Store Data on Computer Disks Process Data through SPSS 16.0, Excel, Lotus 1-2-3 Write report in MS Word or MS Publisher Vista Publish Standard Report using ACE Desktop Publishing System

UNIVERSITY ASSESSMENT

STUDENT TRACKING

Data Elements for Student Tracking System
Student Attributes at Entry
Subject ID number (SSN)	Admissions/placement test scores
Date of birth	Prior college attended/transfer credits
Gender	Financial aid need/award type and level
Race/ethnicity	Reason for attending/goal at this institution
Native language	Program of study
High school attended/class rank and GPA	Resident/commuter status
Student Progress Term-by-Term
Remediation attempted and completed	Program of study
Credit hours attempted and earned term	Cumulative credits attempted and earned at end term
Term grade point average	Cumulative GPA at end of term
Academic standing	Degree or certificate awarded
Follow-up Indicators, Graduates and Leavers
Transfer/bachelor/baccalaureate school	Relationship of job to college program
Credits accepted/lost in transfer	Annual salary
Transfer/bachelor/baccalaureate school program	Employer/industry and location
Cumulative credits earned/GPA at transfer school	Employer ratings of student job preparation
Degree awarded/program of degree	Alumni association/fund donor
Employment status
DATA STORED ON COMPUTER DISKS	DATA STORED ON COMPUTER DISKS

UNIVERSITY ASSESSMENT

CONTENT AND CONSTRUCT VALIDITY

SIGNIFICANT TYPES OF VALIDITY
Type	Definition	ACE Test Validity Threatened by	Systematic Prevention & Checks
Content	Extent to which a test adequately samples ACE program objectives	Poor relationship between test items and ACE program objectives	(1) Review of test items frequently missed by students (2) Review of text items misunderstood by students
Construct	Extent to which a test measures the amount learned and not some other extraneous variable	For objective tests: Poorly constructed items that measure test-taking skill rather than mastery of material For other test formats: Problems with raters and scoring procedures	Use good item writing practices Development of good rating criteria: train for & check on raters' consistency Check correlation with other related & unrelated information

Note: Reliability seeks to ask and answer one central question---Is the degree to which test scores are free of measurement errors due to things like student fatigue item sampling, student guessing?

Objective tests are threatened by poor item construction, rater or scorer variability, and inconsistent application of scoring/ra