Website Evaluation Tool
Research


[ Research ]   [ Conferences ]   [ Journals ]   [ Newsletters ]   [ Workshops & Lectures ]  

SPAT has gone through extensive scientific testing to verify that it works. It has been tested with health care professionals and with laypersons. Everyone has learned from SPAT and found that after learning SPAT they constantly evaluate web pages.

Further research is being done with SPAT and the findings will be posted here when complete. Research in Process

Research

A funded research project: La Rue, E. M., Sereika, S., & Engberg, S. (2008). The validity and utility of a tool for evaluating web pages presenting health content ($6,602.20). Pittsburgh, PA: University of Pittsburgh Center for Social and Urban Research, Steven Manners Faculty Development Award. http://www.ucsur.pitt.edu/manners_2008.php

Abstract:
Health is one of the most popular subjects accessed on the World Wide Web. From the Internet people easily access health information and some of these people are turning into information empowered patients. While consumers (laypersons) are accessing information, they may not be accessing quality health information. Because the U.S. has a large population of consumers with health literacy levels that are less than proficient, there is a great need for education and tools to help consumers access quality health information.

There are many web page evaluation tools available but none have been validated; they are too time consuming to use and most require paper and pencil to complete a check list. One tool that does not require paper and pencil, is quick and easy to use, and has shown face validity is SPAT. SPAT is a mnemonic tool used by people to help them pre-evaluate web page health information for accuracy. Findings from a previous pilot study measuring the adoption and usability of the SPAT tool with certified diabetes educators (CDEs), revealed a statistically significant change in the way web pages were evaluated after learning and using SPAT.

To continue testing the validity of the SPAT tool, criterion-related validity needs to be measured. The proposed pretest posttest experimental design pilot study focuses on the training for using the SPAT tool and the application of the SPAT tool by consumers. We expect the results of the proposed study to assist in converting the paper based SPAT instructional sheet to a web-based format and to provide support for future work in examining the impact of SPAT has on self-care behaviors.

   Conferences

LaRue, E. M., Roemer, J., & Terhorst, L. (Jul 2010). Guiding and assisting patients to evaluate web page information: Building a partnership. Paper presented at the American Association of Diabetes Educators "37th Annual Meeting." San Antonio, TX.

Purpose:
This interactive session will describe the purpose of a study that sought to evaluate: the impact of the site, publisher, audience (bias, readability), and timeliness (SPAT) mnemonic web page assessment tool on the consumers' ability to accurately judge web page quality, the approach that consumers use to examine web pages across the SPAT components, and the criterion validity of SPAT tool. The research hypotheses, were that: knowledge of SPAT would assist a person to critically evaluate a web page before accepting the information; knowing the mnemonic tool would trigger the consumer to evaluate a web page for the source of the information and timeliness of the information; using SPAT to evaluate a web page would assist in the consumer in selecting more appropriate and reliable health information for themselves. Along with describing the actual study and the results, real-life case scenario's will be shared describing the practical use of SPAT in patient care, and participants will apply the SPAT mnemonic to real web pages to gain an understanding of it's application and possible use.

Brief Description/Methodology:
This single group pre- and posttest quasi-experimental design pilot study measured Consumer's use of the web page evaluation tool SPAT to evaluate the diabetes content of 25 preselected web pages as "good" or "bad." The 25 web pages represent the gold standard for content-criterion validity testing. The browser software, Clipmarks, was used to highlight and store selected elements of the web pages noted when evaluating the page. Consumers were introduced to the SPAT tool via verbal teaching and paper. After demonstrating competency with the mnemonic, the consumers re-evaluated the twenty-five gold standard web pages in reverse order from the initial evaluation.

Statistical and/or Analytical Methods Used:
A gold standard in which to measure the SPAT tool was established by having 2 Certified Diabetes Educators analyze 50 web pages presenting diabetes information from the Internet. They evaluated only the content presented on the web page, not aesthetic design or usability. Based on their diabetes subject expertise, the CDEs gave an overall rating of 'good' or 'bad,' to the quality of the information. In order to determine how closely subjects matched the CDE opinion before and after SPAT, pre- and post-SPAT responses were examined and coded as a match (1) or not a match (0). Matched responses were then summed across the 25 pre-tests and 25 post-tests to yield a number of agreements pre-SPAT and a number of agreements post-SPAT. Tests of normality indicated that the pre- and post-match variables were normally distributed, therefore a parametric dependent samples t-test was use to determine if the mean difference between pre- and post-matches was significant.
Reliability testing for SPAT was also performed in this study through the use of the web browser add-on software application Clipmarks?. The Clipmarks software permitted the consumer to "clip' anything on the web page that helped them to formulate judgment about the quality of the page. After making a quality decision about the page the consumer emailed the "clippings" to an email account for the study. Each email can contain text and any number of images in clipped sequence by the consumer. This method of measurement resulted in nearly 2,000 emails.

Results:
A significant improvement in the healthy consumer's ability to correctly assess the quality of the health content after knowledge of SPAT occurred. The mean number of agreements climbed from 14.9 before SPAT to 16.2 after SPAT.

Conslusions:
Participants in this study were able to independently use a tool to analyze the health information presented from the Internet. They used the SPAT criteria information sheet demonstrates ease of mnemonic application and a specific technique to help determine accuracy of information. The guidelines can be quickly and effectively used in a clinical environment. From this experience, the participants learned the importance of being skeptical of the health information presented from the Internet and that reinforced the importance of being a critical health information consumer. As a final result from this study, each participant felt more confident in his or her ability to accurately evaluate web-based information.


LaRue, E. M. (May 2009). The validity and utility of a tool for evaluating web pages presenting health content. Poster presented at the Medical Library Association Conference. "iFusions." Honolulu, HI.

Purpose:
The purpose of this study was to: Evaluate the impact of SPAT, a mnemonic web page assessment tool, on the consumers' ability to accurately to judge web page quality, the approach that consumers use to examine web pages across the SPAT components (site, publisher, audience [bias, readability] and timeliness); and to test the criterion validity of SPAT.

Setting/subjects:
Participants included 30 subjects over the age of 18, with the ability to speak and read English, acknowledge themselves as a weekly Internet user, who neither have diabetes nor have responsibility for directly caring for someone with diabetes.

Brief Description/Methodology:
This single group pre- and posttest quasi-experimental design pilot study measured consumers' use of the web page evaluation tool SPAT to evaluate the diabetes content of 25 pre-selected web pages as 'good' or 'bad.' The 25 web pages represent the gold standard for content-criterion validity testing. The browser software, Clipmarks, was used to highlight and store selected elements of the web pages noted when evaluating the page. Consumers were introduced to the SPAT tool via paper and verbally. After demonstrating competency with the mnemonic, they re-evaluated the 25 gold standard web pages in reverse order from the initial evaluation.

Results:
Thirty healthy consumers participated in this study. Most were between 18 to 23 years of age (n=10) and most had spent some years in college (n=11). Eight subjects had a Bachelor's in Science degree and three had Ph.D.s.
Schooling was significantly correlated with the usage of SPAT. Before learning the SPAT method, the consumer's choice did not match the gold standard often (r=.419, p=.026). After learning the SPAT method their choices on the quality of the web pages matched the gold standard significantly more (r=.492, p=.008).

Discussion:
To our knowledge, this is the first report of criterion validity and reliability testing for a consumer usable tool to evaluate webpage content. SPAT does make an impact on the consumer's ability to judge the quality of a web page.


LaRue, E. M. (May 2007). A Study on the Adoption of a Web Page Content Assessment Tool: SPAT. Paper presented at the Medical Library Association Conference. "Information Revolution: Change is in the Air." Philadelphia, PA.

[ View Study Presentation ]

Purpose:
This research study proposes to measure the acceptance of a web page assessment tool developed by this investigator. The purpose of this study was to validate the assessment tool's use in analyzing web page content quality and reliability. Setting/subjects:
Certified Diabetic Educators within a metropolitan and semi-rural setting

Brief Description/Methodology:
A case-controlled research design was used to collect data from a convenience sample of 38 Certified Diabetes Educators (CDEs). Each CDE completed a closed-question demographic questionnaire gathering data on their perceived level of skill in accessing information from the Web. To evaluate the effectiveness, and personal usefulness of SPAT, each subject was asked to review two sets of pre-selected freely available diabetes information web pages. After reviewing set one diabetes web pages, the CDE completed a semi-structured questionnaire querying their assessment process. Next, SPAT was verbally introduced by the investigator. Following the introduction of SPAT, set two diabetes web pages were reviewed by the CDE then a second semi-structured questionnaire was completed, thus finishing the formal interview. To complete the study on the adoption of SPAT, a follow-up email was sent approximately three weeks after the introduction of the tool.

Results:
There was no statistical difference found in the CDEs' evaluation of the URL, author, and date, pre-and-post introduction of SPAT. There was statistical significance for evaluating a web page's text and intended audience. While the differences in mean scores between the pre- and post-test were narrow, the near perfect effectiveness of SPAT manipulation is notable as proof that if one knows of the tool SPAT, and uses it they are evaluating a web page. With a 78% return rate for the follow-up questionnaires, 47% of the CDEs changed their evaluation process 'Somewhat' by knowing SPAT. Seven percent said their web page reviewing method changed 'extensively.' Overall, SPAT had made an impact on the way CDEs review web based information.

Discussion:
The findings from this study add to our understanding of the diffusion and adoption of information technology by CDEs and their utilization of the web page evaluation tool SPAT.


LaRue, E. M. (2006, May). Evaluate the Evidence: an Innovative Method for Web Page Evaluation. Poster presented at Annual Conference of the Medical Library Association, Phoenix, AZ.

[ View Poster ]

Purpose:
The purpose of this poster is to present the process of developing and then validating an instrument for the evaluation of web pages.

Background:
As more users rely on web-based health information there is a growing need to identify reliable information sources. Currently there are a few evaluation approaches but they require paper-based materials for note taking or utilize a subjective ranking system. In each case, these approaches require an extended period of time to assess the web pages and do not provide a validated measure of accuracy or quality.

Methods:
SPAT (Site Publisher Audience Text) has been developed to provide an assessment instrument that will establish the validity and quality of a health-related web page. Initial evaluations of SPAT have been conducted through student focus groups, and user task analysis. To complete the research a pre-and-post test design will be carried out with certified diabetes educators.

Results:
Preliminary results find SPAT to be memorable, functional, and reliably useable for all ages.

Discussion:
SPAT does not require note taking materials and does not have a ranking system for validity and quality. The SPAT instrument guides the user through a systematic process of web page assessment. If the user does what the acronym reminds them to do, they will have performed an inclusive assessment on the quality and validity of the web page.


LaRue, E. M. (2003). "Coming About" Consumer Health Instruction. Paper presented at the Annual Meeting of the Southern Chapter of the Medical Library Association, Coral Gables, FL.

[ View Coming About Presentation ]

Purpose:
This paper discusses how a library in a private institution can teach consumer health literacy without seeing the public.

Brief Description:
If health care practitioners know how to evaluate Web pages and use MedlinePlus, then they can teach their patients how to find accurate health information from a reliable Internet source. This paper will report how a librarian participated in developing curriculum, secured two hours of lecture time, and assisted in creating homework assignments for a new course offered by the School of Nursing.

Results:
The idea to teach Web page evaluation techniques, and consumer health literacy to nursing students was proposed to the School of Nursing when they were developing a new course on health information on the Internet. The vice dean for the School of Nursing contacted the library asking for a participant in establishing the curriculum for the new course, Web-based Health Information for Consumer Education. The librarian worked with two faculty members to develop the course outline and was able to secure time for a two-hour librarian lecture. In preparation for the lecture a Web page evaluation tool (SPAT) was developed.

Evaluation:
Following the lecture on the consumer use of health information from the Internet, health literacy, and the methodology of SPAT, a homework assignment was given to reinforce the lecture. Student's comments were very positive and indicated that they had, and will have, opportunities to use MedlinePlus and Web page evaluation techniques with patients during their clinical work. The faculty member leading the course and the faculty member attending the class both remarked favorably on the lecture and class exercise.

back to top

Journals

La Rue, E. M. (2011, May/June). Finding Health Information on the Internet. Diabetes Self- Management, 28, 24-27.

[ Read Article ]

La Rue, E. M., Mitchell, A. M., & Gonzalez, K. (2009). Reach Out! : A social community web site evaluated by SPAT and DISCERN. Journal of Consumer Health On the Internet, 13(1), 77-79.

[ Purchase Article ]

La Rue, E. M. (2008). Development and Evaluation of SPAT: A Web page Assessment Tool. Library Hi-Tech, 26(2).

[ Purchase Article ]
Newsletters

LaRue, E. M. (2006). A study on the adoption of a Web page content assessment tool: SPAT. University of Pittsburgh.

[ View Complete Newsletter ]  [ View Article Only ]
Workshops & Lectures

LaRue, E. M. (2011, October). Serious Searching and Evaluating Web Pages. Presentation for Pittsburgh Connects Homewook-Brushton computer center.

LaRue, E. M. (2011, October). Serious Searching and Evaluating Web Pages. Presentation for Pittsburgh Connects Bloomfield Garfield Corporation computer center.

LaRue, E. M. (2011, April). Serious Searching. Presentation given at a continuing education workshop for the Pittsburgh Public School System.

LaRue, E. M. (2011, January). Save Time: Find Only Quality Information. Presentation given at the Oakland Based Diabetes Support Group. Pittsburgh, PA.

LaRue, E. M. (2010, May). Understanding Electronic Health Literacy. Workshop presented at the Factuly Development Program - ELITE: Emerging Learning & Integrated Technologies Education, University of Pittsburgh, School of Nursing, Pittsburgh, PA.

LaRue, E. M. (2010, April). Health Literacy 2010. Workshop presented at UPMC St. Margaret's Hospital Nursing Ground Rounds. Pittsburgh, PA.

LaRue, E. M. (2009, October). Patient Care: Attention to health literacy. Presentation given at the Western Pennsylvania Chapter of HIMSS (Healthcare Information and Management Systems Society) 3rd Annual Healthcare Trade Faire & Regional Conference, Cranberry, PA.

LaRue, E. M. (2003, May). SPAT that web page. Workshop presented at the Kern County School District, Lamont, CA.

[ View Workshop Presentation ]