1Title

Increasing Relevance of Smoking Cessation Messages
in an Online Software Agent Environment

Todd Shimoda, Linda Stapel
Colorado State University

todd.shimoda@colostate.edu

2 What is the problem we address?

smokers trying to quit need and seek relevant advice

current computer-mediated message systems (typically online or kiosk) delivering smoking cessation messages are limited in delivering the most relevant advice, compared with human counselors

3 What do we mean by relevance?

Grice (maxims of communication):
relevance is an expectation that something spoken will guide the hearer toward the speaker's meaning

Wilson and Sperber (building on Grice):
an input (a sight, a sound, an utterance, a memory) is relevant to an individual when it connects with background information he or she has available to yield conclusions that matter to him or her

Schamber (information science):
the fundamental criterion for evaluating the effectiveness of information retrieval and use, relevancy is a bundle of many types of cognitive judgments and other evaluative criteria:
   --       content
   --       importance
   --       credibility
   --       comprehensiveness
   --       specificity

Brouard & Nie (neural cognitive model):
    --       adaptive resonance theory is a natural process of information retrieval using two-layer neural networks
    --       inputs into the network attempt to associate and reinforce stimulus representations and internal representations
    --       computation is measure of strength of the assertion "if Node A is activated then Node B is also activated."
    --       with high strength, the network locks into a resonant state

4 How do we achieve relevance?

learning something about the user for a start

determine the user’s current stage of smoking behavior,  i.e., Stages of Change Theory (Transtheoretical Model) and processes that help predict movement from stage to stage

Pre-contemplation (not planning on quitting):
    --       Consciousness raising (information and knowledge)
    --       Dramatic relief (role playing)
    --       Environmental reevaluation (how problem affects physical environment)

Contemplation (planning on quitting in the near future):
    --       Self-reevaluation (assessing one’s feelings regarding behavior)

Preparation (preparing to quit within one month):
    --       Self-liberation (commitment or belief in ability to change)

Action (in the process of quitting for less than six months):
    --       Reinforcement management (overt and covert rewards)
    --       Helping relationships (social support, self-help groups)
    --       Counter-conditioning (alternatives for behavior)
    --       Stimulus control (avoid high-risk cues)

Maintenance (has quit for more than six months)

Termination (behavior change permanently adopted)

5 What are dimensions of interpersonal communication?

Combining theory use with interpersonal communication allows the highest possible relevance
primarily, interactivity and social presence

according to Heeter, interactivity’s six dimensions:
    --       complexity of choices available
    --       effort required of users
    --       responsiveness to user
    --       monitoring information for the user
    --       ease of adding information
    --       facilitation of interpersonal communication

increasing interactivity in computerized communication, the user’s perceived social presence (the feeling of interacting with a person) should increase high degree of social presence increases desirable outcomes such as:
    --       recall
    --       satisfaction
    --       behavioral change intentions

allows a search and retrieve process to be updated and improved in successive iterations through user feedback

6 How did we implement the theoretical and interpersonal constructs?

the software agent environment:
    --       user input
    --       user database
    --       message database
    --       agent reasoning scripts and actions
    --       display of actions performed

technical specs:
    --       dynamic website
    --       SQL Server 2000 relational database
    --       scripts and actions are if-then rules and search algorithms
    --       ColdFusion markup language for displaying HTML pages

7 What initial user information is entered in the user database?

    --       users’ smoking history
    --       Fagerstrom Test for Nicotine Dependence
    --       previous quitting attempts
    --       quitting stage (i.e., stage of change)
    --       perceived difficulty of overcoming quitting obstacles
    --       demographics.

if users are currently quitting or previously attempted to quit they are asked to rate the helpfulness of the types of smoking cessation advice, nicotine replacement therapy, or other techniques they used.

if users are quitting they were asked to report their confidence in staying quit

if they are not currently quitting or considering quitting they were asked when they believed they would eventually quit

8 What is the Fagerstrom Test for Nicotine Dependence?

a scale approximation of a smoker’s relative addiction to nicotine:

1. How soon after you wake up do you smoke your first cigarette?
    0-5 minutes (3), 6-30 (2), 31-60 (1), after 60 (0)

2. Do you find it difficult to refrain from smoking in places where it is forbidden (e.g. restaurant, library, cinema)?
    Yes (1), No (0)

3. Which cigarette would you be the most unwilling to give up?
    First in the morning (1), Any of the others (0)

4. How many cigarettes per day do you smoke?
    10 or fewer (0), 11 to 20 (1), 21 to 30 (2), 31 or more (3)

5. Do you smoke more frequently during the first hours after waking than during the rest of the day?
    Yes (1), No (0)

6. Do you smoke when you are so ill that you are in bed most of the day?
    Yes (1), No (0)

Sum:
0-2   Very low dependence
3-4   Low
5      Medium
6-7   High
8-10 Very high

9 In what form are the smoking cessation messages?

a short paragraph of related information, sources are:
    --       US National Institutes of Health
    --       American Cancer Society
    --       American Lung Association
    --       project staff

for each message, administrative data and meta-data include:
    --       title
    --       source
    --       created date
    --       revised date
    --       general category (concept, strategy, fact, example)
    --       stage (from Stage of Change Theory)
    --       process (e.g., consciousness raising)
    --       age appropriateness
    --       gender
    --       race (broad, physiological distinction)
    --       ethnicity (political distinction)
    --       culture (lifestyle distinction)
    --       keywords

10 What is an example of a message?

Title: Be prepared for relapse or difficult situations

Source: National Cancer Institute

Category: strategy

Stage: pre-contemplation

Process: consciousness raising

Age: adult

Race/Ethnicity/Culture: neutral

Keywords: relapse, alcohol, smokers, weight, mood

Message:

Most relapses occur within the first 3 months after quitting. Don’t be discouraged if you start smoking again. Avoid drinking alcohol. Drinking lowers your chances of success. Being around smoking can make you want to smoke. Many smokers will gain weight when they quit, but usually less than 10 pounds. There are a lot of ways to improve your mood other than smoking.

11 How are messages selected?

three types of agents
    --       generic (same messages for all users)
    --       tailored (based on stage and other user attributes)
    --       feedback-driven (start with tailored then use feedback to refine message selection)

each initially selects 10 messages

Generic:

same ten messages regardless of user profile

five reasons to quit:
    --       health
    --       cost
    --       social
    --       professional
    --       personal

five key strategies to quitting:
    --       get ready
    --       get support
    --       learn new skills and behaviors
    --       get medication and use it correctly
    --       be prepared for relapse or difficult situations

12 How does the tailored message agent work?

messages are selected using several factors:
    --       the user’s stage of change
    --       rated importance of reasons given for quitting
    --       level of nicotine addiction (Fagerstrom index)
    --       obstacles to quitting rated as most difficult
    --       rated helpfulness of techniques used in previous quit attempts

for example, a user in the contemplation stage
    --       three messages based on the ratings of importance given by the user for reasons for quitting
    --       a message is given related to setting a target date for quitting
    --       one or two messages dealing with nicotine replacement and counseling are selected based on the user’s Fagerstrom index
    --       two or three messages are selected based on the user’s perceived difficulty in overcoming obstacles to quitting
    --       two or three messages are selected based on rated helpfulness of support used in previous quitting attempts. If ten messages have been selected, the process ends. If not, the remaining messages selected are effective strategies for the contemplation stage.

13 How does the feedback-driven agent work?

1. the agent script first selects ten messages using the tailoring procedures

2. messages are presented to a user one at time and the user rates the relevance of the message

3. the users are then asked if they are satisfied with that message or if they would like to see other messages 

4a. if satisfied, the next message from the tailored set is displayed.

4b. if the user wants to see new messages, they are shown three messages are selected using a similarity algorithm based on number of matches and non-matches to meta-data essentially:
    --       if the original message is rated high, substitute messages that are similar are chosen
    --       if the message is rated low, messages that are dissimilar are chosen

5. users in this group end up with ten messages that vary from the original ten tailored messages depending on the feedback

14 How did we compare the effectiveness of the three types of agents in our pilot study?

1. participants
    --       51 university students aged 18 to 25 who had smoked within the six months prior to the study
    --        randomly assigned to one of the three groups
    --       paid $40 for completing the study

2. pre-test questionnaires

3. smoking cessation messages presented

4. message relevance was rated on a Likert scale, "relevant means personally meaningful and helpful"

5. social presence measured using a series of six-point semantic scales
    --       impersonal-personal
    --       insensitive-sensitive
    --       unsociable-sociable
    --       cold-warm
    --       passive-active

6. message recall: one week later the participants returned to the computer lab. They were asked to recall as much as they could about the messages they received

7. post-test questionnaire: rate their stage of quitting, quit confidence or when they believed they would eventually quit

15 What types of smokers were in the study?

mean age was 19.6 (SD = 1.5, range = 18-25)

22 were female and 29 were male

racial/ethnicity:
    --       40 were white/Caucasian (non-Hispanic)
    --       6 Latino/Hispanic
    --       4 Asian/Pacific Islander
    --       1 black/African American

smoking behavior
    --       71% less than half pack per day,
    --       25% one-half to one pack
    --       4% one pack to one-and-a-half packs.

started smoking regularly
    --       40% in the past two years
    --       50% between 2 and 5 years
    --       10% over five years

16 What were the stages and nicotine addiction indexes of the participants?

the mean rating of nicotine addiction on the Fagerstrom index (0-10) was 2.0 (SD = 1.9, range = 0-7)

Frequency table results for Fagerstrom Index (0-10)

Frequency table results for pretest stage of change

17 What was their quitting behavior?

most of the participants (84%) had tried to quit smoking one or more times

most had tried to quit more than one time

about one third of the participants three or more times

about two-thirds quit for less than a month at the longest

Mean rated importance of reasons for quitting smoking (1-4)

those in the pre-contemplation stage (n = 18) were asked when they believed they would quit in the future.
    --       5 within 1 year
    --       7 within 1-5 years
    --       4 within 5-10 years
    --       1 in more than 10 years
    --       1 never

18 What were their perceived obstacles for quitting?

Mean rated obstacles in quitting smoking cessation (1-4)

19 What were their rated helpfulness of smoking cessation techniques?

Mean rated helpfulness of smoking cessation techniques and sources (1-4)

20 What were the results of the rated relevance of messages?

means and SD for each group:
    Group 1 (generic)     27 (4.1)
    Group 2 (tailored)     26 (5.8)
    Group 3 (feedback) 30 (4.0)

an analysis of variance (ANOVA, one-tailed) was performed on the total reported message relevance (total possible 10-40):
    Group 1 and Group 2 not significant (F < 1)
    Group 1 and Group 3 significant F(1, 32) = 3.6, p = .03
    Group 2 and Group 3 significant F(1, 32) = 3.2, p = .04

21 What were the results of the measures of social presence?

the five social presence items were summed (total possible 5-30), mean and (SD)
    Group 1 (generic)     19 (4.5)
    Group 2 (tailored)     19 (4.4) 
    Group 3 (feedback) 20 (4.1)

   no significance was found between groups (F < 1)

correlation between social presence and relevance ratings:
    Group 1    moderately correlated, r(15) = .53, p < .05
    Group 2    low correlation, r(15) = .19, p > .05
    Group 3    highly correlated, r(15) = .90, p < .001

22 What were the results of message recall?

recall score was calculated by counting the number of matches between the participant’s free recall and the messages given that participant

scores for each message ranged from 0 to 3, the range of possible summed scores was 0 to 30.

scores were low, the mean for all participants was 2.2 (SD = 2.1).
    Group 1 (generic)     2.9 (3.0)
    Group 2 (tailored)     2.0 (1.5)
    Group 3 (feedback)  1.6 (1.1).

   no significant difference was found

23 What were the results of intention to quit or stay quit?

overall, there was a migration trend from the pre-contemplation stage to the contemplation stage.
    11 participants (22%) moved to a higher stage
      7 (14%) went to a lower stage.

data were too sparse to perform statistical analyses by group
    Group 1 net = +2
    Group 2 net = -1
    Group 3 net = +3

24 In the feedback driven group, how many messages did the participants view?

the mean number of different messages selected from the original ten was 1.3 (SD  = 1.7, range = 0-5)

the correlation between social presence and number of differences was low, r(15) = .30, p > .10

25 What were the main results and limitations of the study, and suggestions for further research?

--       the feedback-driven process did change the experience for the study’s participants, even though the study was limited in time and exposure to the messages

--       the feedback process produced more relevant messages (10-11% higher total), which alsotended to correlate with higher social presence

--       in a smoking cessation program, participants will be given access to the messages either by printing them or allowing access to the website though a login process

--       a smoking cessation program would conducted over several weeks, the intention of this study was strictly to measure specific variables to evaluate the relevance hypothesis

--       measuring relevance in a real-time system might have to take some other form to avoid users growing weary of rating messages

--       a more refined relevance algorithm should be created using data from several users to find patterns of relevance with user characteristics