And . . . already behind on the dissertation blog updates.

Anyway, here's the update and some reminders to myself for future posts:

-Spent lots of time last week putting finishing touches on DSG proposal - really helped me clarify thoughts on the content/argumentation interaction.

-READ a bunch of papers while writing our group's Nat'l Association for Research in Science Teaching paper about refining our argumentation learning progression

-Re-read the vonAufschnaiter et al (2008) paper about argumentation and content - spend a future post talking about how that study is the starting point for the content/arg relationship analysis part of this project

-Re-read Gotwals & Songer et al (2009-2013) work on developing a learning progression and assessment items for development of "complex thinking" in ecology.

Next week is SPRING BREAK (aka time to work on dissertation) - plans 

The main activity of this week was drafting a proposal for some dissertation support funding.  There is a required section in this proposal called "Analysis" where you are supposed to discuss how you are planning to go about analyzing your data - not in general terms (i.e. I'm going to fit a such and such model, or I'm going to develop a set of codes and apply them), but in a way that shows how exactly you are going to use results of analysis to answer research questions.  I'm still working on this section, so this will be a little thinking out loud.  For now, I'm just going to concentrate on how I'm going to analyze the data coming out of the assessment, rather than the interview data with scientists. 

I will have two primary sources of data: transcripts of think alouds with high schools students while they complete each item and the set of completed assessments from ~200 students. 

My first question is whether we can develop items with high reliability and high validity - I often throw these words around a lot (why of COURSE we want high reliability and validity), but there are lots of kinds of reliability and validity.  So I think step 1 is to think about what types of reliability and validity are most important to me for this project.  *note to self - think about this*  But, I think the basic answer in this proposal will be that I will run a basic psychometric analysis (looking for items with good discrimination, and reasonable internal reliability), test for unidimensionality (EFA, e.g.), and then use unidimensional and multidimensional IRT analysis - I will be looking to see that my items span the range of student ability.  I wouldn't want to see all items clustered at the bottom or at the top of the ability distribution. 

My second question is about the relationship between content knowledge and argumentation.  Lately, on another project, we seems to be leaning toward a more specific question: does the ability to argue at a certain level depend on having certain content knowledge?  I'm not sure I'm ready to get that specific.  If I do, I have been reading about the structured constructs model (SCM, under development by Mark Wilson's group) - this allows one to test dependencies within a data set.  Perhaps I'm not quite ready to make these types of hypotheses yet - perhaps a better question is what types of content knowledge do students seem to be bringing to the argumentation task (e.g. specific ecology knowledge, more general epistemic knowledge), to what extent does the quality of their arguments and critiques rely on this specific knowledge, and (for the future) is there a way to test the dependency?

Tasks for next week:  DSG: Analysis, Statement of Qualifications, finish budget
Complete Us

This week I have been writing a grant proposal - Stanford has a Dissertation Support Grant program, and I am applying for funds to cover things like transcription, paying scorers, providing participating teachers with honoraria?  honorariums?  Some payment for their hard work.

Mostly, it has been a useful exercise in trying to pull together a coherent story of what I am trying to do and why it matters.  Though I gave a talk on my proposal in December, this short (~15 pages, including references etc.) format is forcing me to edit down to the bare minimum.  Then, I will use this + the other unconnected pieces of my proposal writing to comprise the final dissertation proposal (which will be a much longer docu


Other ongoing tasks:

• Gathering some sample responses to Item #1: Why does the moose population change?.  Need some early data to clarify how I will score and analyze written responses, especially with regard to RQ1:  Relationship between content knowledge and argumentation.  Right now just sending out to friends/family/colleagues.
  

• Trying (unsuccessfully) to upload interview .mp3 and .wav files to an audio transcription site.  I have been thinking hard about how best to use my time, and finally made the decision to go ahead and upload all of the interview transcripts that I haven't transcribed yet to a service that will do it for me.  I don't currently have funding to cover it (see DSG funding proposal writing above), but it is such an incredibly time consuming process, that I think to move forward with the analysis of scientists interviews, I need help transcribing.  ANYWAY, the entire discussion above is currently irrelevant, because my files are too big, and the site won't accept them.  Working on splitting files now.  If anyone has advice on how to easily split .wav files into smaller ones, I am all ears.
  
  

I have one item bundle that has been conceived, developed, built on Qualtrics, and is ready for beta testing by a population that isn't necessarily my target population (i.e. high school biology students).

Why is it so hard to write these items? 

I have been involved in writing items such as these for several years now through my work on the Learning Progression in Middle School Science project.  The primary rationale for spending time conducting interviews with a set of practicing ecologists before attempting to write any items situated in ecological contexts was to make it easier.  To not spend so much time pulling my hair out trying to think of a worthy question for argument.  To not spend so much time thinking about the types of evidence that would be most relevant.  And yet, even after conducting and analyzing a set of interviews that were full of interesting and engaging ecological questions, it took many hours to identify an argument and develop an item that would allow students to engage in meaningful, written argumentation.

My initial requirements for these items (taken from an early proposal draft from 9/2013)

• Two competing views or hypotheses

• A context that does not require elaborate tables, graphs or text
  

Here was what I proposed to do:

• Conduct semi-structured interviews with practicing ecologists (What are some longstanding arguments in your field?  Can you tell me about an argument you have had with someone else in your field?  Can you help me think of some context that you think might be approachable enough for high school students?)
• Use interviews to define the “flavors” of argument that exist in ecology.  Create a taxonomy of types of arguments to be had in the field of ecology.  Also, use interviews to generate list of scenarios.
  

Initially, I hypothesized that there were fundamental concepts within ecology that were arguable.  Arguments of this ilk could include (1) which theory accounts for an identified pattern (why do we see this pattern?)?  (2) Does something fit an established definition?  Related, but perhaps not the same, are basic taxonomic arguments, e.g. is this species A or species B?  Is it a plant or an animal?  Is it alive or not?  (3) Is this a change we observed in an ecosystem net positive or negative?  (e.g. does it increase the stability of an ecosystem, or decrease it?  Does it have a positive, negative, or neutral affect on an organism.

Preliminary analysis of interview data showed evidence of all three types of arguments described above.  The one that was discussed most frequently and at the greatest length was Type 1: Why do we see this pattern in nature?  (note to self:  blog post re: evidence of all 3 argument types, blog post re: the importance of multiple working hypotheses in ecology).  The argument that such a question sets up is one consisting of two competing, causal claims (note to self: blog post re: "causal claims" as defined by us on LPS). 

Right now, I am considering trying to stick with this very particular type of argument in ecology as the basis of all assessment items.  Based on the interviews, this seems to be the dominant argument type within the field of ecology, and other educational researchers have had previous success with the other two types of arguments (e.g. see Sampson's book with examples of #2: the taxonomic argument, many of the socioscientific argumentation tasks look similar to #3: the value-laden argument)

This first item (Why does the moose population change?) is based on source material from an interview with a marine ecologist who thinks (and argues) a lot about the primary controllers of food chains: are they controlled from the top down (e.g. by predators)?  Or are they controlled from the bottom up?  The structure of the argument has been preserved in the item, but the context is different.  The moose and wolves on Isle Royale have been studied for many years, and there are lots of educational materials, including easy-to-understand graphs and figures, available for this context.  So, the argument moved from the marine intertidal off the West Coast of the U.S. to a terrestrial ecosystem on a wilderness island in the middle of Lake Superior. 

Head on over and engage in an argument about moose and wolves . . .

I am trying to do a better job tracking my progress on my dissertation.  I will try to post twice per week.  Wish me luck.