Open@VT

Last summer I wrote a brief post about access to scholarship in a MOOC co-taught by Tom Sanchez, Professor in Urban Affairs and Planning, shortly after the course’s initial offering. After TechniCity was offered again this past spring, I thought I would ask Tom more questions about the course.

Tom Sanchez

The interview below took place via e-mail over a few weeks in late July and early August. Other than a couple of typos and linking text, no edits have been made. However, the order of the questions has been changed to make more thematic sense. Thanks to Tom for his patience with my questions.

First, tell us about the course and your goals in co-teaching it.

The course is about how technology is being used to improve cities through capturing and utilizing new sources of social, economic, and environmental information. Ideally this information will lead to better decision-making by individuals and their communities. Jennifer Evans-Cowley (at Ohio State University) and I both taught courses about cities and technology and while discussing online education at a conference, decided to combine efforts and teach a MOOC on the topic. Neither of us had taught a MOOC before, so it was also an experiment with the MOOC platform. Ohio State University is a Coursera member (Virginia Tech is not), so we co-taught the course on Coursera. We knew it would be a lot of work, so we decided to teach it together with the help of two doctoral students and an instructional technologist from Ohio State.

Can you give us some demographics of the course participants, and whether there were any changes in them from the first offering?

The first time we offered the course there were over 21,000 students signed up and the second time we had over 11,000. The locations of students were nearly the same both times with about 4% from Africa, 18% from Asia, 1% from Australia, 33% from Europe, 36% from North America, and 8% from South America. We only collected simple demographics like age, gender, and location. The average student age was 33 years old and most were male (65%). The students were from nearly 100 countries and nearly 60 percent of them living in cities with 500,000 or more people. These characteristics were similar for both course offerings.

How many students completed the course?

Like other MOOCs the participation rate was relatively high, but the completion rates were low. In the first class we had 343 complete the course and about 185 in the second class. Large numbers of students view most of the course videos and other materials as well as engage in online discussion, but assignments and projects tend to reduce the numbers of students actually completing the course.

Do you know if there were any course participants from Virginia Tech?

This past Spring we had students taking the MOOC for credit at Ohio State (about 25) and Virginia Tech (about 12). We created a separate Google+ Community for these students to engage with each other and discuss course topics and projects. The instructors spent more individual time with students taking the class for credit and provided direct comments on projects/assignments.

What support (technical, administrative, or otherwise) did you have here at Virginia Tech for co-teaching the course, and were there any areas where you could have used more support?

I received support from Anne Moore, Gardner Campbell, and Jennifer Sparrow (InnovationSpace). Unfortunately none of them are still at VT. Anne and Gardner provided resources for me to buyout a course to concentrate on developing the MOOC. Jennifer and InnovationSpace provided much needed help with video production. The technical help from Ohio State (Tom Evans) was invaluable because no one at VT was working with Coursera. It would have been impossible otherwise. If you’re going to build a MOOC you’ll need to know about video and I’m not sure if most universities are ready to help large numbers of faculty with this.

What is your sense of the interest from Virginia Tech faculty in teaching a MOOC?

A colleague in UAP, Jocelyn Widmer, has extensive experience teaching online and has expressed interest in MOOCs. Other than that I haven’t heard much discussion.

Has co-teaching the course affected your teaching at Virginia Tech?

The course has changed my outlook on teaching. It was much more work than I anticipated, and I learned quite a bit in the process. I think our course is well-suited to the online format, but may not work for other types of courses. And not all students care for distance learning, online courses in particular. I believe online courses will improve over time as we adapt technologies to different learning styles. The key is learning how to learn, regardless of the platform.

The TechniCity e-book (PDF) is a nice resource to make available. Was it intended mostly for prospective students?

The e-book is a good way for us to show prospective students what the course is about. I think it is also a good way for us to document what we’ve accomplished. Because the course is about rapidly changing technology it will be interesting to build a record of topics and activities. Assembling the e-book is a good way to reflect on course structure.

You have an article in press about the course- does it include the results of the post-course student survey?

Yes, we include summaries of the pre-, mid-, and post-course surveys in the article (forthcoming in the International Journal of E-Planning Research). These were valuable sources of information as we made changes to the course. The post-course surveys help us to gauge interest in particular topics, activities, and overall structure. One of the challenges is that because the course is free, if students don’t like it they can just stop participating with no penalty. We have a survey for those who un-enroll but the response rate isn’t too high. These are the people we’d really like to get more information from to improve the course because those who finish it tend to be the ones who liked it.

Last year we communicated briefly about the difficulty of ensuring that the course materials were openly available to everyone, which can be a problem for peer-reviewed articles in an international environment. What was your experience this time around?

We mainly relied on open and accessible materials available on the web. In our course, many of the case studies and recent research has not been published in peer-reviewed form yet, with the best resources already available for free on the internet. With so much free information available, we have plenty to cover our topics.

Do you think MOOCs have influenced the need for openly available information?

I’m not sure, but so far I haven’t seen any evidence of it. Many of the MOOCs that I’m familiar with are relying on video/lecture, assignments, and discussion, rather than “reading” materials from books, journals, etc. So much is already available online so in many cases there seems to be plenty of options.

Course ownership has been a faculty concern with some MOOCs. Who owns TechniCity?

Jennifer Evans-Cowley and I consider the course materials to be openly accessible even though Coursera isn’t necessarily open (unless you sign up for a course). We haven’t run into a situation yet where we had to tell someone we couldn’t share information from or about the course. That I know of we have not had any situations where Coursera restricted us from doing so either. The question of “ownership” hasn’t come up.

MOOCs are often cast as a disruptive innovation that could threaten higher education– what is your take?

In my opinion MOOCs are just another form of online education. Perhaps the disruptive aspects are the scale and cost. The issue with new models of education is that both the instructor AND the learner need to understand the process and expectations. I think we should actually be teaching more classes about learning, knowledge, and discovery – but unfortunately you don’t see those words used on job descriptions so we don’t make them priorities. The only threat to higher education is higher education itself. In many cases these are very slow moving institutions that are more than happy to keep doing things the old way. I think the university business model is in dire need of change.

Looking at the syllabus, open data seems to play a large role in this course—how would you summarize the challenges and opportunities for cities in providing it?

The challenge is to organize the huge amount of data generated by and within cities, and make it available in meaningful and useful ways for residents, businesses, and entrepreneurs. I think cities can provide the data and have the crowd come up with innovative ways to use it. This is especially exciting when it enhances transparency and accountability. An example is the City of Baltimore (see: https://data.baltimorecity.gov/). The site makes data available as well as analytical tools, maps, and other visualizations. There are some great research opportunities as well.

Do you think students were well prepared for interaction with data, or do you see a need for more training in the use and manipulation of data?

I think this is a particular need in the field of planning, especially with the rise of “big data” and associated analytics. Planning programs tend to require quantitative methods, but usually in the form of statistics and not really “data science”. Many students coming to planning have design interests so data analysis isn’t of particular interest. There will continue to be more and more demand for data experts.

Do you plan to teach the course again next year?

Yes, we’ll be teaching it again, probably in the Spring. This will give us another opportunity to improve the course using feedback we received from students so far.

One participant in our faculty and graduate student panels during Open Access Week at Virginia Tech was Josh Nicholson, founder of a new open access journal, The Winnower. Josh is a PhD candidate at Virginia Tech studying the role of the karyotype in cancer initiation and progression in the lab of Dr. Daniela Cimini. The Winnower will serve the sciences, with sections for different disciplines as well as a science and society section. The new journal will launch in January or February, and is currently looking for beta testers. Some buzz has already been created through a post on the AAAS blog and a Q&A, and The Winnower is active on Twitter and Facebook.

Josh Nicholson

Our e-mail interview occurred over several weeks. The questions and answers below have not been edited, except to add an occasional link. I have also grouped similar topics together, so the questions are no longer in their original order.

How did The Winnower come about?

Ever since I began publishing science articles I have asked: does the publication system make sense? The short answer has always been: no. I think most scientists who have published an article would agree with this but they are often too involved in playing the so-called “tenure games” to do anything about it. Well I don’t have to worry about tenure yet (if ever) so I can focus on the problem at hand and try and actually do something about it.

Why the name?

The Winnower is a tool used to separate the good from the bad. This is a main objective of The Winnower, to identify good pieces of research from flawed pieces based on open post-publication review.

Will the journal use a Creative Commons license or allow authors to choose?

Content published with The Winnower will be licensed under a CC BY license.

Will the journal be able to accommodate data as well?

The journal will accommodate data but should be presented in the context of a paper. The Winnower should not act as a forum for publishing data sets alone. It is our feeling that data in absence of theory is hard to interpret and thus may cause undue noise to the site.

Will there be any screening process before an article appears?

No, articles can be posted on The Winnower immediately. This should not be taken as an endorsement that they are correct but rather a signal that they need to be reviewed, much like the “preprint” system. Of course, articles that are reviewed will be easily distinguishable from those that are not. The site is designed to encourage reviews of papers, indeed it is why we are called The Winnower: to separate the bad from the good. To limit possible spamming of the system as well as to sustain The Winnower there will a charge of 100 dollars per publication.

How will you accommodate the need for fast review in an open peer review system?

The Winnower will strictly utilize open review. This means that all publications will be open to review and all reviews will be open to read. Publication in The Winnower will occur immediately after submission and reviews will be open for variable amounts of time so that authors can make edits based on the reviews. It should be noted that papers will always be open for review so that a paper can accumulate reviews throughout its lifetime. Reviews can be solicited from peers upon submission and reviewed by The Winnower community we hope to build. Based on the system we are building we believe the number of reviews should reflect the number of times the work is read.

At what point can an author say that a paper has been peer-reviewed?

An author can say it has been peer-reviewed as soon as a paper receives a review. But we hesitate to say that this paper has passed peer review because doing this causes some problems. Indeed, as you may be aware all work published now has “passed” peer review but that has done nothing to limit the high rates of irreproducibility. In fact, it may be a cause of it. We want to change the conversation from “passing” peer review to what is the percent confidence scientists have in this paper. To accomplish this we will be implementing semi-structured reviews (i.e. turning reviews into a measurable quantity).

How will reviews be a “measurable quantity”?

Much like reviews are performed by the National Institutes of Health scoring will be implemented for different criterion. Obviously there will be no way to score free form reviews but various questions can be assigned a numerical score. PLOS Labs is working on establishing structured reviews and we have talked with them about this. We think it would be great if there is an industry standard to use for structured reviews, but until then we will implement the best system that we can think of.

What do you mean when you say that peer review can be a cause of irreproducibility?

Peer review, as it stands now, is more or less a pass/fail system. So, if you design 4 experiments to test a hypothesis and only 3 confirm your hypothesis you are likely to leave out the 1 experiment that did not fit your hypothesis in order to pass peer review. The problem is that ultimately you can’t hide from nature, she will reveal her truth one way or another. If there is no system to pass or fail and you wish your paper to stand the test of time you will include all results, even those that contradict your hypothesis. Moreover, editors are literally selecting for simple studies but very often studies are not simple and results are not 100% clear. If you can’t publish your work because it is honest but poses some questions then eventually you will have to mold your work to what an editor wants and not what the data is telling you. There is a significant correlation between impact factor and misconduct and it is my opinion that much of this stems from researchers bending the truth, even if ever so slightly, to get into these career advancing publications.

How can you ensure that each paper is reviewed, or receives enough reviews?

Authors when submitting their research will be encouraged to invite reviewers directly to review their paper. Some may argue this will allow authors to invite their friends and the reviews will be biased. We think the transparency of reviews will limit this from happening. In addition to authors driving reviews to the site each article will display a prominent “write a review” button.

Isn’t bias often hidden? For example, if a submitter invites friends to review, wouldn’t that relationship be invisible to readers, and reviewers could go easy on criticism and exaggerate praise?

This is certainly a possible problem that could arise but it is not anything new with our system. Currently, scientists are allowed to suggest those that should and should not review their papers. Indeed, you heard this blatantly revealed during Open Access Week by a researcher [Note: Josh is referring to the faculty panel during which Dr. Good said some journals prompt authors to suggest reviewers]. Arguably there is an editor to limit any bias but the editor themselves could be biased one way or another. While The Winnower won’t eliminate bias (we are humans, after all) the content of the reviews can be evaluated by all because they will be readily accessible. [Note: reviewers could list competing interests in the template suggested on The Winnower’s blog.]

You recently wrote a blog post “Sexism in Science” that cites an article advocating abandoning secrecy. But other research concludes that double-blind review is best, and since even the article you cite mentions other studies in which female representation is better when gender is unknown, wouldn’t double-blind review do a better job of eliminating sexism?

Double blind review is indeed better than single blind review in regards to eliminating sexism in science but this does not mean that it is the best. As far as I am aware there has been no test between open review and double blind review. Any instances of sexism that do occur in open review can be addressed and fixed because they can be exposed unlike closed review. In the Sexism in Science blog I discuss a few cases in which blatant examples of sexism in science occur. In the end many have been remedied because of the open dialogue that occurs on the internet.

Does open peer review mean that all authors and reviewers must reveal their real names?

Yes.

How will you ensure that reviewers are using their real names?

This is not easy, but we think with the system that we are building reviewers will want to use their real names. Reviews will be assigned DOIs and over time we hope to put the reviews on the same level as the research. Indeed, I can imagine researchers that specialize in reviewing and being rewarded for doing so. Full time Winnowers, if you will. But regardless if a reviewer uses their real name or not , the transparency of reviews will discourage personal/inappropriate reviews. It is the serious criticisms/reviews that will be difficult for authors to respond to. I strongly believe that if you’re scared of open peer review then we should be scared of your results.

Do you plan to use altmetrics on the site?

Yes, we will use various metrics on the site, including altmetrics. We want to shift the focus from the journal to the article itself and we think employing various article-level metrics is the best way to do this.

Have you decided on an altmetrics service and will some revenue go toward that?

Yes, we will be using Altmetric and yes some of the revenue will indeed go towards that.

At what point does payment occur, and are you concerned with the possible perception that this is pay-to-publish?

Payment occurs as soon as you post your paper online. I am not overly concerned with the perception that this is pay-to-publish because it is. What makes The Winnower different is the price we charge. Our price is much much lower than what other journals charge and we are clear as to what its use will be: the sustainability and growth of the website. arXiv, a site we are very much modeled after does not charge anything for their preprint service but I would argue their sustainability based on grants is questionable. We believe that authors should buy into this system and we think that the price we will charge is more than fair. Ultimately, if a critical mass is reached in The Winnower and other revenue sources can be generated than we would love to make publishing free but at this moment it is not possible.

From what funds do you think most scientists will pay the $100 fee?

I believe that most academic scientists will pay the $100 fee with grant money. If they do not currently have grant money the fees could theoretically be paid for by departmental funds or even personal funds.

Is The Winnower a for-profit or non-profit enterprise, and are you registered as such?

The Winnower is a for-profit limited liability company.

Is there a preservation plan for the content in case the journal does not continue?

Yes, we will be using the CLOCKSS program.

Is it possible for an author (or journal staff) to withdraw an article?

Yes, it is possible to withdraw an article and it is also possible for us to retract the article if necessary.

Since many scientists do need to play “tenure games”, wouldn’t the Winnower’s lack of indexing, impact factor, etc. serve as a disincentive to submit or review?

Yes, this is certainly an obstacle The Winnower will have to face but it is not only an obstacle for The Winnower rather it is an obstacle for the entire scientific community. We think we need to get away from judging scientists based upon IF or other measures of prestige and we are not alone. The San Francisco Declaration on Research Assessment (SF DORA), which has been signed by nearly 10,000 researchers and publishers in less than a year, calls for new ways to evaluate researchers. As the community moves away from journal-level metrics and into article-level metrics The Winnower should be well positioned to thrive. Indeed, we will utilize many article-level metrics as well as information from reviews themselves.

With most journals, if I submit a paper that is rejected, that information is private and I can re-submit elsewhere. In open review, with a negative review one can publicly lose face as well as lose the possibility of re-submitting the paper. Won’t this be a significant disincentive to submit?

This is precisely what we are trying to change. Currently, scientists can submit a paper numerous times, receive numerous negative reviews and ultimately publish their paper somewhere else after having “passed” peer review. If scientists prefer this system then science is in a dangerous place. By choosing this model, we as scientists are basically saying we prefer nice neat stories that no one will criticize. This is silly though because science, more often than not, is not neat and perfect. The Winnower believes that transparency in publishing is of the utmost importance. Going from a closed anonymous system to an open system will be hard for many scientists but I believe that it is the right thing to do if we care about the truth.

Is there anything else you would like to add?

The Winnower will also feature two sections called “The Grain” and “The Chaff.” The Grain will be short essays by authors of papers that have received 1,000 citations or have passed a specific Altmetric score. In these essays authors will describe the work and the story behind the work (i.e. was it initially rejected, was it funded, where did the idea come from etc.). They will be very similar to the former series Citation Classics run by Dr. Eugene Garfield. Indeed, Dr. Garfield has expressed much enthusiasm for The Winnower to pursue this. In parallel, we will be launching a section called The Chaff that highlights retracted papers. These papers will be written by authors of retracted papers in order to really find out why studies failed or what led to the authors to fabricate data etc. We want to position papers published in The Chaff in a non-accusatory manner so that we may learn from these papers. The Chaff will not be a forum to castigate authors of retracted papers.