The MedLitBlitz: our week-long birthday party with more than 300 of our closest friends
Posted on 25th May 2017 by Cochrane Crowd & Mark2Cure
How can we make it easier for health practitioners to access current evidence to inform treatments? One of Cochrane’s responses has been Cochrane Crowd, a platform that enables anyone with an interest in health to dive into tasks that help identify and describe health research. Cochrane Crowd is making it easier for health researchers to find the latest, high-quality evidence on what treatments work and don’t work, in turn helping evidence reach the health practitioners who need it. You can view a 2-minute explanation of Cochrane Crowd here.
Since the launch of Cochrane Crowd in May 2016, more than 5,600 people from 118 countries have signed up to take part.
In the lead up to our first anniversary we decided to have a big celebration, joining forces with another innovative platform Mark2Cure to run the MedLit Blitz, consisting of three events: a webinar, a 24 hour Cochrane Crowd screening challenge, and a 24 hour Mark2Cure challenge. This is the story of our MedLit Blitz birthday celebration.
Before the Blitz
As with all our events, as the time drew close, we wondered whether anyone would join in. We had felt a little jittery before our last challenge, and this time we were running three events within a week. What were we thinking?
As we gathered for a pre-webinar practice the week before, only 19 people had registered, and while our spirits were high we did wonder if we had overreached. But the nerves tipped into excitement and then exuberance as the webinar count rose, and by the time MedLit Blitz started we were ready for action.
Webinar
By the day of the webinar 145 people had registered and many had emailed to ask if it could be recorded. Which of course it could, once we figured out how to use the webinar software! On Monday 8th of May, Anna and Ginger gathered at what was a reasonable time for both (not a given, in this business of international collaboration!) and presented a lively, interactive webinar. They introduced the platforms, described how people can get involved, and invited everyone to join us for the remaining two birthday events. As promised through pre-Blitz tweets, Anna also shared the story of Crowd’s first challenge, which didn’t quite go according to plan.
You can listen to the webinar, view the slide deck, and read responses to participants’ questions here.
Cochrane Crowd Challenge – perspectives from Australia
The very next morning UK time, our 24 hour Cochrane Crowd Challenge began. Anna, Gordon and I gathered on Skype to talk through the challenge schedule, before Anna suddenly realised she was down to her last tea bag and rushed out for reinforcements and I sorted out dinner (in Australia) before settling in for the start.
We’d set a goal of 20,000 classifications over the 24 hours. We considered it a stretch goal, based on the pace of last year’s Cochrane Crowd Countdown to One Million Challenge, but really we didn’t know how many people would take part and whether we’d get anywhere close.
UK day, Tuesday 9th May
In the final pre-challenge minutes I watched the time tick over on my laptop and phone, in between tweeting, Skyping and emailing, and realised they were at least half a minute out of sync, which was surprisingly disconcerting.
And then at 10am UK time we were off…
The group that had been patiently waiting to start set off at a rapid pace. For a while the classification count climbed so quickly that it was hard to take a reading. At around ten minutes in we had confirmation that all was smooth from a technical standpoint: a milestone behind-the-scenes moment. At around 20 minutes I took stock of my adrenalin-fuelled state to note we’d reached 700 classifications already! And to take at least one bite of a cold dinner.
We managed to outpace our first half-hour in the second, and by 11am UK time we’d screened more than 2,000 classifications. At that early stage things looked good for achieving 20,000; but, with many of our core group in Europe we knew if we weren’t close(ish) by bedtime in the UK it might be touch and go.
After the initial fast-paced hour, some of the Crowd headed into their working day, and things settled into a more even pace. At around the three hour mark, as we hit 4,700 classifications, it was getting late in Australia and time to head off for zzzzs. It was hard to say goodnight to Anna and Gordon and to power down that night.
UK night, Tuesday 9th May
Up with the kookaburras, I opened up my laptop and asked what I’d missed. This came back within a nano-second, “a crackin’ day full of high drama, tantrums, tears and pure, pure joy”. Thanks, Gordon.
It was 9pm UK time, the pace was picking up, and around 14,000 classifications had been made. Presumably dinners had been had, dishes done (?) and a larger crew had once again gathered to screen together, albeit in their own homes.
At that point Anna, Gordon and I felt fairly confident of reaching 20,000. But we didn’t realise it’d come quite so soon.
Over those first few hours of my day we watched the collective pace skyrocket. Between 8 and 9pm in the UK, 665 classifications were screened. Triple that amount were screened the following hour, and more than four times that between 10 and 11pm. People didn’t want to go to bed as we were SO CLOSE to 20,000. By then breakfast had been placed in front of me but I simply couldn’t eat so close to this milestone!
At 11.30pm UK time we crossed that magical number of 20,000 and I felt an enormous sense of relief. I caught the eye of a friend on the other side of the café and signalled two thumbs up. Someone tweeted to ask if I’d enjoyed breakfast.
After a team check-in, the new milestone of 25,000 classifications by 10am UK time had been set, and Anna and Gordon clocked out for some well-earned sleep.
The finale: UK morning, Wednesday 10th May
You only have to glance at the chart below to get a sense of the final hours of the challenge.
We crossed the 25,000 milestone just on 9am UK time, with Anna, Gordon and I having been glued to the tally, as well as screening, for the few hours prior.
And if that wasn’t enough, the Crowd continued to pick up pace in the final hour – an hour that was about as exhilarating as it gets in the scheme of online events, surely! Having clocked the milestone, we set ourselves the task of making the final hour the most productive, and indeed that is what we did. We screened 4,282 classifications – 1,714 more than the prior highest count! What a finish. I hopped into another cold meal and jumped across to a celebratory Skype with Anna and Gordon.
Mark2Cure challenge
While we at Cochrane Crowd were celebrating, Ginger from Mark2Cure was finishing her preparations for the 24 hour Mark2Cure challenge, or Mark2Curathon, which started at 7pm UK time on Thursday 11th May. Ginger says,
“While I was hopeful we’d be able to continue the momentum we saw in the Cochrane Challenge, I also wasn’t confident because Mark2Cure requires more extensive training. I knew we have some very capable and enthusiastic Mark2Curators who would rise to the challenge, but I wasn’t sure whether the Cochrane Crowd would enjoy the task.
Turns out, I didn’t need to worry so much. Although we had a smaller turnout, about half of them had also participated in the Cochrane Challenge. Furthermore, one of the top contributors from Cochrane actually completed every available Entity Recognition task and then continued to contribute to the Relationship Extraction task!
Every available abstract in the Entity Recognition Module was completed at least once by the Mark2Cure screening challengers, and challengers submitted over 16,000 annotations!”
Prizes
The top three contributors to MedLit Blitz were:
- Shireen Rafeeq
- Deirdre Beecher
- Christine Kent
Congratulations to these three prize winners who will all receive a special prize pack from Cochrane Crowd and Mark2Cure.
We’ve also awarded a prize to the top Cochrane Crowd contributor in the year since our launch. This prize has been awarded to Mersiha Mahmic-Kaknjo who has screened a whopping 77,228 citations since she joined. A prize pack is also heading her way.
In numbers
Webinar:
Registrants: 145
Participants: 75
Average time in session: 63 minutes (the entire webinar)
Average attentiveness: 93 % (the average percentage time attendees had the webinar as their primary screen – sensational result!)
Crowd Challenge:
Participants: 146
Classifications made: 29,424 (er, seriously, wow, we originally aimed for 20,000)
Reports of RCTs found: around 400
Time reached 20,000 classifications: 11.30pm UK time Tuesday 9th May
Time reached 25,000 classifications: 9am UK time Wednesday 10th May
Most productive hour: 9-10am UK time Wednesday 10th May (4,284 citations)
Cups of tea consumed: several hundred
Mark2Cure challenge:
Annotations submitted: over 16,000
Most productive times: first 2 hrs and last 2hrs of the challenge
Three participants annotated entities (genes, diseases, and treatments) for every abstract available, and then continued to contribute via the relationship extraction module.
Words from the team
[phone rings] sorry mum! can’t chat now MedLitBlitz underway (Anna)
“A crackin’ day full of high drama, tantrums, tears and pure, pure joy”. (Gordon)
“The nice thing about global challenges with global partners is that someone is likely alert when you’re failing to stay awake. (Ginger)
“There’s no way I’m stopping for dinner now. 45 minutes to go, and aiming for 2,250 more classifications!” (Emily)
Final words from Anna
Events like MedLitBlitz are a stark reminder, if one was needed, that together we really can make a difference. Both the Cochrane Crowd and the Mark2Cure contributors are helping to transform the way we manage information and ultimately turn it into knowledge. As co-lead on Cochrane Crowd, and having watched it grow from its first contributor to over 5000, I know I have one of the best jobs in the world. Connecting up with Mark2Cure for this event was an absolute pleasure and I look forward to lots more joint events. Health is a serious subject but that doesn’t mean our endeavours to discover the evidence can’t be fun!
Happy citation screening everyone.
