Search filters with Julie Glanville

What are search filters and why do we use them? How can we use them effectively? These questions convey the main content of the second workshop given by Julie Glanville from the York Health Economics Consortium.  clip_image002So, what are search filters then? Search filters are strategies to find a particular kind of research, type of population (age groups, inpatients), geographic area etc. In a 2014 paper (Beale S, Duffy S, Glanville J etal. Choosing and using methodological search filters: searchers’ views. Health Info Libr J. 2014 Jun;31(2):133-47.), users were surveyed about their use of filters. The main reasons are: to refine systematic review searches, to assist in answering short questions (PubMed Clinical Queties eg) or to asertain the size of the literature when doing a scoping search. Why did users choose the filters they did? The most common answers were performance during validation and sensitivity/specificity information. What about appraisal? Can you critically appraise a search filter? There is a tool for that and it is available at the ISSG Search Filter website. Julie talked about the main issues that you want to know about the filter (these are considered in the tool): what is it designed to find, and how are the terms identified, prepared, tested and validated?

When appraising a filter, there are are a number of issues within each question to consider. For the first question, 2 of the obvious ones are – does the filter use a database I can use too? And – does the filter have the same idea about the topic that I do? The second question involves questions like how were MeSH terms selected and were the terms selected in an organic or highly structured way? How the filter was prepared relates to the questions above: how were the terms combined for testing?

Here we come to a tricky bit – sensitivity and specificity. And precision – where does that fit in? I’ll be honest here, these concepts still trip me up and I have to take time thinking about them before I can take action. I’m working from the handouts Julie prepared and memory. Sensitivity: how many relevant records are found out of all the relevant records available? Specificity: how many irrelevant records are not retrieved. Precision: The number of relevant records retrieved out of the total retrieved by the search strategy. If a research design is  common, the more likely the retrieved record using the design will be relevant. So when you want to use a filter, perhaps you should look at the precision reached during testing. Or if you want to be as sensitive as possible, perhaps looking for sensitivity percentages is the way to go. What do you think/do?

Ok – now for validation! First things first – validation is not testing (but it should be part of the development phase). If you test the filter in the same reference set as the first testings, that is not validation. Validation requires the filter to be tested in a different reference set, or in a database (real world). This determines that the filter maintains performance levels shown in initial testing. Julie demonstrated with a test filter [W.L.J.M. Devillé, P.D. Bezemer, L.M. Bouter Publications on diagnostic test evaluation in family medicine journals: an optimal search strategy Journal of Clinical Epidemiology, Volume 53, Issue 1, January 2000, Pages 65-69] that validation performance fell from 89.3% to 61%. Haynes filters also dipped from 73.3% to 45%. That is quite a lot! Julie surmised that it could be something to do with the reference set or that neither perform consistently.

Sometimes a filter isn’t quite what you are after – nearly there but not quite. So you edit it. Warning! This isn’t the same filter (which has been developed through research). It is a filter you have made based on another. If you do this (and I admit that I have) it is very important to record this and why you made the change, plus acknowledging the original researchers. Also, be prepared for unexpected consquences from your modification.

The workshops were excellent – everyone learnt new things and I came away inspired.  Julie is a great trainer and presents difficult concepts in a very accessible way. If you ever have the chance to attend a workshop or training given by Julie, grab it! It is well worth it.

Text mining with Julie Glanville

The two advanced searching workshops went brilliantly with lots of group discussions and questions from the audience. The only downside to the day was the too-cold air-conditioning. Can you concentrate with icy air being blasted on you? Luckily the day was warm  so lots (including me) went outside during breaks to warm up. Lots of ground was covered over the day and gave everyone lots to think about.

The first workshop was about using text mining tools to build search strategies. I had heard of PubReMiner – a tool that mines PubMed records clip_image002using words from your query to bring up MeSH terms, top authors and journals, word frequency and other data. I used this tool last week to get a list of words for a search about spiritual care I am working on. Another tool similar to PubReMiner is GoPubMed (you could use both to build strategies as each have different visuals). MeSH on Demand, a tool provided by the National Library of Medicine, takes a piece of text, say a systematic review protocol, and mines PubMed to bring up relevant MeSH terms and PMIDs. If you have a strategy ready, you could use this tool to find out if relevant PMIDs have been captured in your results. Both these tools identify single words and MeSH terms, but what if you want to identify phrases? There is a tool for that and it is called TerMine. The example Julie used was a full Medline record, but you can use larger pieces of text. Then there is EndNote which you can use to analyse frequency data from databases other than Medline. There is a bit of work involved to set it up to do this though. And I guess that the records that you analyse would have to be relevant ones in order to build up a list of terms. If you can do this to the full text in EndNote, that would be great! I will have to do some experimenting. The last two freely available tools demonstrated was Quertle and Knalij. Knalji is fun tool that looks like it demonstrates relationships between concepts. And Quertle, which I had forgotten about, now has relationship terms (called power terms) that you can use to connect concepts and bring up records mined from the web. And if you want to take a break from looking at heaps of text, you can take an alternate route by using a tool called VOS Viewer.

So now that you have a selection of terms, how do you combine them into a search strategy? The most popular concept model is PICO, but not all questions fit into this model. What if you have questions about public health or adverse effects? What do you do when you have questions that don’t lend themselves to A AND B AND C strategies? This is where searching for multiple combinations comes into play. I did one of these recently and one combination I used didn’t come up with a relevant article I found using a different combination and vice versa.

The last section of this 1/2 day workshop was about ensuring strategy quality. A common method is to have some gold standard articles to see if your strategy  captures those. You could also use the Related Items in PubMed to find relevant articles to test with. Other methods include testing with specialised sets eg CENTRAL or a service called Epistemonikos (Database of Best Evidence). Another way to test your strategy is to work out the ratio of relevant records to irrelevant records. And finally, there is the Capture-Recapture method. This method, which I hadn’t heard of before, is a way of estimating size by caputuring and then recapturing records.

Lunch! Then the following post will be about workshop 2.

Reporting systematic review strategies

tablet-blister-2-1307761-mA group of medical librarians (including yours truly) have been meeting once a month after work to discuss an article of interest. This is a very informal group which has just got off the ground, but we are all having a good time so far.

The article we discussed last month was Golder S, Lake, Y.K, Zorzela L. Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews. Health Information & Libraries Journal 31(2) 92-105. The main points of the article is reporting of search strategies needs to improve. Having more librarians on board will help to achieve this goal. This survey found that only 9% of librarians were involved. Compared to other systematic reviews, reviews of adverse effects searched a range of databases including Medline. What is concerning though is that surveys of other reviews found that 56% of reviews only searched Medline. This is very concerning. This might be for a number of reasons: cost, access, or expertise. The most popular databases to search are free to search. These databases (not including Medline) are: Cochrane Database of Systematic Reviews (paid for by govt in some countries), Database of Abstracts of Reviews of Effectiveness (free via the CRD), FDA website (free), the internet (free but type of browser not mentioned), citation searches (some free) and clinicaltrials.gov (free).

We talked about floating subheadings (I remember learning about them when an undergrad), which allows you to pair a subheading with a subject that doesn’t include the subheading you want to use. This is useful in looking for reports of adverse effects, but could bring up lots of other unrelated reports. One of the librarians showed us a form she uses for systematic reviews. It included the strategy (plus database name and platform used), total numbers and numbers after deduplication, plus a flowchart. Very nice and something I’d like to use in my own practice. We also talked about a blog called PubMed Search Strategies (a blog for sharing strategies), which is a very useful resource, especially if you find a good strategy for a topic you are searching for – no reinventing the wheel!

We might not meet next month because next week is the two workshops led by Julie Glanville, and then there is the holidays. My next post will be writing about those workshops.