When I found out that Andrew Booth from ScHARR was going to be presenting at the IRG Advanced Searching Workshop, that was a decision maker for me. My education plan includes teaching methods for finding qualitative research and since I know Booth is an excellent teacher, I though it a brilliant opportunity. I also know from previous experience that Booth loves acronyms so I had to laugh when I saw the opening slide of his presentation (above). Systematic reviews of qualitative research is increasing, and according to Booth, there are around 12 a month hitting the databases (the search strategy used to determine this interested me: Topic=(“qualitative systematic review” OR “qualitative evidence synthesis” OR “qualitative research synthesis”) OR Topic=(metastudy OR metasynthesis OR “meta synthesis” OR “meta ethnography” OR “meta ethnographic” OR “metaethnography” OR “metaethnographic”) OR Topic=(“systematic review of qualitative”). The main difference between quantitative and qualitative SRs is that while the former seeks to pool numerical results, the latter seeks to find themes or constructs – it is an interpretive exercise which aims to gather insights. Qualitative SRs answer questions such as how and why interventions work or don’t work, what outcomes matter to patients, and what patient experiences of disease are to name a few. The hardest thing about qualitative evidence is appraisal, so I was glad to find out about a method which Booth says is similar to GRADE. CERQual‘s aims to assess how much confidence can be had in the evidence in qualitative reviews. There are four components to this: methodological limitations of the included studies; coherence of the review; relevance of the studies to the research question; and adequacy of the data. Creating a search strategy can use these components as a guide: use methodological filters to find high quality studies; use sampling to retrieve papers that provide a fair representation of the phenomenon; use question mnemonics to guide search strategy formation; and use alternate search strategies to locate similar studies. There are a few question mnemonics to choose from: SPICE, ProPheT, ECLIPSE, SPIDER. It was fun to test these tools with a qualitative research question – one of the really useful workshop activities. Now onto something tricky – searching! The next activity was assessing whether an article was qualitative research or not and if not, why it came up in the retrieval set. I really liked this activity and will copy it in my own sessions – it is one where you can learn a lot from. And of course, there was another acronym to help with identifying qualitative research! ESCAPADE asks what methods, approaches and data were used. Now for qualitative filters. Filters are search strategies that identify papers using specific study designs or publication types and subject terms/free text terms used in high quality studies. Some of them are one liners: MeSH Qualitative Research (though it has some limitations as this subject added in 2003). Many of these filters are available via the ISSG Filter Resource. Booth mentioned the health services research filter available through the PubMed Topic Specific Queries (which I have to say I haven’t looked at often) which includes a qualitative research option. The third activity was an interesting and useful exercise too and one worth reusing: what weakness can you spot in a filter and what is one instance in which a filter could be useful? Next up was the sampling search method. I have a hard time with this because one of the mantras with searching for SRs is comprehensiveness. But this is impossible to achieve in reality for many reasons. SRs address specific questions and have strict inclusion and exclusion criteria and so there are set boundaries – the key is to be exhaustive within these. In qualitative research, the boundaries are more fluid. Is sampling the answer and what is it anyway? Sampling is used to reach data saturation (until information repeats itself and nothing new is gleaned). So instead of having more of the same information, only retain information that explains a concept, confirms interconnections or assists in argument formation. Where can studies be found? Don’t rely on databases – use expansive searches to include footnotes (risk of confirmation bias), citation tracking and snowballing, theses and books and other grey literature. A concept that Booth mentioned was sibling studies – different studies in the same context (I might not have got this right). Clustering is a search strategy technique used for identifying sibling studies. This is a difficult concept for me even though I use some of the techniques in isolation. I guess it is a matter of being a little more systematic (!) in my approach. Phew – it was an intense session which could have gone on all day. Well worth attending – and I encourage you to attend any sessions run by Andrew Booth – they are really good.
And did I do the acronym challenge? I did but stopped 1/2 through the session because there were so many and I was loosing track!
It has been little over a month since the HTAi2015 Annual Meeting in Oslo. I had better start writing about it! I only attended the HTAi Information Resources Group (IRG) pre-conference workshop this time as the conference content wasn’t very relevant to the job I have now. The day after EAHIL, I flew to Oslo and arrived fairly late. The workshop was the next day so I went straight to bed. I was looking forward to this workshop as there were some great presenters lined up and some interesting sessions planned. The workshop was divided into three sessions with mini sessions in the last session. It was a very full-on day but very worthwhile. The first session focused on medical devices, a topic I am not familiar with but needing to know more of. What are medical devices? Well first off, they are not drugs and do not work in a immunologic, pharmacologic, or metabolic action on or in the body, but may be assisted by these. Devices can be machines, software, reagent, instrument, implant, appliance, or apparatus. The market for devices in the UK is 11bn pa, just 2bn under the pharma market, so it is big business. Devices are tightly regulated in that they must work as described by the manufacturer and are safe, so there is a fair amount of information available about these aspects. But what about other sorts of information needed for decision-making? Information about devices is muddied by lots of issues like multiple manufacturers, limited and slow development (long follow-up), difficulties with adverse effects reporting and difficulties with effectiveness trials (RCTs not considered the best study design / small sample sizes). There are also other problems such as multiple device names, limited subject headings available (eg Adverse Events subheading in Medline refers to drugs only) and the fact that a lot of information and evidence about devices is grey literature (and could be further muddied by devices being used for purposes other than the intended one). There are some moves to improve information. One is the AdvanceHTA project which has created a taxonomy for devices. After testing on completed HTAs, it seems to be working. The other is the IDEAL Framework which aims to describe each stage of evaluation and the study designs and reporting frameworks that best suit each stage. There is a series about this in the BMJ. However, the main thrust of this session was how the Scottish Health Technologies Group (SHTG) disseminate information about devices. SHTG have a number of evidence products they produce (freely available but Scotland NHS context) and these are based on questions they receive from topic requests. The resources used to produce thesr reports are local information first, then Cochrane, ECRI, EUNetHTA POP, then primary studies and trials registries.Horizon scanning resources (such as Euroscan) are used for some reports. SHTG is currently evaluating their new Innovative Medical Technology Overview (IMTO) information product. Device manufacturers submit evidence about their products that they want adopted by NHSScotland to the SHTG which then evaluate it. The final product includes economic, device performance, and patient, organisational and safety considerations. These reports are sent directly to managers and procurement departments. Other evidence reports are disseminated to decision and policy makers, and clinicians via a variety of means.
Take home message: as Australian and UK health services are similar, it is worthwhile checking out SHTG reports and including them in your evidence reviews. Also, keep in mind that the evidence base around devices is of lower quality than pharmaceuticals. However, if clinicians at your workplace are conducting trials of surgical interventions, consider alerting them to the IDEAL Framework. I will be notifying the New Technologies committee about this.
I don’t want to get into politics too much but the latest media twitterings about the federal health budget and trimming Medicare (and perhaps the Pharmaceutical Benefits Scheme as well) has me a little annoyed. The thing is that the mechanisms for trimming are already in place. What I am referring to is the federal health technology assessment process and the role of the Medical Services Advisory Board (MSAC), the Pharmaceutical Benefits Advisory Board (PBAC) and the Prostheses List Advisory Committee (PLAC) (replacing the Prostheses and Devices Committee (PDC)). At one of the Health Technology Assessment International (HTAi) conferences I attended, I learnt that the original remit of MSAC was to review all technologies funded by Medicare. But MSAC funding was only able to stretch to reviewing new technologies for approval. I think this is still the case. This means that there is probably a large number of legacy (the word I heard used to describe items in Medicare before the advent of MSAC) technologies still being reimbursed by the federal government that are ineffective and costly. So, why doesn’t the federal government increase funding to MSAC, PBAC and PLAC (and modify the remits) in order to review all technologies reimbursed by government so that ineffective, costly and perhaps unsafe technologies can be removed? It will be a increase in spending in the short term but in the long term, could save money.
I am not an expert in this area (ie – definitely not a health economist!) and am just reporting about what I know and have learnt. If readers have any comments, I would love to read them!