Mobile health or mHealth refers to the use of smart or portable devices that provide services and information on health.¹ Those devices allow access to apps that work on mobile phones, tablets, or computers. Some of their potential benefits include improving access to healthcare services and avoiding medication errors using clinical algorithms. More than 50% of the physicians use them in their daily practice, particularly the drug guidelines and medical calculators (79% and 18% use, respectively).² Therefore, their role in patient safety has been increasing; however, it is also a priority to determine the appropriate monitoring to regulate all the requirements these apps should meet.¹

Drug interactions (DI) have become one of the main challenges in clinical practice worldwide because of the increase of polypharmacy in inpatients and outpatients. They are a significant cause of medication-related adverse events. A patient taking only two medications simultaneously already has a 13% higher risk of experiencing a medication-related adverse event; such risk increases up to 58% in the patients receiving five drugs.³

DIs are challenging to identify during drug development and even during post-marketing surveillance. Thus, for several years, the researchers' main concern —related to drug safety— has been working up methods for identifying and following-up truly harmful DI by analyzing pharmacovigilance data, scientific literature, and even the available information on social networks.⁴

Therefore, this translational research on DI depends, to a large extent, on real-life data from clinical practice. That relies on the information available for health professionals to access knowledge-based clinical decision support systems about interactions, their biologic plausibility, their mechanism of action, and their clinical relevance. Nevertheless, there is no standard golden resource to obtain information about DI; on the contrary, reports about potential interactions between drugs differ among the diverse drug compendia.⁵

Over the years, several studies have analyzed different DI databases (bibliographic, electronic, institutional, and commercial) regarding their functionality, content, transparency, and quality. Since 1978, the need to evaluate the sources of information available to look for interactions between drugs (for instance, the Physician's Desk Reference) was recognized.⁶ More than 30 publications have been written in the last ten years regarding the performance of DI databases.⁷ Still, there is no golden standard consensus instrument to evaluate the tools providing information about DI.^8,9

Despite the variety for assessing and comparing the existing resources providing information about DI, most of the trials concur in claiming that the consistency in the different tools is low (less than 20%).^10,11 Even so, most of the medical existing apps containing information about DI are best-quality.¹²

Several works coincide in claiming a significant variability in how drug compendia report a DI.^9,12-15 In other words, some tools assign the maximum level of severity to a particular interaction, while others do not even register an interaction or that the interaction may be possible.¹⁶ Some authors have compared public databases (open access) with private ones (subscription payment) and have found that the private databases meet the transparency criteria (owner, funding, editorial staff, the process to include and classify the interactions).⁹ However, there is still a high mismatch in the techno-scientific information they all manage, regardless of their accessibility.^9,16

The DI apps mostly compared are Cerner Multum, ClinicalKey Clinical Pharmacology, Drug Interactions: Analysis and Management, DrugBank, Drug Facts & Comparisons, Drugs.com, Epocrates, Lexicomp, Medscape, Micromedex, RxList, Skyscape, and Stockley's Drug Interactions.^9-16 Some studies have compared them based on the information of interactions they provide about specific therapeutic groups. That information confirms their variability and low matching. For instance, Lapoint et al. (2014) evaluated several apps.¹⁷ They found worrying deficiencies in how the apps present the safety warnings of six extended-release and long-acting opioid formulations.¹⁷ Likewise, several widespread DI compendia inaccurately reported plausible and severe interactions between antiepileptic^11,16 and psychotropic drugs.¹⁵

All that is of great concern because the inconsistency among DI apps is a risk factor, resulting in suboptimal prescription practices that may lead to catastrophic consequences, mainly in high-risk medications or patients with higher susceptibility to experiencing adverse drug events.¹⁶

Therefore, the comparative studies of informatics tools providing information about DI are still a need. The analysis of these electronic resources requires continuous updating for several reasons: their technology progresses, the information they contain changes and updates, and the different apps' availability varies according to the geographic region and the access platforms. Then, every day there is more scientific evidence about new DI; thus, the prescription patterns and the drug combinations are modified accordingly. Therefore, there is a need for appropriate databases that respond in real-time to real-world needs.

In Latin America, particularly in Mexico, the hospitals increasingly use international DI apps to make clinical decisions and research on pharmacovigilance, medication errors and patient safety.^18-23 Nonetheless, no published article has compared the available apps to establish their pertinence in the Mexican context. Thus, we posed the following research questions (RQ):

• RQ1: Which mobile applications or databases providing DI information are available in Mexico within the popular app stores?
• RQ2: What features related to structure and content would be the most useful to do a comparative analysis of the ease of use and accuracy of DI apps?
• RQ3: How do the different DI apps compare in consistency when detecting the presence of potential interactions and classifying them based on their severity?

In this paper we carried out an exploratory study to answer these RQs, using a self-designed DI app appraisal tool and a list of real-life DI examples. The aim was evaluating and comparing the structure, content, and consistency of selected DI apps available in Mexico.

Study Design and Setting

An exploratory, cross-sectional, one-time evaluation study was conducted in Mexico during 2020.

Participants

Not applicable.

Interventions

A custom-made evaluation instrument was designed and used to assess the ease of use as well as the type and completeness of the information the ten selected mobile applications provide on drug-drug interactions. It was used to compare the consistency between the selected DI apps when reporting potential interactions and classifying them according to their severity.

A list of 22 real-life examples of potentially interacting pairs of drugs commonly prescribed to hospitalized geriatric patients was used.

Operational Definitions for this Study

• Structure. The set of technical attributes of a mobile application related to its characteristics, functionalities and ease of use (usability) of its system interface.
• Content. The set of main clinical information elements needed to properly evaluate a DI based on current and verifiable evidence.
• Consistency. The ability of a DI app to detect a potential drug interaction and categorize its severity accordingly.

The study was developed in three phases:

1. identification and selection of drug interaction apps,
2. apps evaluation according to content and structure, and
3. assessment of the consistency of the interaction reports.

Identification and Selection of Potentially Relevant Drug Interaction Apps

A preliminary list was made of commercial mobile apps available in Mexico based on an online search using keywords related to "drug interactions" and a search on the iOS and Android systems' app stores. Apps and electronic databases —referred to or analyzed in previously published papers found through the same keywords— were also included.

The DI software applications were eligible for the final list for analysis if:

1. they were either accessible online from a desktop computer or through a smartphone app;
2. they were available in English or Spanish; and
3. they addressed to health professionals.

The apps excluded from the abovementioned list were the following:

• apps requiring a subscription payment;
• apps with no temporary free access or a trial period;
• those apps that offered a free trial period but asked for a credit card when activated, discount coupons or PayPal account;
• the ones with restricted access; or
• those addressed only to patients.

Apps Evaluation According to Content and Structure

A quick literature review was carried out to identify articles containing tools or lists of criteria for DI app-quality assessment. A PubMed search was conducted using the terms ("Drug Interactions"[Mesh]) AND ((("Drug Information Services/standards"[Mesh]) OR ("Databases, Factual/standards"[Mesh])) OR ("Mobile Applications"[Mesh])) Filters: from 2015 - 2021. Full texts of the retrieved articles (n=21) were reviewed to extract potentially relevant assessment criteria for apps targeting DI.

Due to the wide heterogeneity found in assessment criteria for DI apps in different studies, the authors of this study designed their own tool to appraise the structure and content of the previously selected DI apps based on several papers.^{9,12,14,24-26} The final appraisal tool included nine "content" criteria and eleven criteria related to "structure" (see Table 1).

A four-level Likert scale for frequency was used to evaluate the "content" criteria, grading them on a scale from yes (indicating no deficiencies) to no (pointing major deficiencies):

• Yes. Content related to the criterion is every time thorough, comprehensive and/or complete;
• Yes, most of the time. Content related to the criterion is often thorough/comprehensive and/or complete;
• No, most of the time. Content related to the criterion is rarely thorough/comprehensive and/or complete;
• No. Content related to the criterion is never thorough/comprehensive and/or complete.

For the rubric evaluating "structure" criteria, usability was assessed on three levels (efficient, regular, and low); while the rest of the criteria was dicotomically classified as "yes" (meet the criteria) or "no" (did not meet the criteria).

Assessment of the Consistency of the Interaction Reports

The DI apps' consistency was compared based on the number of potential drug-drug interaction alerts they generated and the severity level assigned to a group of 22 real-life examples of interacting drug pairs. Every drug-drug interaction reported had a point. The dyads were chosen from the potential interactions found more frequently during the medication reconciliation and prescription validation processes to hospitalized geriatric patients in a hospital in Puebla City, Mexico (unpublished data). The pairs of drugs used to measure the consistency in the apps' interaction report were the following:

1. Aspirin - Enoxaparin
2. Biphosphate, sodium - Telmisartan
3. Clarithromycin - Levofloxacin
4. Clopidogrel - Enoxaparin
5. Clopidogrel - Omeprazole
6. Esomeprazole - Glimepiride
7. Calcium gluconate - Ceftriaxone
8. Hydrocortisone - Levofloxacin
9. Ketorolac - Enoxaparin
10. Ketorolac - Ketoprofen
11. Ketorolac - Levofloxacin
12. Ketorolac - Ranitidine
13. Metronidazole - Fluoxetine
14. Propranolol - Sucralfate
15. Rivaroxaban - Voriconazole
16. Rosuvastatin - Fluconazole
17. Tacrolimus - Fluconazole
18. Tramadol - Levofloxacin
19. Tramadol - Metoclopramide
20. Tramadol - Ondansetron
21. Omeprazole - Diazepam
22. Omeprazole - Methotrexate

Data Analysis

The authors DAVP, AIHM, and LSVG independently evaluated the extent to which the apps met every "structure" and "content" criterion. The author LICP drafted the consensus and solved disagreements together with the other authors. Evaluation of these criteria was qualitative.

To assess "consistency", all authors independently reviewed the content of each drug-drug interaction pair, and disagreements were discussed until a consensus was reached. Basic descriptive statistics were used to calculate the scores. As described by Morte-Romea et al.²⁷, the "sensitivity" was defined as the capability of the DI app to detect the 22 selected drug-drug interactions, expressed as a percentage.

Ethics

There was no need for Ethics Committee approval.

DI Apps Selection

The initial search identified 24 potential mobile applications that may be considered in this study. After applying the inclusion and exclusion criteria, ten out of them were included in this analysis, and 14 were discarded.

The following applications were excluded from further evaluation due to limitations in their accessibility or because of their approach: BOT PLUS, Critical-Medical Guide, Davis's Drug Guide, DRUGBANK Drug-Drug Interaction Checker, Drug Interactions Facts, GenieMD, Hansten and Horn Drug Interactions, HealthTap, IBM Micromedex Drug Interactions, IBM Micromedex Drug Reference, Lexicomp Lexi-Interact, Medinteract.net, Stockley's Drug Interactions, and ZibdyHealth.

The following ten applications formed the final list chosen for analysis: Drugs.com, Epocrates, Guía Farmacológica 061 Andalucía, iDoctus, Medicines Complete (AHFS Drug Information), Medicines Complete (Martindale: The Complete Drug Reference), Medscape, UpToDate, Vidal Vademecum, and WebMD.

It is noteworthy to mention that we had complete access to Medicines Complete (AHFS and Martindale) and Vidal Vademecum because our university has a subscription to these electronic tools. Likewise, Hospital Ángeles Puebla allows us to use the UpToDate application in its facilities; such resources require a subscription payment. We would not have had access to these applications without a payed subscription.

DI Apps Structure and Content

Table 2 shows the evaluation of the apps based on their structure. Eight applications were graded as poor usability. UpToDate was considered regular, even though it presents the information in Spanish. Drugs.com has a more efficient platform and meets every single of the 11 structure criteria.

Table 3 shows the evaluation of the content criteria. Most applications provide information about the type of interaction, except for Epocrates and Guía Farmacológica 061 Andalucía tools.

The most unmet criterion was the differentiation of the data; that is to say, the provided information is consistent and complete, but the language and approach are different, depending on the user who searches (health professional, patient, consumer). Only Drugs.com met this criterion.

Most of the applications did not meet total or partially the requirements related to the data provision about other interactions (for instance, with food, beverages, or herbal medicines) and bibliographic sources that support the presented information. The applications that best met the content criteria were Drugs.com (with 9/9) and UpToDate (6/9 criteria). Epocrates did not wholly or partially meet most of the requirements related to content.

Additionally, Sociedad Española de Farmacia Hospitalaria (SEFH) guidelines for drug interactions (published in a PDF format)²⁸ were analyzed because they are available in Spanish and are widely quoted in publications about drug interactions in Spain. As it is an electronic book, it was evaluated differently. Their usability was graded as regular. It does not allow an automatic search; it only offers revision by chapters because the guidelines are organized according to therapeutic groups. It is open access and fully met four criteria related to content:

1. it describes the mechanism of action of the interaction;
2. it provides examples of cases;
3. it gives additional information as interactions with food and beverages, and
4. it offers the complete bibliographic sources that support the document.

Nevertheless, the guidelines do not classify the interactions according to their severity or differentiate between information addressed for health professionals versus patients.

Besides, the guidelines do not entirely meet the rest of the criteria for all the interactions they report. They include information about potential interactions for 23 active ingredients out of 30 established in the list of validation of 22 pairs of drugs with possible interaction.

However, we only found the 2014 edition and no later updating. The SEFH guidelines were drawn up along with Medinteract.net, a tool exclusively available for SEFH members, and that requires a payment to access from the website of this electronic database.

DI Apps Consistency

Table 4 shows the consistency evaluation of 9 out of 10 mobile applications. It reports the potential interactions of 22 pairs of drugs commonly prescribed to geriatric patients in a Mexican hospital (unpublished data, personal communication).

Guía Farmacológica 061 Andalucía was evaluated differently because it does not have an automatic search button for interactions. It only provides drug monographs for each drug, and in the "interactions" section reports the therapeutic groups and some specific drugs with a potential interaction. In this tool, we only found information sheets about drug interactions for 11 out of 30 of the active ingredients in the list of the 22 pairs of drugs with potential interaction. These guidelines are adapted to the pre-hospital emergency setting at Consejería de Salud de la Junta de Andalucía, Spain; thus, the number of included drugs is delimited.

When validating every single of the 22 drug interactions, Drugs.com was the only tool with all the reported interactions, followed by UpToDate and Epocrates, with 15 out of 22 pairs of drugs reported with potential interactions (see Table 4). The most significant inconsistency among the evaluated applications is how they classify the 22 possible interactions based on clinical significance or severity.

In general terms, Epocrates, Guía Farmacológica 061 Andalucía, iDoctus, Medicines Complete (Martindale), Medicines Complete (AHFS), Medscape, Vidal Vademecum, WebMD, and Guía de la Sociedad Española de Farmacia Hospitalaria presented a larger number of non-compliances with the analyzed content and structure criteria; they also reported a lower number of searched drug interactions.

DI analysis is vital when reviewing the prescriptions' appropriateness. It is a safety barrier in inpatient care because, undoubtedly, the patients deserve that healthcare professionals observe the optimal information to give them safe and adequate care.

In this document, we present an analysis of 10 DI apps that may guide students and health professionals to choose the most convenient tool for them. It may also help the institutions ensure access to the most efficient apps in clinical decision-making.

Some applications may become popular among health professionals because of their accessibility and convenience of use. Still, it is crucial to know whether the information the applications provide is useful in the clinical setting.

Some of the technical advantages of most of the analyzed applications were:

• they have a website adaptable to tablets and smartphones formats;
• they have mobile applications, available in iOS and Android operative systems;
• they have search filters, and when typing the interest drugs, they automatically show the interactions or have a specific section for interactions, except for Guía Farmacológica 061 Andalucía.

Although usability is maybe a subjective topic, our analysis found that most DI applications got a low grade in this criterion due to the many "clicks" or steps required to browse the platform and to be able to find all the information about DI (see Table 2).

The languages of the available analyzed apps are also an essential criterion because even Spanish-speaking health professionals with a vast knowledge of a second language (for example, English) may misunderstand a specific concept or meaning. Thus, he or she may be at risk of making a medication error. There are many studies reporting medication errors derived from language barriers between healthcare providers and patients. Nevertheless, we did not find investigations enhancing the implications when the health professionals consult drug information in a language different from theirs, which may cause medication errors.

Concerning the content, it is somewhat hazardous that most of the analyzed applications do not provide the bibliographic sources they use to support the information about potential DI. Most of them lack information about interactions with food, beverages, nutritional supplements, alternative medicines, or herbal medicines. Only Drugs.com and the Spanish guidelines (Guía Farmacológica 061 Andalucía and SEFH guidelines) have such additional information; however, it is restricted to some food, beverages, and plants.

Most of the information the DI apps provide is deficient regarding the case examples or studies previously reported about the interaction, the groups of patients vulnerable to the different DI, and the differentiation of the information addressed to healthcare professionals versus those provided to patients and consumers. This latter is crucial and valuable for healthcare practitioners because it helps educate the patient in more simple terms and highlight what the patient must monitor and report to the caregiver to take timely precautions in case of an adverse event.

It is worrying that only half of the applications classify the interactions based on the severity and describe the recommended clinical management. The tremendous significant variability in the information provided about the interaction mechanism is also consequential. These three scientific elements are quite relevant to facilitate clinical decision-making, above all, in the hospital setting. Applications like Epocrates, Medicines Complete (Martindale and AHFS), Guía Farmacológica 061 Andalucía, and SEFH guidelines, not even report a severity grade of the DI.

Some studies reported that when electronic support systems for prescription are nearby, the physicians cancel the drugs' safety alert between 49% and 96% of the cases.¹⁷ It has also been reported that it is more likely that doctors pay heed to medication alerts showing more severity. Still, it is unlikely that doctors simultaneously administer drugs with high-risk interactions.⁵ Therefore, the recommendation is that the clinical support systems prioritize high-risk drug interactions to avoid an excessive number of alerts overwhelms the professionals about potential harm. That will also prevent ignoring the application.⁴

Since prescription errors rank first place in the causes of medication-related adverse events, the drug information compendia must allow immediate access to accurate data, presenting clearly and efficiently the necessary warnings to the prescriptor. Such information must be based on evidence to ensure the pharmacotherapy is safe.¹⁷

The evaluation of the 22 pairs of drugs indicated Drugs.com is a helpful application because it reports potential interactions and includes all the information. Thus, this "open access" database is at the head of the list because it is free and establishes interaction severity, type, mechanism, groups of patients at risk, and clinical management. It also lists the bibliographic references supporting the data and offers information for healthcare professionals, patients, and consumers.

One of the main restrictions, above all in Mexico, is the availability of payment informatics tools, mainly in public hospitals. Regarding electronic resources about drug information, some institutions —mostly national reference hospitals— have access on an ad hoc basis to commercial data, whose quality is supported by internationally renown publishers. For instance, Centro Médico Nacional "Siglo XXI" (CMN SXXI) from Instituto Mexicano del Seguro Social (IMSS) has access to Micromedex and has used it to measure the prescription frequency of drug combinations with potential interactions in inpatients in internal medicine¹⁸ and to determine the association between the presence of possible interactions and the mortality rate of hospitalized elderly patients.¹⁹ Other researchers from CMN SXXI have also used the drug interaction app Stockley's and Hansten and the bibliographic tool Tatro Drug Interactions to investigate clinically relevant drug-drug interactions in the elderly.²¹ It has been reported that other Mexican reference hospitals have also used Micromedex to search for the most prevalent severe interactions in older adults with dementia²⁰ and even in family medicine centers from IMSS to identify potential interactions in outpatients > 50 years old.²⁹ Other tools like Medscape and Drugs.com have also been used to determine the prevalence of possible DI in pediatric patients admitted to the Emergency Department²³ and in inpatients with schizophrenia²², respectively.

Micromedex was not included in this study because we did not have a subscription and, in general, is one of the most expensive resources of drug information in the country. Therefore, it is good news that a free access application has obtained the highest compliance of the criteria evaluated in this study.

Nevertheless, Drugs.com has two disadvantages, at least for its optimal use in Mexico: the language and country of origin. This latter significantly influences because Drugs.com does not include monographs or information about DI for some drugs used in Mexico (but not in the USA). The same happens with the other analyzed applications, except for Vidal Vademecum, because it uses Mexico's tropicalized version.

Drugs.com was the only drug compendia reporting interactions between esomeprazole-glimepiride, ketorolac-ranitidine, and propranolol-sucralfate. Even though the interactions are reported as mild, information is supported by bibliographic sources and a detailed description of the interaction mechanism. This database's advantage is that it provides complete information and classifies the interactions, facilitating decision-making in real-time.

Clopidogrel-omeprazole (severe), calcium gluconate-ceftriaxone (severe/contraindicated), and ketorolac-ketoprofen (severe) were the only three pairs of drugs that drug compendia concur about their severity. However, there were applications that not even mentioned the existence of potential interactions among those pairs.

Finally, the most outstanding inconsistencies were observed for the pairs hydrocortisone-levofloxacin, tramadol-ondansetron, omeprazole-methotrexate, and omeprazole-diazepam, because some applications classified them as severe potential interactions, others as moderate or mild; while others not even reported them.

Due to the diversity of studies comparing the different applications and the lack of a standardized instrument to evaluate them, it is impossible to grade one or another as definitely better. Consequently, more studies are required to assess the different drug compendia's implications and suggest standardized instruments and evaluation criteria. Several experts panels have agreed that the crucial elements for clinical decision support tools and healthcare professionals to improve patients' care are: mechanism of action of the interaction is consistent with the drug plausibility, severity of the potential interaction, its impact on the patient, susceptibility factors related to the groups of patients that might be affected, and the recommended strategy for the management of the interaction.^25,30

Several published guidelines focused on making a standardized report of DI may be useful for hospitals. In other words, based on one or several mobile applications, the hospitals may develop internal guidelines relevant to their daily activities and needs.^26,30 Undoubtedly, to avoid adverse events in daily clinical practice when using the applications, there must be particular interest in healthcare professionals' educational and training programs, specifically in enhancing digital health literacy.¹ Furthermore, pharmacists should participate in the design, specification of criteria, and evaluation of the information technologies to be implemented in the medication hospital processes.

In Mexico, electronic medical records and computerized prescriber-order-entry systems are still emerging. Thus, accessible, comprehensive, and reliable drug compendia have a crucial role in safe medical care. Therefore, the accessibility to these tools enables the latest training for future health professionals.

Our findings show that open access DI apps are not necessarily incomplete or imprecise with respect to paid databases, as previously reported⁹, which is good news for hospitals and health professionals who do not have access to paid databases in their institutions or privately.

Even though this document only assessed a subset of all the available mobile applications, the sample includes popular resources well-known in Mexico as reference guidelines on DI. Thus, our exploratory analysis must be regarded cautiously due to the subjectivity of some items we used to evaluate and the exclusion of some other known but not free drug compendia.

Nevertheless, studies like ours are always useful for the health professional who looks for the existing platforms' weaknesses and strengths and ponders to include one or some of them in their daily clinical practice.