FAQs

Who is ORCHA?

As the world's leading independent health and care app evaluation and distribution organization, ORCHA helps governments and health and social care organizations to choose and deliver quality- assured apps. ORCHA's vision is to revolutionize care through the safe integration of digital health solutions into all aspects of health and care services, leading to more patient-centered, effective care. For more information, please visit ORCHA's website here.

What are the criteria underlying the assessment process?

The Assessment Process is based on criteria selected by ORCHA, the American College of Physicians, and the American Telemedicine Association, drawing from digital health standards and best practices in the US and globally. The detailed framework is described here.

Who carries out your Assessments?

Our Assessors are recruited and trained by ORCHA from a wide array of backgrounds and roles. They are not experts in any particular field of App assessment but rapidly become experts in interrogating Apps to answer the questions posed by our Assessor Development Team who are all experts in the relevant areas such as clinical, technical, regulation, user experience and design.

More complex apps also go through an Enhanced Clinical Assessment, which is conducted by physicians or other qualified healthcare professionals.

How does the Assessment Process work?

Our Assessment involves an evaluation of over 350+ elements of an App, and validates crucial information, such as compliance, with relevant Data and Privacy regulations and Medical Device regulations; it also assesses an App against a range of other areas of standards and best practice.

The ORCHA Assessor Development Team have translated the evaluation criteria into a set of objective questions that the Assessors are looking to answer. They set out very clearly what evidence the Assessors need to find to answer a question affirmatively, and they also determine the consequences of the answer to each question in terms of positive or negative points being awarded, which drives our scoring process.

Our Assessors are guided through each assessment through our online review tool, and this ensures that all the relevant questions are investigated. Where our Assessors encounter scenarios that the Assessor Development Team havent specified,the Assessors refer these back to the relevant member of the Assessor Development Team for guidance.

For the ACP app library, Apps that are classified as Tiers Bii, Ci and Cii all go through an Enhanced Clinical Assessment process, conducted by physicians and other healthcare professionals with expertise related to the App category. Through this process, the content sources, published evidence and use cases are reviewed,and the clinical assessor generates a subjective summary of the app.

For some Apps, there is a further level of scrutiny applied if they are classified as Medical Devices, although the level of additional scrutiny here is still dependent on the level of classification they reach. Class i medical devices, for example, are largely based on a self-certification approach.

Even Apps that have undertaken and published clinical trials or equivalent studies, and have been subjected to relevant peer review, while clearly showing a great commitment to establishing their clinical credentials and efficacy, may not properly comply with other crucial aspects of an Apps overall delivery,such as data security or user configuration etc., which we also evaluate.

Against this backdrop and in this context, we believe our process is as robust as it is possible to be without undertaking a much more detailed and costly evaluation that would limit the range and scope of Apps we can consider. We also believe and constantly monitor and test our process around this that the combination of the measures we consider does give a very accurate view of an App, which usually aligns to the view that other,more detailed assessments deliver, and is as such a reliable overall indicator of an Apps quality.

How do you assess directly whether an App works?

Our assessment is not the same as a full user test of an App, and it doesnt test all of the claims an App Developer might make about what the App can or might do. While for some Apps this may not be a particularly onerous task, for others this would involve a huge amount of testing and trials to validate all aspects of safety and effectiveness. Please review the published evidence linked from the assessment report for reviews of the effectiveness of an App.

For some Apps that fall within the definition of medical devices, this is something that they will potentially undertake as part of their certification as a medical device where dependent on their classification they will need to provide evidence to a Notified Body (such as the Food and Drug Administration) of the Apps compliance with specified standards.

Our Assessments aim to rapidly assess an Apps overall compliance levels with regulation, standards and best practice, to provide a proxy view of whether it is something to engage with or not. High levels of compliance do tend to indicate a higher level of quality in the end product, and vice versa.

How do you keep your Assessments up to date?

We monitor all the Apps available on the iTunes platform and Google Play on a weekly basis, and this tracks both new Apps that have come to market and also updates and new versions of apps that are already available. Where we detect an update or new version of an App that we have not already assessed, we automatically flag the existing assessment with a notification that it relates to a prior version, and we put the new version of the App back into our Review Queue to be reassessed. It is important to only rely on Assessments that relate to current versions of an App as updates and new versions can materially change an Apps performance, risks and issues.

How do you maintain independence from industry influence? Do you get paid for a review?

We are compensated primarily by those who seek access to the app library, not by the app developers .Developers can pay a nominal amount to see their review in advance of publication.

They can also pay a small fee to be reviewed earlier than our schedule. But this does not influence our review, as our approach is documented and is entirely objective with no room for subjectivity.

How is the DHAF scoring determined?

Questions within the DHAF are scored, earning the app positive points or negative points depending on the nature of the question and the response. Each Review Domain (set of questions) produces a Domain score. These Review Domain scores are then combined to create an overall DHAF percentage score. Only apps with a score of 65% or higher are included in the ACP App Library. Any score below 65% would indicate that an App has some issues that users should investigate further prior to using this App. Scores below 45% indicate that an App has considerable issues or challenges and in its current form is potentially unhelpful or unsafe. This scoring process aims to deliver an objective and unbiased evaluation of all Apps so they are treated fairly irrespective of their current popularity or financial position.

Can any app that scores above the 65% threshold be recommended by physicians?

Apps which score 65% or above on the DHAF have been identified as Apps which follow data privacy laws, have evidence and/or assurance of the Apps efficacy and/or safety and meet recommended usability/accessibility standards. That said, a physician should not just recommend apps based on this scoring. It is up to the physician's judgement as to whether they believe a certain App is appropriate and useful for a patient. We recommend a physician reviews the content of the app alongside the DHAF results to help determine if the App will meet their patients needs.