How did the Mayo Clinic choose a vendor for AI in pathology?

Artificial intelligence (AI) is considered a major technology development for anatomic pathology. More and more clinics are now designing their platform requirements in order to integrate AI into their digital pathology workflows as it is expected to elevate pathologists' diagnostic capabilities, predict outcomes, and suggest therapeutic options.^2-5

A recent publication, "Democratizing Artificial Intelligence in Anatomic Pathology", by Thomas J. Flotte, MD et al., describes how one of the top-ranked hospitals in the world, the Mayo Clinic, selected Aiforia as a vendor for AI-powered pathology image analysis.

Mayo Clinic’s 6 criteria for platform providers

To create an ecosystem that permits everyone to become involved in developing AI algorithms in anatomic pathology, the Mayo Clinic defined six key criteria for evaluating potential platform vendors:

1. Approachable and intuitive: The platform must be designed for users with a wide range of technical expertise and interest. It is important that entry-level users can easily use the platform.

2. Variety of modeling options: The platform will be used for a large variety of experiments, and therefore, it needs to be flexible and accommodate many different approaches. The modeling capabilities must include rare event detection, segmentation, individual feature and spatial quantitation, classification, and prognostication. The platform provider must also show evidence of prior successful AI model development from many different researchers with different approaches and endpoints.

3. Internal and external collaboration: The users may want to collaborate with colleagues within or outside their own institution. The platform needs to permit validated users outside the firewall to access the program.

4. The platform must meet the minimum requirements of the IT and security groups: The IT and information security teams assessed how easily a vendor’s solution integrated into Mayo Clinic’s cloud platform and what vulnerabilities existed.

5. Seamless transition from discovery to clinical deployment: The platform needs to support both the discovery, that is, the development of algorithms, and deployment in the clinical workflow.  It would be very frustrating for a user to develop a great algorithm and not have an opportunity to use it in practice.

6. Platform scalability over time: Although the pilot starts with a limited number of users, it is important that the platform scales to a more significant number of users in the future.

After a careful review of these criteria by pathologists, IT, and security teams, Aiforia was selected as a vendor for AI-powered pathology image analysis both for clinical workflow and translational research.

How were the pathologists supported and engaged throughout the implementation process?

A comprehensive AI ecosystem was created to aid the implementation and AI development, supporting pathologists with varying levels of expertise. The AI ecosystem received great feedback from the users, and it was proven to break down entry barriers, reduce the overall cost of AI development, improve AI quality, and enhance the speed of adoption.¹  Cornerstones were the following:

Training and onboarding plan: An education committee consisting of Aiforia, the Department of Laboratory Medicine and Pathology, and institutional education group representatives created a training plan for onboarding the platform users and provided ongoing support. Aiforia was responsible for designing and implementing an onboarding curriculum that consisted of training sessions, full-length lecture recordings, FAQs, and short educational videos illustrating specific common tasks. An institutional community website was created to house all the materials, including a discussion forum and a ticketing system for user support. 

Training instructors from Aiforia: Aiforia’s support team regularly held open office hours, providing the users with an opportunity to ask questions, receive personal guidance, discuss and resolve barriers, and learn from each other.  Instructors were also available for one-on-one tutorial sessions with users when needed. 

Support resources from Mayo Clinic: A cytotechnologist with extensive experience in AI was nominated as an Aiforia® Create superuser. The superuser was available to guide and support users in AI model development, for example, on how to annotate images effectively. A data scientist was assigned to assist users with design and data interpretation, and a project manager provided logistic support, communication management, risk and problem resolution, and metric tracking. Mayo Clinic’s IT group resolved all incoming community tickets regarding the institutional cloud.

Results so far: From discovery to clinical deployment

Based on the article¹, 84 users from the Mayo Clinic completed the training in the years 2022–2023. A total of 30 out of 31 projects progressed through the model development process of annotating, training, and validation. 15 abstracts were submitted to national meetings, 13 of those to the USCAP 2024. 

See more USCAP-related content here:

• List of poster presentations utilizing the Aiforia® Platform at USCAP 2024 →
  
• Case study: Developing an AI model to determine invasion in pulmonary adenocarcinoma →

The first clinical application deployed was Aiforia’s Ki67 AI model for breast cancer diagnostics. The analysis of patient samples started in March 2023. There are currently three tests in the Mayo Clinic’s diagnostic test portfolio using Aiforia.

Another interesting result of the collaboration between Mayo Clinic and Aiforia is QuantCRC - a prognostic AI model that identifies important histological features of colorectal cancer and provides a recurrence prediction estimate useful for treatment decisions. Watch our on-demand webinar to hear more about this exciting use case and learn how prognostic AI models are revolutionizing the future of pathology image analysis.

References

1. Flotte et al. (2024). Democratizing Artificial Intelligence in Anatomic Pathology. Arch Pathol Lab Med. https://doi.org/10.5858/arpa.2023-0205-OA
2. Flotte, T. & Bell, D. (2018). Anatomical pathology is at a crossroads. Pathology, 50(4), 373–374. https://doi.org/10.1016/j.pathol.2018.01.003 
3. Cohen, M. (2022). Practical Linear Algebra for Data Science (1st ed.). O’Reilly Media.
4. Bera et al. (2019). Artificial intelligence in digital pathology–new tools for diagnosis and precision oncology. Nat Rev Clin Oncol., 16(11), 703–715. https://doi.org/10.1038/s41571-019-0252-y 
5. Niazi et al. (2019). Digital pathology and artificial intelligence. Lancet Oncol., 20(5), e253–e261. https://doi.org/10.1016/s1470-2045(19)30154-8