The surgical field now has seen a major development. After Virtual and Augmented reality, now comes Mixed reality in the form of HoloLens project. This headset from Microsoft is thought to bring a revolutionary change in the fields of surgery across the globe.
Back in 2017, Microsoft’s HoloLens 2 mixed reality headset was used to operate a patient with a collarbone fracture. There were joint orthopedic surgeons, Professor Thomas Gregory from Paris and Dr. John Erickson from New Jersey. By performing shoulder arthroscopy the bone was repositioned. This process of arthroscopy involves placing a small camera in the joints to determine what is causing the patient’s shoulder pain.
Both the surgeons, Gregory and Erickson were connected to Gobbato’s headset via the “Microsoft Dynamics 365 Remote Assist” application, and the view was shared on the computer screen through “Microsoft Teams”. The patient and the holographic images generated by Gobbato through CT were visible. Among them, one showed the damaged clavicle of the patient, and another showing his healthy clavicle. Three surgeons from 3 different continents discussed how to approach the procedure, consulting each step and sharing each bone’s approaches.
This procedure was part of a HoloLens project that focused on showing how doctors can be benefited from HoloLens 2, and how doctors from all over the world collaborate for work. Initiated by Gregory, who did his first surgery with the use of HoloLens 2. This edge included doctors and surgeons from 13 countries on 5 continents.
12 surgeons between November to January used HoloLens projects for performing a surgery that ranged from a knee procedure in the United Arab Emirates to a shoulder replacement in South Africa. While this was done others joined by Gregory took turns in observing the output. This project today culminates with a chain of roundtable discussions on the contribution of mixed reality in the health care sector.
For Gregory, this was a medium to convey the benefits of what he calls as “the smartphone for surgeons.” Many people throughout the globe access smartphones to receive information on various niches, but for surgeons that are not the case. The surgeons have little access to the patient’s data inside an operation setup.
The first HoloLens project was introduced in 2016. From that time Gregory saw it as a potential instrument for surgeons to use it for communication. All of a sudden he was able to use hand gestures and voice commands to call for information without touching the keyboard or mouse.
He is the chief of the “Department of Orthopedic and Trauma Surgery” at Avicenne Hospital in Paris. Along with that, he is also a professor of orthopedic surgery at the University Sorbonne – Paris – Nord.
The HoloLens can be operated with hand gestures and also voice commands. This allows the surgeons to get a view of the three dimensional holographic images of the patient’s body that have been created from X ray or other scans. Those virtual images can be moved by the surgeons to get the view from different angles. Along with that, they can even access the patient’s data while the surgery is on and can even ask for other documents or contact other specialists during the surgery.
Dr. John Sledge On The HoloLens Project For Spinal Fusion Surgery
Dr. John Sledge, an orthopedic surgeon in Lafayette, Louisiana performed a 10 hours spinal fusion surgery with the HoloLens project. Sledge carried up X rays and scans from the lower back portion of the patient to discover the pieces of hardware that needed to be removed and find out what is the best position for his instrument to access them. He shortly loaded the images of his patient to explain the preparatory history to other surgeons.
Sledge confesses that without the HoloLens project he would have been limited to some of the things from the computer screen or the patient’s scan.
Along with that the HoloLens project is also useful if the surgeon is planning the surgery or even for training, says Sledge. For example, there can be instances where with a holographic illustration of a patient’s shoulder the surgeon can determine their next step.
Sledge also adds that despite printing a 3D image of the bone, he can use the Hologram to practice the operation on this. For training, three dimensional interactive models can be made of any surgical procedure.
Dr. Stefan Greiner’s Experience With The HoloLens Project
Orthopedic surgeon Dr. Stefan Greiner had never used the HoloLens project before. But he had agreed to participate with Gregory in the project as he is his good friend. Initially, Greiner was doubtful about the idea. He thought this device to be like any other device that will be used in rare cases.
But once he learned the use of it he started to see more possibilities of the same. He uses the device to see images while performing any orthopedic surgery to avoid looking up and down where the images are shown on the computer screen.
Dr. Stefan Grenier ruins the “Department of Shoulder and Elbow” at Sporthapaedicum Regensburg which is one of the largest sports medicine facilities in Germany.
This device can also be used to elaborate and prepare patients for surgery by making them look at the holograms or animations.
Dr. Ashish Babhulkar After Using HoloLens 2 By Microsoft
Another instance just like Dr. Stefan Greiner is Dr. Ashish Babhulkar. He also had never used HoloLens before he signed up for Dr. Gregory’s project. Dr. Ashish Babhulkar is the head of the Department of Shoulder and Sports Injuries at Deenanath Mangeshkar Hospital and Research Center in Pune, India. He did two surgeries of reverse shoulder replacement with Dr. Gregory and an orthopedic surgeon from the U.K.
He further says that his experience with HoloLens has convinced him to believe that this will go long in the world of medicine.
About The Device
The HoloLens itself and its forerunner, the “Kinect line of motion” – sensing devices, both were created by Microsoft Technical Fellow Alex Kipman. He is an engineer who knows the need for devices that will merge the physical world with the digital sphere in a three dimensional way.
The very first HoloLens was launched in 2016, and then came HoloLens 2 in November 2019. This new device is well equipped with “sophisticated environmental mapping hardware” and an “eye tracking camera” that allows it to know the space that is being used and also the focal point of the user. An application known as Microsoft Dynamics 365 Remote Assist was designed that allowed HoloLens to connect with Microsoft Team so that users can connect to people all over the globe.
A decade ago it would have been impossible to think about surgeries using HoloLens and the use of mixed reality technology. But today in 2021 HoloLens is being used in every field of medicine. It is used from “virtual hospital rounds” for medical students to treating Covid 19 patients.
Elena Bonfiglioli, Microsoft’s managing director and regional lead for “Healthcare and Life Sciences” in Europe and the Middle East says that the use of HoloLens in surgeries is new and has been propelled by Gregory and other pioneering physicians all over the globe.
The First Ever User Of HoloLens: Dr. Thomas Gregory
Gregory’s department has been using mixed reality technology for almost a year now and has used that for almost all his surgeries. He is even called by his younger colleagues for assistance regarding the HoloLens projects.
Microsoft partnered with Gregory in January 2020 to launch their Institut Moveo, which is a center for research and teaching that is completely dedicated to artificial intelligence in surgery. This center is located at the “University Sorbonne – Paris – Nord. This accommodates scientists, surgeons, and also mathematicians to provide training on HoloLens surgery.
The future plans of the HoloLens project include a collection of HoloLens surgery data and building an algorithm based on that. Gregory views HoloLens as an instrument that can be used at a much lower cost than a “surgical robot” used for surgery and can make AI in surgery accessible for all doctors.