H4H Final Project

Laparoscopic Surgery

⧉ Benefits and Problems

Over the past decades, intra-abdominal minimally invasive surgery (MIS), which is also known as laparoscopic surgery, has progressively become more popular than open surgery owing to its low risks and small incisions. Current laproscopic cameras commonly provide full HD vision of the scene in 60fps, some even provide 2K and even 4K resolution video. Yet, this kind of intervention introduced a loss of direct vision and tactile feedback upon the scene for the surgeon. Furthermore, it provides only a surface view of the organs and cannot show anatomical structures and surgical targets located beneath the exposed organ surfaces.

⧉ Supplementary Images

There are some other visual assisted data that are commonly used during the sugery together with endoscopic images.

• Preoperative images: MRI and CT present a good resolution and image quality. Since they are acquired prior to the intervention, they are often digitally processed
• Intraoperative images: CBCT, open MR scanners, and 2D US provides live but worse resolution quality images.

MRI is a medical imaging technique that uses a magnetic field and computer-generated radio waves to create detailed images of the organs and tissues. Compared to a CT scan, MRIs provide more detailed information about the inner organs (soft tissues) such as the brain, skeletal system, reproductive system and other organ systems. For intra-abdominal surgeries, MRI scans are usually obtained before the operation and can be pre-processed for special needs.

Laparoscopic Augmented Reality

Augmented reality (AR) has been introduced to various disciplines of MIS in the last 20 years. It overlays the assisting information on the live laparoscopic video and appears to be a viable solution and appears to be a viable solution to alleviate the problem of lacking feedback.

However, in abdominal MIS, correctly augmenting a laparoscopic scene remains challenging, due to the non-rigidity of abdominal tissues and organs.

Target

The purpose of this work was to prototype a laparoscopic Augmented Reality system with preoperative Magnetic Resonance Imaging (MRI) and intraoperative laparoscopic ultrasound(LUS).

Components

The system can be divided into pre-operative stage and intra-operative stage.

⧉ Preoperative Stage

• Take abdominal MRI scans: patients are supposed to take their abdominal MRI scans and get the best quality view of the volume
• Preprocess and analyze MRI volume while have the surgical plans: doctors can tune the MRI volume to get the best view and modify the MRI data while discussing the surgical plans together. After this step, potential sugical trajectories should be marked and saved for later use.
• Talk about the surgical plan to patients: patients can view their surgical plans on mobile devices with doctors together, thus getting to know the details of surgery.

⧉ Intraoperative Stage

• Track laparoscopy and LUS with EM tracker: track the surgical devices together with LUS with electromagnetic to achieve image registration.
• Real-time alignment of LUS and MRI: achieve MRI-to-video registration in real-time.
• Overlay MRI on endoscopic video
• Follow the planned trajectories to do the operation

The following image illustrates the operation process:

According to the order of stages, here lists the hardwares used in our system besides traditional laproscopic surgeries:

The hardware settings during the surgery are:

The most basic component is to superimpose the rendered MRI data to the real-time laparoscopic video. To achieve this, we need to firstly preprocess the MRI images and then figure out the approaches of registration and tracking.

⧉ Preoperative Stage

MRI preprocessing is based on the request from surgeons. However, there are some common techniques that can be applied.

• Volume Rendering: visualizes all the data. Different transfer functions can be applied to distinguish some structures that are hard to tell otherwise.
• Instance Segmentation: segment different organs, tissues with help of machine learning.

⧉ Intraoperative Stage

Most of laparoscopic AR approaches should be composed of two distinct parts: an initial static registration of the augmentation to the scene and a tracking procedure to maintain its accuracy in presence of motions, deformations and occclusions.

• Registration: our solution proposes to mount the EM sensors on the 3-D laparoscope and the LUS probe. To be specific, the EM tracker is attached to the device at the handle. Later, a calibration and jitter correction is used to get the position of those devices.
• Tracking: to overlay the pre-operative MRI on to the real-time video, an automatic ultrasound-MRI registration is used

In conclusion, we prototyped a fully integrated solution to current laparoscopic surgeries with preoperative MRI and intraoperative ultrasound images based on EM tracking and 3d MRI-2d Ultrasound-endoscopic video alignment.

The system was designed to potentially used in pratical clinical environments.

The major advantages of out solution are that:

• It provides intraoperative guidance with a rapic identification of subsurface targets and critical structures.
• It spares the surgeon from having to mentally match information from different sources to the scene.
• It increase providers' spatial awareness by lifting anatomical ambiguities and thus resolve the problem that endoscopic video only provides narrow FoV.
• It provides the possibility to guide the resections by displaying cutting trajectories and margins planned beforehand on a virtual model.

1. Bernhardt, Sylvain, et al. "The status of augmented reality in laparoscopic surgery as of 2016." Medical image analysis 37 (2017): 66-90.
2. Liu, Xinyang, et al. "Laparoscopic stereoscopic augmented reality: toward a clinically viable electromagnetic tracking solution." Journal of Medical Imaging 3.4 (2016): 045001.
3. Sen, S., S. Malik, and S. Salhan. "Ultrasonographic evaluation of lower uterine segment thickness in patients of previous cesarean section." International Journal of Gynecology & Obstetrics 87.3 (2004): 215-219.
4. 3D-4K Laparoscopic Operating Rooms at Samitivej Hospital
5. Radiopaedia:Subseptate uterus
6. MRI Scan - what happens?
7. Laparoscopic Ultrasound Guided Tumor Ablation
8. Jody A. Charnow "Preoperative MRI May Benefit Some Prostate Surgery Patients", renalandurologynews, April 1, 2015
9. Anna Giorgi, "Laparoscopy", healthline, September 6, 2017
10. AffordableScan Team, "MRI cost of the Abdomen", 2018
11. "What is an MRI of the chest, abdomen or pelvis?", professionalradiology
12. MRI for diagnosis and treatment of prostate cancer
13. Intraoperative Ultrasound Imaging of the Liver