The surgeons’ desire for technologies that transcend the limitations of the human eye is growing prominent. Intuitive, an innovation company focused on minimally invasive care, is transforming vision technology to assist surgeons in achieving superior visual perception. The company’s advancements include the work of Intuitive’s Vice President of Advanced Imaging, Jeffrey DiCarlo, who speaks on the imaging technologies that will revolutionise surgical practices.
Understanding what the surgeon wants to see
DiCarlo likens the design process to playing Whac-A-Mole. “If you spend all your time getting one thing perfect, a completely different issue crops up—one that was masked by the first thing you fixed. It’s important to stay focused on the big picture, which is what the surgeon needs to see during surgery.”
To better understand what to focus on, DiCarlo and his colleagues rely heavily on surgeons for input and advice. “Often,” he said, “what surgeons want from the image is different from what myself and my colleagues expect.” Not long ago, he was talking to a surgeon, who said that a new endoscope image looked “digital.” At first, DiCarlo was confused by this—after all, Intuitive’s vision systems have always been digital. But then the surgeon invited him to watch a procedure so he could see for himself. In that setting, DiCarlo understood what the surgeon was referring to. Because they are making alterations to patients’ tissue, surgeons tend to focus on the area at the centre of the image, the tip of the needle.
“What we realised,” DiCarlo said, “is that a lot of the time, that’s where the surgeon is looking—where to put the tip of the needle. So that part of the image must be especially clear.” DiCarlo and his colleagues modified the system, eliminating the issue.
DiCarlo said many surgeons are interested in technologies to help them see more than the human eye typically does. “We want to give the surgeon new visual information,” he added. Some Intuitive systems already offer Firefly, a technology that uses near-infrared fluorescence for real-time visualisation of blood vessels, lymphatic vessels and other often-hard-to-see tissue. DiCarlo and his colleagues are working to develop new imaging technologies that will further enhance a surgeon’s understanding and view, beyond what the naked eye can see.
As challenging as his job can be, DiCarlo loves it. “There’s a ton of brainstorming,” he said. “A ton of ‘hey, can we improve this? Can we improve that?’ There’s always a solution. And it’s important work. I mean, we’re helping patients, helping doctors help patients. To me, that’s very fulfilling.”
Sixteen milliseconds to adjust each image
According to DiCarlo, Intuitive’s vision system is much more than simply a camera. “It really involves so much more than that. There’s the light source; how can we optimise that, so the camera picks up the information it needs? There’s image processing; how do we integrate the raw information from the lenses and sensors so that it’s as accurate as possible? Then there’s the display; how do we present the images so the surgeon has the clearest, sharpest view?” DiCarlo said.
As this suggests, the work requires input from many different kinds of experts, including electrical and mechanical engineers, software engineers and industrial designers. “At Intuitive, everybody knows that no one person can solve the problem,” DiCarlo said, “Everybody works together to find the best solution. That’s the part that I really enjoy, the part that I look forward to every day.”
These experts must find solutions that solve a range of constraints. Because robotic surgery is minimally invasive, the vision system must be extremely compact. For example, the lenses on the tip of the endoscope—there are two, one for each eye, so the surgeon can have three-dimensional binocular vision—are each less than four millimetres in diameter, allowing them to fit into an eight-millimetre endoscope. Three millimetres is roughly the size of a tiny screw. Intuitive’s latest endoscope system, introduced in 2019, has another advantage: it keeps more of the visual field in focus. All tissue that lies between two and 14 centimetres from the lens is sharp, ending the need for surgeons to constantly refocus the camera lens. The company also manufactures its own endoscopes, which helps create a seamless system.
Space isn’t the only limiting factor: there’s also time. Every image must travel what Intuitive engineers call the ‘imaging pipeline’—the sequence of steps that span from image acquisition to final display. Between the light hitting the camera sensor and the surgeon seeing the scene in the viewfinder, the image must go through more than 10 separate steps. To ensure that the surgeon sees exactly what’s happening as it happens, the image is refreshed 60 times a second.
“That gives us 16 milliseconds to get the image adjusted before the surgeon sees it,” DiCarlo explained. “That’s the limit because there’s another frame coming down the line.”
Some of the tasks that the vision system must accomplish in this tiny time frame are:
- Adjusting any pixels that don’t look as they should, so the surgeon doesn’t see any blips or spots on the screen. The image contains more than a million pixels and invariably a small percentage have issues, for example not representing the correct hue or tone. A tuning algorithm corrects these.
- Modifying lens distortion. For obvious reasons, the lens in the Intuitive vision system is quite close to the surgical site. To ensure that the surgeon can see all relevant tissue, Intuitive uses a wide-angle lens. Such lenses include more of the scene, but also “bend” the image; this curvature must be continually straightened.
How Intuitive innovates on vision technology
When he was getting a PhD in electrical engineering at Stanford University, Jeffrey DiCarlo became fascinated by a question that might not seem all that relevant to electrical engineering: how do we perceive the world visually? This subject is quite pertinent. After all, visual perception—how the eye and brain grasp and process reality—occurs via electrical signals travelling through the nervous system. DiCarlo began working with a psychology professor who also taught electrical engineering, focusing on how humans discern colour and texture.
He is still fascinated by this topic; in fact, he’s spent the past two decades thinking about visual perception. He’s put his fascination to practical use. For the past 12 years, he’s worked at Intuitive, helping to develop better ways for surgeons to see what they need to see when they’re using the company’s robotic surgical systems.
For any surgeon, being able to see clearly during a procedure is crucial. This is perhaps especially true for minimally invasive surgery, where the clinician is looking at a monitor, rather than directly at the patient’s anatomy. Since its inception, Intuitive has focused on giving surgeons a bright, well-defined view of the tissue they’re operating on. As the company has grown and its products have evolved, this emphasis has only increased; there are now more than 100 people working on the vision technology for the company’s various robotic systems.
Over the past quarter-century, Intuitive has added key innovations to its vision technology. The company’s first vision system included three-dimensional vision so that surgeons could see tissue in a more realistic way. After a few years, Intuitive added HD. More recently Intuitive engineers have reduced the size of the camera so that it can fit onto the tip of the endoscope—so-called chip-on-tip technology. This reduces the size of the endoscope, without lowering the quality, which can reduce the size of the incision. It can also help give surgeons increased maneuverability and control during procedures. The system has also steadily improved the subtlety with which it presents the colours, tones and textures of patients’ tissue—crucial information for surgeons.
“Intuitive has made a real investment in image quality,” said DiCarlo, “That’s something that surgeons really care about.”
