Development and preliminary validation of a new task-based objective procedure-specific assessment of inguinal hernia repair procedural safety

Background: Surgical videos coupled with structured assessments enable surgical training programs to provide independent competency evaluations and align with the American Board of Surgery’s entrustable professional activities initiative. Existing assessment instruments for minimally invasive inguinal hernia repair (IHR) have limitations with regards to reliability, validity, and usability. A cross-sectional study of six surgeons using a novel objective, procedure-specific, 8-item competency assessment for minimally invasive inguinal hernia repair (IHR-OPSA) was performed to assess inter-rater reliability using a “safe” vs. “unsafe” scoring rubric. Methods: The IHR-OPSA was developed by three expert IHR surgeons, field tested with five IHR surgeons, and revised based upon feedback. The final instrument included: (1) incision/port placement; (2) dissection of peritoneal flap (TAPP) or dissection of peritoneal flap (TEP); (3) exposure; (4) reducing the sac; (5) full dissection of the myopectineal orifice; (6) mesh insertion; (7) mesh fixation; and (8) operation flow. The IHR-OPSA was applied by six expert IHR surgeons to 20 IHR surgical videos selected to include a spectrum of hernia procedures (15 laparoscopic, 5 robotic), anatomy (14 indirect, 5 direct, 1 femoral), and Global Case Difficulty (easy, average, hard). Inter-rater reliability was assessed against Gwet’s AC₂. Results: The IHR-OPSA inter-rater reliability was good to excellent, ranging from 0.65 to 0.97 across the eight items. Assessments of robotic procedures had higher reliability with near perfect agreement for 7 of 8 items. In general, assessments of easier cases had higher levels of agreement than harder cases. Conclusions: A novel 8-item minimally invasive IHR assessment tool was developed and tested for inter-rater reliability using a “safe” vs. “unsafe” rating system with promising results. To promote instrument validity the IHR-OPSA was designed and evaluated within the context of intended use with iterative engagement with experts and testing of constructs against real-world operative videos.