kadireks/PanelPilot_main_repository

PanelPilot: car damage inspection

Every dent, on the right panel.

PanelPilot turns a phone photo of a car into a panel-level damage report: what damage there is, which panel it is on, and how confident the model is. It was built for rental pickup and return inspections, where forms, repair quotes and claims are all written per panel. CMU 24-679 Design and Prototyping with AI, Project 1, by Krit Adireksarn and Shane Xiong.

How it works

1. Damage detector (damage_best.pt): YOLO11m, fine-tuned. It finds damage boxes in 6 classes: dent, scratch, crack, shattered glass, broken lamp and flat tire.
2. Panel segmenter (panel_best.pt): YOLO11s-seg, trained from scratch. It outlines 21 exterior panels.
3. Matcher (pipeline.py, rule-based): assigns each damage box to the panel mask it overlaps most (at least 5%). If none overlaps, it uses the nearest panel within 5% of the image diagonal. Otherwise the finding is marked unassigned and flagged for review.

Findings with confidence below 0.5, no panel, or a nearest-panel match are flagged for the inspector to check.

Run it

1. Open the GUI notebook in Colab: https://colab.research.google.com/drive/1tV3VFuaduzwY3O3ufn7KzO_MAIe3kuhz?usp=sharing (the same notebook is in this repo as Project1_CarDamage_Colab.ipynb).
2. Choose Runtime → Change runtime type → T4 GPU.
3. Choose Runtime → Run all, then open the printed *.gradio.live link, which stays valid for about 72 hours.

The notebook downloads this repo and installs everything it needs: ultralytics, gradio, huggingface_hub, reportlab, opencv-python and pandas. Each photo takes about 0.5 s on a T4 GPU and about 10 s on CPU.

Using the app

• Enter a car/rental ID, choose pickup or return, choose overall or detail shot, upload a photo, and click Analyze.
• Each finding is drawn as "# damage confidence → panel", and affected panels are shaded on a front/side/rear damage map.
• In the Findings tab you can accept, re-label or reject a finding. You can also click the photo to add damage the model missed.
• Photos of the same car add to one record until you click Clear and start a new car.
• The Export tab produces a PDF inspection report, or a ZIP with CSV, JSON, the damage map and the annotated photos.

Files

file	purpose
Project1_CarDamage_Colab.ipynb	Gradio GUI and PDF report, the end-to-end demo
pipeline.py	runs both models, matches damage to panels, draws results
cardiagram.py, car_views.png	front/side/rear damage map
damage_best.pt	damage detector weights
panel_best.pt	panel segmenter weights

Models and data

	link
Panel segmenter (trained from scratch)	https://huggingface.co/shanexf/car-panel-yolo11s-seg-scratch
Damage detector (fine-tuned)	https://huggingface.co/shanexf/car-damage-yolo11m-ft
Manual dataset (565 labelled photos)	https://huggingface.co/datasets/kadireks/cardamagepanel-originals
Augmented training set	https://huggingface.co/datasets/shanexf/cardamagepanel-rotaug
Damage data	CarDD (Wang et al., IEEE T-ITS 2023), https://cardd-ustc.github.io/

Results:

• The damage detector reaches 0.90 test mAP50, up from 0.23 for the original public weights, and its false-alarm rate on clean cars is 4%, down from 75%.
• The panel segmenter reaches 0.89 validation mask mAP50.

Limitations

• Glare can be detected as a scratch or shattered glass. The GUI has a separate threshold for glass and lamps.
• Very close or extreme-angle photos may show no panels. Use detail-shot mode and name the panel yourself.
• Panels have no left/right labels, and rare panels such as the roof and license plate are less accurate.
• Neither model has been tested end-to-end on new real-world photos. A "no damage" result depends on the threshold (default 0.25) and is not proof that a car is undamaged.
• An inspector should review every finding before it is used for any charge or claim.

License

Research and coursework use only. The damage weights are trained on CarDD and fall under the CarDD licence agreement, so CarDD images are not included in this repo.

AI tool usage

Claude (Anthropic) helped write the Gradio GUI and PDF report code and draft our report and slides. Gemini in Colab helped write and debug the training notebooks. We reviewed, tested and edited all AI-generated code and text, and every decision about the problem, data, models and design was made by the authors.