A fine-tuned video transformer that names the stroke in a clip, served as an API, with its limits written down.
What I didFinal-year project: fine-tuned the model and built the API around it.
Every number on this page links to the file it came from, pinned to table-tennis-stroke-api @ 5c37bba and Table-Tennis-Stroke-Classification-using-Advanced-Transformer-Architecture @ 17918df.
Headline results
The short version
The problem
Table tennis training is full of footage and short of labels. I wanted a model that names the stroke in a video clip, as a first step toward analysing a player’s technique.
What I built
My final-year project at FAST-NUCES. I fine-tuned VideoMAE, a Vision Transformer for video, on the MediaEval table-tennis dataset from the Université de Bordeaux: 50 GB of player-centred clips across 21 classes. Then I wrapped the model in a FastAPI service. Upload a clip and get back the stroke, a confidence score, the inference time and the top five classes. The weights are on the Hugging Face Hub, there is a Docker image, and an ONNX export runs on CPU.
How I tested it
Validation accuracy on 233 clips: 85.8%, or 200 correct. I kept the best checkpoint by that score, so it is a validation number and not a separate test result, and the site says so wherever it appears.
What I would do next
Add tests for the API, measure other camera angles, try pose estimation on the strokes the model confuses, and run jevcal on its confidence scores.
How it works
Pick a request and watch it go through the system. The red ones are the requests I use to attack it. They end where the system is supposed to stop them.
The response carries the stroke, a softmax confidence, the inference time and the top 5 classes, in about 1 to 2 seconds on a CPU.
- Video clip
- FastAPI /predict
- Preprocess
- VideoMAE
- JSON response
A clip goes in and a labelled prediction comes out. The known limits, uncalibrated confidence and one camera angle, are listed in the assertions rather than hidden in the diagram.
Test results
10 claims, each with the bar I held it to and what I measured. 4 failed and 2 are unclear. They stay in, because a list where everything passes tells you nothing.
| Result | Claim | Expected | What I measured | Source |
|---|---|---|---|---|
| PASS | The model classifies 21 stroke classes from video | n/a | 85.8% validation accuracy (200 of 233 clips) | README.md ↗ @ 5c37bba |
| FAIL | 85.8% is an estimate from a held-out test setChoosing the best checkpoint by validation accuracy makes that number optimistic. I report it as a validation number and nothing more. | an untouched test split | no; it is validation accuracy on the split used to keep the best checkpoint, and no separate test number is reported | README.md ↗ @ 5c37bba |
| UNCLEAR | It beats the best earlier result on this dataset | a like-for-like comparison with 68.78% (HCMUS, MediaEval 2021) | 85.8% against 68.78%, but one is validation accuracy and the other a published result, so the gap is encouraging rather than proven | README.md ↗ @ 5c37bba |
| PASS | The API returns a prediction in a second or two on CPU | n/a | POST /predict returns the stroke, confidence, inference time and top 5, at about 1 to 2 s per clip | README.md ↗ @ 5c37bba |
| PASS | The model can run outside the PyTorch stack | n/a | an ONNX export script and an ONNX Runtime CPU script are in the repo | README.md ↗ @ 5c37bba |
| PASS | Anyone can reproduce inference | n/a | the weights are on the Hugging Face Hub and download on first start, and there is a Dockerfile | README.md ↗ @ 5c37bba |
| FAIL | The confidence score is calibratedThis is the problem the Jev audit and jevcal measure and fix. I have not applied jevcal to this model yet. | n/a | no; a wrong prediction can still come with a very high score | README.md ↗ @ 5c37bba |
| FAIL | Similar strokes are told apart | n/a | no; loops are sometimes confused with hits, and serve spin types with each other | README.md ↗ @ 5c37bba |
| UNCLEAR | It works from other camera angles | n/a | not measured; the training clips are player-centred videos from one facility | README.md ↗ @ 5c37bba |
| FAIL | The API has automated tests | n/a | no; tests for the API are still on the roadmap | README.md ↗ @ 5c37bba |
The numbers
Each headline is the conclusion. Open “View the data” under a chart for the exact values.
The model scores 17 points above the best earlier result in our report, but the two numbers are not like for like.
Validation accuracy against the best earlier result cited
85.8% is validation accuracy on the split used to pick the checkpoint. 68.78% is the best earlier result cited in the project report (HCMUS, MediaEval 2021).
200 of 233 are right. The usual confusions are loops against hits, and serve spin types against each other.
The 233 validation clips
Only the total is reported, not a per-class confusion matrix.
What went wrong
A confident wrong answer is still a confident answer
- How it surfaced
- The softmax score is not calibrated, so a wrong prediction can come back with a very high score and look exactly like a right one.
- What I did
- Not fixed here. It is documented as a limitation, and the Jev audit and jevcal are my follow-up work on measuring and correcting exactly this.
Look-alike strokes blur together
- How it surfaced
- Loops are sometimes confused with hits, and serve spin types (sidespin, topspin and backspin) with each other.
- What I did
- Not fixed. Pose estimation with MediaPipe is on the roadmap.
The training data is one facility from one angle
- How it surfaced
- The clips are player-centred videos from a single sports facility, so the model may generalise poorly to overhead, broadcast or behind-the-player views.
- What I did
- Not measured and not fixed. I say so here instead of implying robustness.
The API shipped without tests
- How it surfaced
- The service works, but nothing in the repo asserts that it keeps working.
- What I did
- Tests for the API are the first open item on the roadmap.
Choices I would defend
Fine-tune a video model instead of training one
VideoMAE is a Vision Transformer pre-trained by hiding most of each clip and reconstructing it, then fine-tuned here for stroke classification.
Report the number with its caveat
The best checkpoint was kept by validation accuracy, so 85.8% is a validation figure and is labelled as one.
Keep the service small
FastAPI exposes /predict and /health, and /health returns the model name and device (CPU or CUDA) so a deploy can be checked in one request.
Export to ONNX
The model can run with ONNX Runtime on CPU, outside the PyTorch stack.