Model not improving even after fine-tuning and training from scratch

[ep0p]

I've been struggling to improve my model's performance using a real-world dataset, but no matter what I try, the results are disappointing:

• Fine-tuning: Tried multiple pre-trained models but got poor, incoherent outputs.

• Training from scratch: The inference results were consistently bad.

• Data filtering and synthetic data: Cleaned the data as thoroughly as possible and added synthetic samples—still no noticeable impact.

• Hyperparameter tuning: Adjusted learning rates and other parameters with no success.

It seems like using words from documents that sometimes include non-alphanumeric characters messes everything up.

The only slight improvement was with backbone freezing, but even then, the accuracy remained low.

Any suggestions or similar experiences?

[ep0p]

Well, I finally figured out why this might be happening. My dataset contains very clean and neat images of words, but I forgot to consider the impact of noise.

When running inference on noisy documents, the model struggles to recognize the words properly, which makes sense because it was only trained on pristine examples. After testing inference on a clean document, the model performed well.

Lesson learned on my part! Hope this helps someone facing similar issues in the future.

[stevemanavalan]

Hi @ep0p, @felixdittrich92

I tried fine tuning the recognition model and currently facing the issues that you have mentioned above. I tried adding synthetic images with added noise in my training and validation set and that does not seem to make much improvements. I get additional character at the end of some words and depending on the epoch it could be "i" or "!" or "_". However with scale=2 the recognition performs very well when compared to scale=1. Could this be an issue with the resolution of the synthetically generated images used for training ?

Training setup
Dataset: Synthetically generated image samples using real world and synthetic data https://github.com/felixdittrich92/synthtiger/tree/doctr-modified
Train set: ~5M samples
Validation set: ~1M samples
Epochs: 100

[ep0p]

Hi @stevemanavalan and @felixdittrich92 ,

In my tests, I used real cropped words extracted from actual documents with a 300dpi resolution.
My vocabulary included many words that ended or started with punctuation marks.
When I tried to limit these, I ran into issues where the punctuation, at the end of a word, was no longer recognized.

Adding noise to my data, such as gaussian blur or salt-and-pepper, didn’t resolve any issue.
Eventually, I had to explore alternative approaches. One solution that worked better for me was fine-tuning with backbone freezing, which resulted in fewer errors compared to previous methods.

This led me to believe that the models have a persistent bias toward handling punctuation.

In the end, I found that the pre-trained models generally performed better than any of the fine-tuned models I developed.
There are, of course, some spelling errors that need correction, but overall, the performance of the pre-trained models was quite good, better than any attempt of fine tuning.

That was just my experience, though.