File not found error on production when fetching USFM

[Enkidu93]

We hit a whole bunch of these on production over the weekend all from one project:

System.IO.FileNotFoundException 
/api/v1/translation/engines/<id>/corpora/<id>/pretranslations/<book>/usfm
Could not find file '/var/lib/serval/files/<filename>'.

[Enkidu93]

@pmachapman has already fixed the issue for the particular project, I believe.

To make sure this isn't more widespread, I've confirmed that all file ids in all corpora exist. I suspect this may be an issue with just the file names though.

[ddaspit]

How did Peter fix the project?

[Enkidu93]

How did Peter fix the project?

I think he re-uploaded files (i.e. updated them). It looks kosher in the db now.

[ddaspit]

Did the files get deleted somehow?

[Enkidu93]

I've just confirmed that there are no other missing filenames - i.e., filenames that are referenced in parallel corpora on engines but do not exist in the data_files collection. Is it possible that the pod crashed or was updated or something before the DataFileUpdatedConsumer consumed?

[Enkidu93]

Did the files get deleted somehow?

I'm not sure. @pmachapman would know - I'm not sure what all he did. That's possible; the other possibility is that the filename was updated 🤔.

[pmachapman]

How did Peter fix the project?

I think he re-uploaded files (i.e. updated them). It looks kosher in the db now.

Yes, I did reupload them via Swagger.

Did the files get deleted somehow?

I'm not sure. @pmachapman would know - I'm not sure what all he did. That's possible; the other possibility is that the filename was updated 🤔.

I think they were corrupted during upload. Perhaps the connection was cancelled between SF and Serval? Maybe even the upload took place during scheduled maintenance or a deployment.

The build was all blank, and so I reran the build yesterday to see if the build was blank because of the source/target data (the build generated pre-translations, which makes me think that the files were missing/corrupt on upload).

[Enkidu93]

Is there anything else we can/should do to follow up on this? Or should we close it as can't reproduce?