Preserve hardlinks in artifacts

[BuildStream-Migration-Bot]

See original issue on GitLab
In GitLab by [Gitlab user @juergbi] on Mar 27, 2018, 13:46

BuildStream does not currently distinguish between files that an element intentionally creates as hardlinks and files that happen to have identical content. As part of the OSTree commit process we only record the content hash and attributes. When an artifact is staged, matching files are always hardlinked. This includes empty files.

In some situations this can cause issues. E.g., if an unintentionally hardlinked file is modified in place. The SaferHardlinks FUSE layer protects against this during integration. However, the information is also lost when creating images for deployment.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @juergbi] on Mar 27, 2018, 13:49

This was originally considered part of the broader issue about lost metadata #38. Relevant notes from https://gitlab.com/BuildStream/buildstream/issues/38#note_46463504:

• It seems that it will become interesting to store at least a list of attributes in artifact metadata so that an artifact can be properly reasoned about and standalone, regardless of it's storage backend (currently ostree or tar).

• We can also potentially use this to store a virtual inode number in order to record and the state of hardlinks which is lost by ostree. If you commit a directory containing hardlinks, that information cannot be retrieved via ostree at checkout time, related upstream bug report here.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @toscalix] on May 15, 2018, 14:14

marked this issue as related to #38

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Jul 17, 2019, 10:10

marked this issue as related to #910

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @cs-shadow] on Feb 21, 2020, 22:22

[Gitlab user @juergbi] ping :) I think this issue needs to be revisited now that we have replaced ostree-based cached with a CAS cache.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @juergbi] on Feb 22, 2020, 11:48

CAS is very similar to OSTree in this regard. While CAS now supports extension by means of node properties, I don't currently have an idea to support hardlinks without essentially breaking the Merkle tree-based design.

Has anyone encountered problems in real projects due to the lack of support for recording hardlinks? Or is this more of a theoretical concern? [Gitlab user @jjardon], [Gitlab user @valentindavid] Do you have any input on this?

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Nov 4, 2020, 10:48

marked this issue as related to #429

[nanonyme]

@juergbi you considered this a won't fix in context of BuildStream 2, right?

[juergbi]

There was a REAPI upstream discussion to provide some level of control with regards to hardlinks / inode numbers. however, nothing has been added to the REAPI spec so far. See bazelbuild/remote-apis#217

I'm not currently aware of a use case where this is a serious concern in BuildStream 2 and thus, I would tend towards closing this issue. However, if there is a good use case for this and we can come up with a reasonable solution (which we deem better than potential workarounds in BuildStream projects), I'm not opposed to supporting this.