Simplify the build of SWT-binaries and make it more Maven-oriented

[HannesWell]

Discussed in #497

^{Originally posted by HannesWell December 11, 2022}
The build of the SWT-binaries is a quite complex process and I have the impression that it is even more complex than it would have to be, from the fact that o.e.swt contains the sources for all platforms but the build produces a fragment for each platform that contains all, the common and platform specific compiled code and native libraries.
I'm just about the learn the entire build-process of SWT (because of #490), so please correct me if I'm wrong or missing something.

One thing that makes it IMHO complex is the usage of multiple different ant-scripts that copy sources multiple times, back and fourth and that it looks like that the java-code is compiled multiple times and that besides the standard main- and source-jar there is also a zip produced for each platform-specific fragment.
The zip seems to be the one that is presented in the SWT section of the I-build drops:
https://download.eclipse.org/eclipse/downloads/drops4/I20221209-1800/
Is it correct that the o.e.swt and platform specific swt fragments that end up in the eclipse-updates repos (like https://download.eclipse.org/eclipse/updates/4.27-I-builds/) are the regular main- and sources-jar of the corresponding maven project?

I wonder why the code is compiled twice? And from my understanding of the build-process, as described above, this means that the class-files in the jars that end up in the p2-repo are different ones than those presented as swt-zips in the drops.

In general I think that it would be good if the swt.binaries build would be more maven oriented and I think it could be just like:

1. For each platform-specific fragment copy the corresponding java source-files from o.e.swt to the src folder
2. Build each fragment the usual maven way
3. Assemble the swt-*.zip for each fragment, but without recompiling the java-files.

This would also include that all ant based launch configs or procedures are replaced by ones that include executing a Maven build.

A second bigger point that could be improved is where/when the native-libs in each platform-specific fragment are build.
At the moment this happens in https://ci.eclipse.org/releng/job/SWT-Increment_if_needed, which is triggered in the beginning of the daily releng/I-build job.
I wonder if it would be better to migrate that job to steps that run as part of the master-branch build of this repo? The build of the native libraries (if necessary) could even be done in the beginning of the master-build so that the following verification always runs on the native libs that match it. At the moment (as far as I understand) the master branch is build against the native libs build in the last I-build. If the native libs have changed the changed lib could be committed to the swt.binaries repo in the end if the master-build succeeded.
This would allow to get rid of the SWT-Increment_if_needed and if we would use https://www.jenkins.io/doc/book/pipeline/syntax/#parallel maybe even all the platform-specific jenkins job could be removed as well (but I have not yet checked all).

This would also allow to remove the enfored qualifier for the o.e.swt Plugin:

eclipse.platform.swt/bundles/org.eclipse.swt/pom.xml

Line 32 in 1d81184

<forceContextQualifier>v20221208-2302</forceContextQualifier>

It could then just use the qualifier based on the git-timestamp, just like most other Plugins do it. If that qualifier ahd changed it could then be set and committed to the o.e.swt.binaries/binaries-parent/pom.xml, together with the changed native-libs/binaries in the end of this repos master-build.
In the I-builds pipeline the Swt build input stage could then be removed entirely.

@akurtakov, @mickaelistria, what do you think?
Maybe @SDawley or @niraj-modi are you interested in that as well?

[mickaelistria]

Is it correct that the o.e.swt and platform specific swt fragments that end up in the eclipse-updates repos (like https://download.eclipse.org/eclipse/updates/4.27-I-builds/) are the regular main- and sources-jar of the corresponding maven project?

I cannot tell for sure, but I hope so and in case it's not the case, it would be a good target to have it that way ;)

For each platform-specific fragment copy the corresponding java source-files from o.e.swt to the src folder
Build each fragment the usual maven way
Assemble the swt-*.zip for each fragment, but without recompiling the java-files.

Sounds good.

I wonder if it would be better to migrate that job to steps that run as part of the master-branch build of this repo?

At the moment, the master build is not meant to be a production build, it's only for testing. The artifact it does generate are not going through the same amount of testing and validation as I-Build, they're not as reliable.
However, having the master build rebuilding the binaries instead of consuming those from I-Build as you're trying with #514 definitely sounds like an improvement.

[HannesWell]

I wonder if it would be better to migrate that job to steps that run as part of the master-branch build of this repo?

At the moment, the master build is not meant to be a production build, it's only for testing. The artifact it does generate are not going through the same amount of testing and validation as I-Build, they're not as reliable. However, having the master build rebuilding the binaries instead of consuming those from I-Build as you're trying with #514 definitely sounds like an improvement.

At the moment the sources for o.e.swt are committed to this repo (eclipse.platform.swt) the regular way through PRs. In the verification builds of this repo only the binaries for gtk.linux.x86_64 are build from scratch and tested with the tests in this repo. The resulting binaries are not committed to eclipse.platform.swt.binaries.

In the beginning of each I-Build the Jenkins-Job releng/SWT-Increment_if_needed is triggered:
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/c2c0b2e1f87c239d1d784d6c6094f85dd7401453/JenkinsJobs/Builds/I_build.groovy#L232-L236

If there are changes in o.e.swt, that require recompilation of the SWT-binaries, then this job triggers six other jobs, one to build the swt-binaries for each platform: https://ci.eclipse.org/releng/view/SWT%20Natives/
In that case the results build by those jobs are collected and then committed to the eclipse.platform.swt.binaries repo. The extensive testing on all platforms happens in the end of the I-build (similar to and together with all other components of the Eclipse platform TLPs).

The main goal of #514 is to perform compilation and committing of the SWT-binaries as part of the eclipse.platform.swt pipeline. This a first step of this issue to make the build of SWT as a whole simpler. But since every task performed in regards of changing the eclipse.platform.swt and swt.binaries is then done in that pipeline this make subsequent simplifications much simpler, because one does not have to consider other jobs defined elsewhere. And it makes all seven jobs in https://ci.eclipse.org/releng/view/SWT%20Natives/ obsolete (as far as I understood them).

We don't loose much in testing here, because at the moment the swt.binaries are build and committed in the very beginning of the I-builds and the tests are performed in the very end. So if there are test-failures the committed binary changes have to be reverted any way. #514 is just moving the binary compilation step from the beginning of the I-builds to the committing process of this repo (which btw. probably also allows to use git-timestamp based qualifier at least for o.e.swt). The extensive testing still happens at the end of the I-builds (just like for other eclipse TLP components as well).

If your main concern is to duplicate the agents of the releng JIPP/JIRO used to build SWT-binaries to the platform JIPP, we could consider to move the jobs of this repo simply from platform to the releng. While this would be a break the logical 'sorting'/'assignment' of those jobs it would simplify the management of the agents (but maybe would require more other work).
Personally I would choose to take the burdon to duplicate some agents in the platform JIRO.

[akurtakov]

What about running SWT pipeline on releng JIPP? That way we should be able to have all benefits you listed without the extra work.

[akurtakov]

One more reason to go for releng JIPP is the fact that it is configure to push to git repos which I'm not sure is done for Platform JIPP nor whether it's even desired.

[HannesWell]

What about running SWT pipeline on releng JIPP? That way we should be able to have all benefits you listed without the extra work.

Yes, that should work as well, I even suggested that already and would be fine with that :)

If your main concern is to duplicate the agents of the releng JIPP/JIRO used to build SWT-binaries to the platform JIPP, we could consider to move the jobs of this repo simply from platform to the releng. While this would be a break the logical 'sorting'/'assignment' of those jobs it would simplify the management of the agents (but maybe would require more other work).

There might be some extra configuration work necessary to move this repos pipeline to releng JIPP, but it is probably less then setting up the agents. The freeze-period check works without extra credentials/tpkens and tokens for the dash-licenses tools or alike are not necessary in this repo.
So yes from all I know and can see moving https://ci.eclipse.org/platform/job/eclipse.platform.swt/ (and maybe even https://ci.eclipse.org/platform/job/eclipse.platform.swt.binaries/) to the releng JIPP would be the simplest way.
@fredg02 do you know any obstacles?

One more reason to go for releng JIPP is the fact that it is configure to push to git repos which I'm not sure is done for Platform JIPP nor whether it's even desired.

That's a point I didn't consider yet, but yes it makes sense.
So moving to the releng JIPP seems like the best way.

[fredg02]

@fredg02 do you know any obstacles?

If the mentioned Platform SWT jobs do not deploy artifacts to Maven Central... then no.

[HannesWell]

@fredg02 do you know any obstacles?

If the mentioned Platform SWT jobs do not deploy artifacts to Maven Central... then no.

They don't. AFAIK SWT is deployed to Maven-Central together with the other parts of eclipse-platform via separate Jobs in the releng JIPP.

@akurtakov, @fredg02 so do we have consensus to move https://ci.eclipse.org/platform/job/eclipse.platform.swt/ to the releng JIPP? The binaries repo is not necessary as far as I can tell at the moment.

If yes, @fredg02 can you perform the move? Should I create a new helpdesk issue for that or can this be done under eclipsefdn/helpdesk#2448?
I can then create a PR to https://github.com/eclipse-cbi/jiro to add the required agent-labels for the releng JIPP as suggested by you in https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/2448.

[akurtakov]

Let's move it!

[fredg02]

If yes, @fredg02 can you perform the move? Should I create a new helpdesk issue for that or can this be done under eclipsefdn/helpdesk#2448?

I can move it, yes. I will actually copy the job and disable it on the platform JIPP until everything works as expected.

[fredg02]

=> https://ci.eclipse.org/releng/job/eclipse.platform.swt/

[HannesWell]

If yes, @fredg02 can you perform the move? Should I create a new helpdesk issue for that or can this be done under eclipsefdn/helpdesk#2448?

I can move it, yes. I will actually copy the job and disable it on the platform JIPP until everything works as expected.

Sounds great. Thank you! I will let you know once #514 is fully implemented and working.